I'm making an app as a learning experience, and I've run into two major issues:
1) I have an implementation that may or may not be a good idea for Orleans...I just don't know.
2) I'm looking for advice on hosting and deployment, basically: where and how? This is just a learning experience, so small and cheap/free is important.
So, to start... A little info on the app I'm making:
It's a mobile gps based app with MongoDB storage.
The real world is divided up into distinct and persistent regions, blocks of gps locations.
When a user is active in a given region, he becomes visible to every other user in the region, and he begins receiving updates about the region's properties and the locations of other users active in the region.
The user can manipulate the properties of the region.
Users need to be able to explicitly join and leave regions, but also leave after some time-out.
I have two grains for this: UserGrain and RegionGrain.
For issue 1) Is it even a good idea to implement the Region as a grain? A few different samples support this implementation. But, the best practices manual suggests it wouldn't be a good idea, since it is a long-running job (the Region persists indefinitely), and this type of grain could be a bottleneck. But... I need it to act like the "GameGrain" from the "Presence" sample, that players join and leave (explicitly or timing out).
For issue 2)
I have a silo implemented as a console app. I've used amazon elastic beanstalk to deploy a .net web app, but I don't know where to start with a console app. Is amazon a good choice at all? I just defaulted to it from recent usage.
If the RegionGrain is a bad idea... what might a good idea look like?
If the RegionGrain is a good idea... is there a way to tell Orleans to just never deactivate it? If so, I should use Timers for the time-out requirement, and the presence and heartbeat for updates...right? If not, should I just host another console app somewhere that maintains and runs RegionGrain(s)?
If this is still an actual question:
On Issue 1) - Your region grain seems fine to me - it's not a constantly working grain but it's your region registry: when user enters the region - he registers in the region grain (or being registered by an external region monitor if you want to remove user responsibility around this). Such regions can persists indefinitely, but this doesn't mean they are long running jobs.
Long running job in orleans terminology is the one that is doing a lot of active work in a single method (or worse - sleeps for seconds or minutes) - this will hold the grain and thread from orleans thread pool - so such practice is considered bad but it's not really applicable here.
On hosting - Amazon is as good as any other hosting as long as you know how to host and run your work there. We are using Azure for our hosting - Cloud Service is very easy to start with and VM cost is usually on par with Amazon (except Azure free trial is shorter than Amazon 3.months vs 1.year but with more resources).
On Issue 2), timers and other - timers are not persistent so if the grain is deactivated - they are lost and have to be restarted. Reminders are better option to revive grain, as well as there is a method on a grain which allows you to prevent deactivation - DelayDeactivation(), the opposite one for explicit deactivation is DeactivateOnIdle().
Regarding the updates - we found that Orleans Streams are very nice concept to broadcast mass updates (e.g. from RegionGrain to all subscribed\registered users) so take a look on it and it may solve many questions with subscription and broadcasting.
In general, I'd build this a bit differently - make your system reactive instead of pulsing\proactive. e.g.:
User enters the region - so he just registering his presence on the region grain and subscribes for region events.
Any changes to the region are recorded and pushed to Orleans stream - so all subscribers will receive region event.
Any changes to other players - messaged to region grain (so it'll be our source of truth) and then re-broadcasted to other region subscribers.
In case of issues with very noisy regions (when too many changes are happening in the system and too big re-broadcasts - you can make your region just an another subscriber to the region stream - so every player will broadcast to every other via Orleans Stream and to region itself. OR make your region into a broadcast controller which would broadcast all changes in one batch to the stream only after a certain period of time - every second or every 5 seconds...
Hope this helps.
Related
We have created a dotnet core web api project which is using SQL Server database. Now, we are planning to deploy this project to Microsoft Azure.
While the deployment of this application, we are also considering to enable autoscaling option (horizontal scaling).
Before, we do it. We want to have some questions that we want to clarify.
Should we need to add some additional code in our application which allows autoscaling to work properly?
Properly in a sense, as there can be more than one instance of the application running because of horizontal scaling. We are using database and more than one instance is running will it case race condition (i.e., two resources accessing the same data at a time). I mean we can add a transaction (or use locking) in our code to avoid these kinds of scenarios?
I want to know that is there any best practices to follow while implementing that kind of application?
Thank you and waiting for your answers!
Consider the following points when designing an autoscaling strategy:
The system must be designed to be horizontally scalable. Avoid making
assumptions about instance affinity; do not design solutions that
require that the code is always running in a specific instance of a
process. When scaling a cloud service or web site horizontally, do
not assume that a series of requests from the same source will always
be routed to the same instance. For the same reason, design services
to be stateless to avoid requiring a series of requests from an
application to always be routed to the same instance of a service.
When designing a service that reads messages from a queue and
processes them, do not make any assumptions about which instance of
the service handles a specific message because autoscaling could
start additional instances of a service as the queue length grows.
The Competing Consumers pattern describes how to handle this
scenario.
If the solution implements a long-running task, design this task to
support both scaling out and scaling in. Without due care, such a
task could prevent an instance of a process from being shutdown
cleanly when the system scales in, or it could lose data if the
process is forcibly terminated. Ideally, refactor a long-running task
and break up the processing that it performs into smaller, discrete
chunks. The Pipes and Filters pattern provides an example of how you
can achieve this. Alternatively, you can implement a checkpoint
mechanism that records state information about the task at regular
intervals, and save this state in durable storage that can be
accessed by any instance of the process running the task. In this
way, if the process is shutdown, the work that it was performing can
be resumed from the last checkpoint by using another instance.
For more information, follow the doc : https://github.com/Huachao/azure-content/blob/master/articles/best-practices-auto-scaling.md
Regarding this:
Properly in a sense, as there can be more than one instance of the application running because of horizontal scaling. We are using database and more than one instance is running will it case race condition (i.e., two resources accessing the same data at a time). I mean we can add a transaction (or use locking) in our code to avoid these kinds of scenarios?
Please keep in mind that, even if the app is running on a single machine, requests will still be handled concurrently. This means that even on a single machine 2 requests can cause the same entry in the database to be updated. So the above questions about race conditions apply to single instance web apps as well.
Try to avoid locking: the whole point of (horizontal) scaling is to gain performance benefits. By using locks you effectively remove this benefits as only one process at a time can use the locked resource.
Other points of considerations are:
If you are using an in-memory cache you might want to swap it out for a distributed cache.
The guidance at the MS docs
I have a Windows Phone game that requires support for multiplayer. The multiplayer is similar to the one in Wordament: everyone plays the same game; the client gets the game initially, then the each player plays the game on his own without any interaction with the others and when the game ends, the results from everyone are collected and displayed. The difference is: in my application, the game doesn't end after a specified period of time but rather when one of the clients signals it. So, when someone completes the game (reaches a goal), all the others have to be notified that someone won.
My initial thought is pool the server every let's say 5 seconds to see if the game state has been changed. When a client completes the game, it sends a request with that info and all the other clients, upon the next pool request, will get the new status. This, IMO, is the simplest and most convenient solution because all I need is one byte of data to tell me if the game is over or not.
Real time (as in millisecond accuracy) is not critical. As you might have noticed in the previous paragraph, a 5 seconds delay is acceptable.
However, I am asking you, experts, if a duplex channel would be more appropriate for this scenario? I found solutions like Pusher which provide the two way channel but it seems to me that such a solution is very complex and expensive (we have a very limited budget).
Will share my current knowledge.
Pull(Poll)
Simple to implement, widely used.
Examples: Facebook.com, TeamCity web interface, .NET Client for QPID Message Broker
Push
Take a look at this article
Performance of HTTP polling duplex server-side channel in Microsoft Silverlight 3
What I've noticed for myself: need extra efforts for configuration, possible issues with scalability and performance
The only scenario I can think of - exchange of large amount of data on constant basis
Example: Massively multiplayer online games(huges number of events, notification time is extremely critial)
Get changes on demand
Typical for bussines desktop application.
Examples: TFS(refresh grids(tasks and bugs), get locked file status on check out)
Conclusion: Pooling for your task fits ideally
I have an NHibernate MVC application that is using ReadCommitted Isolation.
On the site, there is a certain process that the user could initiate, and depending on the input, may take several minutes. This is because the session is per request and is open that entire time.
But while that runs, no other user can access the site (they can try, but their request won't go through unless the long-running thing is finished)
What's more, I also have a need to have a console app that also performs this long running function while connecting to the same database. It is causing the same issue.
I'm not sure what part of my setup is wrong, any feedback would be appreciated.
NHibernate is set up with fluent configuration and StructureMap.
Isolation level is set as ReadCommitted.
The session factory lifecycle is HybridLifeCycle (which on the web should be Session per request, but on the win console app would be ThreadLocal)
It sounds like your requests are waiting on database locks. Your options are really:
Break the long running process into a series of smaller transactions.
Use ReadUncommitted isolation level most of the time (this is appropriate in a lot of use cases).
Judicious use of Snapshot isolation level (Assuming you're using MS-SQL 2005 or later).
(N.B. I'm assuming the long-running function does a lot of reads/writes and the requests being blocked are primarily doing reads.)
As has been suggested, breaking your process down into multiple smaller transactions will probably be the solution.
I would suggest looking at something like Rhino Service Bus or NServiceBus (my preference is Rhino Service Bus - I find it much simpler to work with personally). What that allows you to do is separate the functionality down into small chunks, but maintain the transactional nature. Essentially with a service bus, you send a message to initiate a piece of work, the piece of work will be enlisted in a distributed transaction along with receiving the message, so if something goes wrong, the message will not just disappear, leaving your system in a potentially inconsistent state.
Depending on what you need to do, you could send an initial message to start the processing, and then after each step, send a new message to initiate the next step. This can really help to break down the transactions into much smaller pieces of work (and simplify the code). The two service buses I mentioned (there is also Mass Transit), also have things like retries built in, and error handling, so that if something goes wrong, the message ends up in an error queue and you can investigate what went wrong, hopefully fix it, and reprocess the message, thus ensuring your system remains consistent.
Of course whether this is necessary depends on the requirements of your system :)
Another, but more complex solution would be:
You build a background robot application which runs on one of the machines
this background worker robot can be receive "worker jobs" (the one initiated by the user)
then, the robot processes the jobs step & step in the background
Pitfalls are:
- you have to programm this robot very stable
- you need to watch the robot somehow
Sure, this is involves more work - on the flip side you will have the option to integrate more job-types, enabling your system to process different things in the background.
I think the design of your application /SQL statements has a problem , unless you are facebook I dont think any process it should take all this time , it is better to review your design and check where is the bottleneck are, instead of trying to make this long running process continue .
also some times ORM is not good for every scenario , did you try to use SP ?
I am writing a c# windows service which will perform some background processing - basically it is a consumer for a work queue.
It needs to not go down (stop processing new items), and if it does go down I need to be notified.
What are some design guidelines and considerations for a) ensuring that such a service is as reliable as possible, and b) sending out a notification if something does go wrong? I have considered, for instance, creating a watcher thread whose only job is to make sure the worker thread is still processing jobs.
There are a number of things that you can do here to help improve the reliability, as well as gauge that you have a solution that is going to meet your needs.
Testing
First and foremost though, the testing process that you go through will need to be a very solid one, test for those "unexpected" situations, loss of network connection, etc. Make sure that you are testing those, and seeing what is happening. Notification on failure, can be a bit of a "mixed bag". For example, you can't e-mail yourself if you don't have network connections available.
Proper Code Design
In addition to setting up valid test scenarios, be sure that your code is a bullet proof as possible, since you are creating a windows service, be sure that you are capturing, logging, and dealing with all errors possible, as if an error bubbles up to the OS, your service will go down.
Monitoring
Consider putting monitoring, in my day-job we have two types of monitoring used, errors are reported the the Windows Event log in some cases and Microsoft MOM is used to notify us of any/all issues that are going on in the environment. A second process that we use is a second scheduled job that every X minutes validates that the critical job is in a "Started" state, if it isn't in a started state, it will re-start it. Not elegant, but it works.
I think a MOM and/or Solar Winds or some other monitoring application which your system administrator might be using to monitor the machine on which the service is deployed & take proper action (send email, ring phones :)
It has become apparent that where I work needs, internally, a "notification system". The issue being that we are very spread out throughout multiple buildings and the bulk of the work force regularly keeps there email closed for hours at a time.
I need to create a simple way to be able to push out a message and have it "pop up" on everyones computer(or a single computer).
My first thought was to write a windows service that calls a winform/wpf app that resides on each computer that simply pops up with the message. Not sure how viable an idea that is but this is just brain-storming.
A different route, I thought, could be an app that resides in the systray on each computer that polls a db table and using the Query Notifications could pop up a message each time a new row is added. Then simply create an insanely basic app for writing a row to that table.
So, what I am asking is if any one else has walked this path. If so, how?
What things did you take into
consideration?
Are either of my ideas valid starting
points or are "egg and my face in
perfect alignment"?
Is there a different way that is even
simpler?
Thanks
Some simple requirements --> Must be "One Way" as I cannot give our user base a "chat" system. Must be, somewhat, hidden so as to discourage users shutting it off. A la system tray or service.
Wouldn't net send save you reinventing the wheel?
I've never done this but I've worked in a call-centre that did use something similar and they're insanely useful. I remember once when everyone got a message saying "does anyone know Mandarin? HELP ME!!" Brilliant. (Luckily someone did.)
Anyway your ideas are perfectly fine. Personally I'd do this as a client/server application. A windows forms or WPF application that sits in the systray could link to a server using a TCP/IP duplex connection using WCF. Perhaps get the client to register to certain groups depending on the department of the PC it's sitting on.
Then when someone wants to send a message they select which group it needs to go to (or all groups), the message hits the server which sends out to all connected clients, and the WPF app on the computer receives the message and pops it up. You don't even need a database except to store the users/groups, and the message history if you need to.
This might be a ridiculous answer but have you considered implementing a chat system? It's simple to implement and well tested.
Here are some possibilities:
http://messenger.softros.com/
http://en.wikipedia.org/wiki/Instant_messaging#User_base
Article on building your own:
http://www.computerworld.com/s/article/9002059/How_to_build_your_own_corporate_IM_system_
The easiest way to do this is to have a simple client on each machine polling a central service for alerts. Each alert should have a unique id so each client can deal with idempotency (you don't want the central service keeping tabs on which clients have "popped up").
I also recommend having a reasonably short lifespan for each alert, so the client only needs to know a very short list of alerts it has displayed and so if a machine was re-started, only a small history of alerts would be displayed.
With 300 subscribers, you'll want the polling to involve a nice long gap too - you don't really want 300 checks every 10 seconds - so you'll have to balance the technical desire for long gaps between checks with the business requirement to get an alert within a certain timeframe.
You could easily achieve this with a NET/TCP WCF service being polled by either a WINFORM / WPF application that is added as a start up program, or a windows service that then spawns a UI to display the notification.
I did something like this a long time ago to coordinate smoke breaks. I simply sent a broadcast packet out on the LAN at a specific port. Worked relatively well, although since anybody could broadcast and everybody would get a popup, it got abused a lot.
I would recommend you SPARK. We have same problem in my firm and finally decided to save time and do not reinventing the wheel and use existing (freeware) solution. SPARK does the job for us.
"Spark is an Open Source, cross-platform IM client optimized for businesses and organizations. It features built-in support for group chat, telephony integration, and strong security. It also offers a great end-user experience with features like in-line spell checking, group chat room bookmarks, and tabbed conversations."
If you cannot use / install existing IMs you might thing about implementing simple "chat" protocol in your app.
It is quite easy do that base on sockets and many articles available.
For example:
http://www.codeproject.com/KB/IP/TCPIPChat.aspx
http://www.codeproject.com/KB/miscctrl/SimpleMessenger.aspx?display=Print
If you need something advanced (eg. receive historical notification, users status management etc) you can consider using openSource Jabber API:
Eg http://www.codeproject.com/KB/gadgets/googletalk.aspx