I am using a web application and a windows service on the same machine as Appfabric.
Both applications reuse same DAL code (dll) which is EF (Entity Framework) Code-First based and accessing the same cache in Appfabric. The code in the windows service is implemented as a Job as part of Quartz.Net
The web application has to support multiple requests off course, and the windows service multiple threads( scheduler and events).
For both, the shared DAL dll creates a DbContext object per http session and thread ContextID or just Thread ContextID for the later. The DAL uses the EFCachingProviders from here. Also, my EF solution uses Optimistic concurrency with a timestamp columns and IsRowVersion in the mapping.
As stated here, the benefit of having a 2nd level cache is to have access to a representation of the original state across processes! But that does not seem to work for me, I get 'OptimisticConcurrencyException' in my use case as following:
restart cache cluster, restart windows service, restart iis -> clean slate :)
Using web app (firefox), I insert a new object A with reference to existing object B. I can see the new row in the database. All ok.
Using webapp in another browser (chrome) = new session, i can see the new object.
Next, the windows service tries to do some background processing and tries to update object B. This results in an 'OptimisticConcurrencyException'. Apparently the process in the windows service is holding a version of Object B with a dated rowversion.
If i restart the windows service, it tries the same logic again and works with no exception....
So both applications are multithreaded, use same DAL code, connect to same database, and same cache cluster and same cache. I would expect the update and insert to be in the appfabric cache. I would expect the EF context of the windows service to use the newest information. Somehow, it seems, that it's 1st level cache in holding on old information...
or something else is going wrong.
Please advice...
Update
Ok, after digging around, i fixed the Update problem of my windows service. Each Manager object with queries the DAL uses a DbContext bound to its Process ID + Thread ID. So in the Execute function of my Quartz Job, all Managers (of different object types) should share the same DbContext which is created by the first Manager.
The problem was, that after the function finished, the DbContext was not Disposed (which happens automatically in the HTTP Session based DbContext manager). So the next time the Job was executed, the same DbContext was found and used, which by that time was dated already (old first level cache???). The 2nd level cache should not be a problem, because that is shared and SHOULD contain newest objects... if any.
So this part is fixed.
New problem
So the web-app creates a new object A, updates an existing object B, the windows-service now works and is able to update the existing (changed) object B with no problem.
Problem:
When i do a refresh of the webapp, it does not see the changes (by the windows service) of object B....
So if the webapp changed a count to 5, 10 minutes later the windows service change that count to 6 and I open the web-app in same or new window/browser, i still see 5, not 6!
A restart of the webapp (iis) does not help, also an iisreset doesn't.
When i do Restart-CacheCluster.... it works and shows 6....
So it looks like the item is in the cache. The windows service updates it, but does not invalidate the item, which is old and used by the webapp....
Or... although the same object, the webapp has its own entry in the cache and the win-app has its own entry (which does get invalidated)....
Which one?
Solution
I solved this myself. The EF wrapper uses the query string as a key to store items in the cache, it seems. So 2 different queries (does not matter if they originate from 2 different application sharing same distributed cache or same application) referencing the same data in the database will have different keys (different query string) and so different places in the cache. Perhaps its not this black-and-white but something like this...
I don't think internally some way of algorithm is used to check if a query touches existing cached objects.
This causes my problem where my windows service does an update and the webapp still sees the old one from the cache which could only be solved by doing a Restart-CacheCluster command.
So how i fixed this:
My windows Service is a batch job triggered by the Quartz Scheduler. After it is done
I clear the whole cache:
private void InvalidateCache()
{
try
{
DataCache myCache = ...
foreach (String region in myCache.GetSystemRegions())
{
myCache.ClearRegion(region);
}
}
catch (Exception ex)
{
eventLog.WriteEntry("InvalidateCache exception : " + ex.Message);
}
}
I don't have an answer, but I hope the thoughts below might point you into the right direction.
If this is only an issue on updates, I would go for reading a fresh instance of the record on every update from the database, and update that. This would avoid optimistic concurrency errors. Note that the DbContext is not thread safe - I don't know if this would cause the issue, but reading every time new would address it.
If you are having this issue on reads, then you would have to track down where the various caches are and which one is not getting updated and why. I am guessing there are various configuration options for caching at each point of usage. Good luck with that.... :)
Related
I have a question concerning redis in a distributed architecture.
Assume I have n clients, either windows desktop applications or asp.net web/web api servers.
One of the clients, lets say client A, hits the cache for a data and has a miss (the data is not in the cache). The client then starts to get the real data (from lets say a database) and then sets it in the cache when it's done.
Client B comes along and wants the same data, does a fetch to the cache and since it's a miss, does the same processing.
Is there a way for Client B to ...(N) not to do the processing (i.e go to the database) until the data is in the cache and fetch the data from the cache instead when it's available?
I understand that on a single app (or web server), using threads it's easy to check that, but in a distributed architecture?
Is this also a correct way of thinking as well? for the wait process that is
If so then could Client A put a flag somewhere stating that he's loading Data X and that all other clients should wait until he's done?
Otherwise, the idea then would be something along the lines of :
Client A requests Data X
Miss in cache
Processes Data X
Looks if Data X is now in cache
If not, add Data X to cache, otherwise, use it and don't store it in cache
Thanks!
As Kevin said, it's called cache stampede.
One of the best documents to do with this problem I have read is Using memcached: How to scale your website easily (comes from Josef Finsel):
What we need in this instance is some way to tell our program that
another program is working on fetching the data. The best way to
handle that is by using another memcached entry as a lock.
When our program queries memcached and fails to find data, the first
thing it attempts to do is to write a value to a specific key. In our
example where we are using the actual SQL request for the key
name we can just append ":lock" to the SQL to create our new key.
What we do next depends on whether the client supports returning
success messages on memcached storage commands. If it does,
then we attempt to ADD the value. If we are the first one to attempt
this then we’ll get a success message back. If the value exists then
we get a failure indication and we know that another process is trying
to update the data and we wait for some predetermined time
before we try to get the data again.
When the process that’s updating
the cache is done, it deletes the lock key.
I have developed an Asp.Net MVC Web Application and deployed it on IIS 8 Server and in my application I am using a list to store online users and display them in a page using the following code
if (HttpRuntime.Cache["LoggedInUsers"] != null)
{
List<string> loggedInUsers = (List<string>)HttpRuntime.Cache["LoggedInUsers"];
if (loggedInUsers.Contains(model.UserName))
{
}
else
{
loggedInUsers.Add(model.UserName);
HttpRuntime.Cache["LoggedInUsers"] = loggedInUsers;
}
}
For some reason, the list gets cleared every night and when I look for the active users, I see it empty.
Is it something that has to be dealt on IIS 8 or is there any better way to implement the Online users using a database table may be..
IIS can recycle your application pool (for several reasons, including idling and no requests, too much memory use, etc...) At that point your application will be unloaded, and then loaded again later. Hence your cache values are gone.
Second, do you have any code that at some point prunes and removes old entries from cache? If not, it means you got a memory leak as it'll continue to grow indefinitely (and thereby trigger application pool recycle).
If you do have prunning code (so the cache is actively managed to avoid indefinite growth), and you need its contents to survive past pool restarts, then you have few options:
Use database. Simply have a table of active users and add/delete there.
Pro: survives even unexpected crashes of app, iis, and even machine itself.
Con: Slow due to db access and db contention point possibilities.
Put code in your application start / end event handlers to serialize contents to a file on end, and deserialize on start.
Pro: faster than db. works during graceful shutdowns.
Con: will not work due to unexpected crash.
Your site probably shuts down after a certain amount of time when there is no activity. Look at IIS settings for Application Pools (more specifically "Set idle-timeout to never in IIS") on google...
I am developing a web services project using the ServiceStack framework.
I would like to create a global object(in my case, a SessionManager object for a GDS system I am working against, it has no relation to ASP.NET sessions) to be accessed by all incoming requests.
However, I am facing a problem whereby ASP.NET will create a new instance of my application and thereby create a new instance of my SessionManager a few times in it's life cycle. I verified this by putting a debugging line on Application_Start and Application_End protected methods in the Global.asax class and realized that the Global.asax class starts and end a number of times in its life cycle. I tried declaring my SessionManager in a static class and used it via a static construct but it still creates new instances of my SessionManager. Not sure why.
So my question is how can I create a proper global (in memory) object that can be access by all requests?
Initially I thought that by using the IoC container and specifying its singleton scope that I could achieve a singleton object, but it doesn't seems like this is the case in the ASP.NET world. So please pardon me for my knowledge in the ASP.NET area as i come from a front end development background. Hope to gain some knowledge in this area from some of the experts in this community. May thanks in advance!
I am facing a problem whereby ASP.NET will create a new instance of my application and thereby create a new instance of my SessionManager a few times in it's life cycle. I verified this by putting a debugging line on Application_Start and Application_End protected methods in the Global.asax class and realized that the Global.asax class starts and end a number of times in its life cycle.
IIS Application Pool Recycling:
What you are seeing here is IIS recycling the application pool. IIS does this to try and prevent memory leaks. You can configure the recycling to occur at specific intervals.
I tried declaring my SessionManager in a static class and used it via a static construct but it still creates new instances of my SessionManager. Not sure why.
Unfortunately static variables don't survive recycling, so if your application is recycled you have to create a new instance of your SessionManager class. Which means you will need to handle persisting and restoring its state across application instances.
By default the recycling process uses an overlapped mechanism, whereby it starts an new instance of your application before terminating the old instance. This means there is no downtime to users while the application instance is shutdown and started. Unfortunately this means that you can't save the state of SessionManager in Application_End and restore it in Application_Start in the new instance because Application_End of the current instance will be called after the other application is up and running. So if you were going to do it that way, you would need to disable overlapping. But remember if you disable overlapping, there may be a small downtime then while the recycling occurs.
This article explains the recycling and the considerations.
How I would handle this:
Disable application pool recycling overlapping
Create a static instance of SessionManager that is created once when the application starts, in Application_Start.
In Application_End save the state of SessionManager to persistent storage so it can be restored in the same state when initialised in Application_Start. Perhaps serialise the state of JSON or XML.
Initially I thought that by using the IoC container and specifying its singleton scope that I could achieve a singleton object, but it doesn't seems like this is the case in the ASP.NET world.
Once you have solved the recycling issues, you don't need to use IoC to access the static object in ServiceStack as long as it is in global scope.
Maintaining the interval schedule after application restart
I have two solutions to maintaining the interval schedule. Solution 1, is simple and requires no external dependencies, though it does require to persist a date value, but this could be to a simple text file. Solution 2, is generic in that most platforms have support for it, with little configuration.
I would use a timer to run the event every 10 minutes, then record the time of the last successful check of the sessions in persistent storage (i.e. a text file, database or external cache). Then if your application is restarted, when it starts up simply determine how long it should be until the next check. This will mean that IIS application pool recycling restarts shouldn't effect the interval.
Pseudo Code:
const int interval = 10; // Run every 10 minutes
double timerInverval = 60 * interval; // In seconds
// Get last run from peristent storage
DateTime? lastRun = GetLastRunTime(); // Substitute with appropriate call from storage
// Determine elapsed time
if(lastRun.HasValue) {
var secondsRemainingUntilNextRun = (lastRun.Value.AddMinutes(interval) - DateTime.Now).TotalSeconds;
if(secondsRemainingUntilNextRun <= 0){
// Run immediately, the web application has been down and missed running the SessionManager job
SessionManager.CheckSessions(); // Substitute with appropriate call
} else {
timerInterval = secondsRemainingUntilNextRun;
}
}
// Set a timer to trigger the SessionManager job after timerInterval seconds
timer.interval = timerInterval;
Alternatively you could create a scheduled task that calls your web application and triggers this action. If the task is triggered independently of the web application, then it doesn't have to worry about maintaining the schedule if the application is restarted. I believe Azure has a scheduler service, or if you run a cloud instance then you can create a system scheduled task.
Your requirements are contradicting: you want in-memory store and you want it to be reliable and persistent and survive IIS application pool recycles. The system memory just isn't such a reliable store. If you need some persistent data you should consider using what is designed for this purpose: a database or even a file on the hard drive for example.
And of course to optimize the performance you could use an in-memory caching layer to avoid hitting your persistent layer every time you need to access the information. Another advantage to using a persistent store is that if your application is hosted in a webfarm across multiple nodes, all those nodes will be able to share the same data.
Just don't rely on IIS recycling. No matter what options you are tweaking in the IIS console, the AppPool might someday simply die wiping everything you have stored in memory and there's nothing you could do about it.
ServiceStack supports redis out of the box, as well as several other caching providers: Memcached, Azure, Disk... so the choice of where to locate your global session provider is still up to you!
You should combine caching mechanism and the singleton pattern. So you define a class that has access to the underlying cache provider, all request have once single entry point to your session manager, and use this cache provider as your data repository.
It will survive recycling, crashes and it will make your life easy, once you have to scale your application.
I got into this not very good situation.. When web application starts - I set up different routes for my services so tenants of my multi-user app connect to:
private static void RegisterRoutes()
{
// Setup URL's for each customer
using (var cmc = new CoreModelContext())
{
foreach (var account in cmc.Accounts.Where(aa => aa.IsActive).ToList())
{
RouteTable.Routes.Add(
new ServiceRoute(account.AccountId + "/mobile",
new MyServiceHostFactory(), typeof(MobileService)));
}
}
}
So, when my site/service starts - it grabs all accounts from the database and sets up the routes.
This is a single point of failure right there. Sometimes servers rebooted in wrong order and if SQL Server not started - this service starts in "weird" mode.
Today web service stopped responding. I checked logs - IIS recycled pool as scheduled (default settings) and started different worked process. Something didn't click and boom - server stopped responding. Routes wasn't registered...
So. My question is.. How to fix it best way? I can put routes to config file, but that will mean I have to maintain those id's in 2 places. Probably not that bad but I'd rather do it differently if possible.
Is it possible to to programmatically try and restart pool? What happens when exception thrown in Application_Start ? Right now I'm not trapping it.
Not sure if this is a "fix" but when we've got similar dependency issues, we make sure the other dependencies cannot successfully start in "weird" mode. In this case, I would bring the app down hard if the sql server isn't avaliable, at least in production. Far better to have nothing being processed than have things being processed wrong.
I have an application that runs as a Windows service. It stores various things settings in a database that are looked up when the service starts. I built the service to support various types of databases (SQL Server, Oracle, MySQL, etc). Often times end users choose to configure the software to use SQL Server (they can simply modify a config file with the connection string and restart the service). The problem is that when their machine boots up, often times SQL Server is started after my service so my service errors out on start up because it can't connect to the database. I know that I can specify dependencies for my service to help guide the Windows service manager to start the appropriate services before mine. However, I don't know what services to depend upon at install time (when my service is registered) since the user can change databases later on.
So my question is: is there a way for the user to manually indicate the service dependencies based on the database that they are using? If not, what is the proper design approach that I should be taking? I've thought about trying to do something like wait 30 seconds after my service starts up before connecting to the database but this seems really flaky for various reasons. I've also considered trying to "lazily" connect to the database; the problem is that I need a connection immediately upon start up since the database contains various pieces of vital info that my service needs when it first starts. Any ideas?
Dennis
what your looking for is SC.exe. This is a command line tool that users can use to configure services.
sc [Servername] Command Servicename [Optionname= Optionvalue...]
more specificly you would want to use
sc [ServerName] config ServiceName depend=servicetoDependOn
Here is a link on the commandlike options for SC.EXE
http://msdn.microsoft.com/en-us/library/ms810435.aspx
A possible (far from ideal) code solution:
In you startup method code it as a loop that terminates when you've got a connection. Then in that loop trap any database connection errors and keep retrying as the following pseudo code illustrates:
bool connected = false;
while (!connected)
{
try
{
connected = openDatabase(...);
}
catch (connection error)
{
// It might be worth waiting for some time here
}
}
This means that your program doesn't continue until it has a connection. However, it could also mean that your program never gets out of this loop, so you'd need some way of terminating it - either manually or after a certain number of tries.
As you need your service to start in a reasonable time, this code can't go in the main initialisation. You have to arrange for your program to "start" successfully, but not do any processing until this method had returned connected = true. You might achieve this by putting this code in a thread and then starting your actual application code on the "thread completed" event.
Not a direct answer put some points you can look into
Windows service can be started Automatically with a delay. You can check this question in SO for some information about it.
How to make Windows Service start as “Automatic (Delayed Start)”
Check this post How to: Code Service Dependencies