I'm trying to reduce the possibility of a race condition if invalidation of my redis cache occurs at the same time I'm retrieving from the cache. Note that invalidation and retrieval happen on two different systems, so I don't know whether this is happening at the same time.
System 1:
InValidateCache() {
_cache.remove(key);
}
System 2:
GetCacheKey() {
string key = _cache.get();
}
Here, key could return the dirty string which has been invalidated in System 1 (since invalidation of cache in System 1 could happen after retrieval of cache in System 2).
How do I make sure that doesn't happen? Is there a retry or another approach I could take to reduce the possibility?
The question lacks the context or usecase for which you need such a solution. However based on the details provided I will try to answer the question.
You could use Mutex based approach to access the cache store and get distributed locks which will prevent the race condition. An overview of the reading/writing operation will look like
Steps:
Acquire the lock
Perform the operation (read | write)
Release the lock
This will be similar to using a Database lock if not exactly the same. You can use the same cache store to acquire the lock. Take a reference from this Ruby gem.
However, I feel it will be an unnecessary layer between the cache store and application unless you want to attach some business logic to the operation. For eg: Two different systems trying to update the cache key at the same time which takes into consideration the previous value of the cache read operation.
Related
I want to use DeleteManyAsync method to delete multiple documents. I will encounter big collections being deleted. In the meantime I would like my new documents to be inserted. I would like to know if my database collection will be locked when DeleteManyAsync is fired.
This is the code I want to use :
List<MyDocument> list= new List<MyDocument>();
var filter = Builders<MyDocument>.Filter.In("_id", vl.Select(i => i.InternalId));
await _context?.MyDocuments?.DeleteManyAsync(filter);
Mongo db locks are a low level concern and are handled at the database server level. You, as a programmer writing a client application using the driver, do not need to concern yourself about the database locks too much.
What I'm trying to say is that when using the C# driver you won't notice any kind of issue related to concurrent write operations executed on the same collection. Locks are handled by the storage engine, not by the driver used at the client application level.
If you check this documentation you can read that, in case of conflicting write operations on the same collection, the storage engine will retry the operation at the server level:
When the storage engine detects conflicts between two operations, one will incur a write conflict causing MongoDB to transparently retry that operation
So, again, the concurrency issues are handled at the server level.
Consider that if you need your application to be highly scalable you should design your system in order to avoid as much as possible concurrent write operations on the same collection. As I said above, locks are handled by the storage engine in order to preserve the correctness of your data, but locks can reduce the overall scalability of your system. So, if scalability is critical in your scenario, you should carefully design your system and avoid contention of resources at the database level as much as possible.
At the client application level you just need to decide whether or not retrying on a failed write operation.
Sometimes you can safely retry a failed operation, some other times you can't (e.g.: in some cases you will endup having duplicate data at the database level. A good guard against this is using unique indexes).
As a rule of thumb, idempotent write operations can safely be retried in case of a failure (because applying them multiple times does not have any side effect). Put another way, strive to have idempotent write operations as much as possible: this way you are always safe retrying a failed write operation.
If you need some guidance about the mongo C# driver erorr handling, you can take a look to this documentation
Update 25th July 2020
Based on the author comment, it seems that the main concern is not the actual database locking strategy, but the delete performances instead.
In that case I would proceed in the following manner:
always prefer a command performing a single database roundtrip (such as deleteMany) over issuing multiple single commands (such as deleteOne). By doing a single roundtrip you will minimize the latency cost and you will perform a single database command. It's simply more efficient
when you use a deleteMany command be sure to always filter documents by using a proper index, so that collection scan is avoided when finding the documents to be deleted
if you measure and you are sure that your bottleneck is the deleteMany speed, considere comparing the performances of deleteMany command with the one of an equivalent bulk write operation. I never tried that, so I have no idea about the actual speed comparison. My feeling is that probably there is no difference at all, because I supsect that under the hood deleteMany performs a bulk write. I have no clue on that, this is just a feeling.
consider changing your design in order to exploit the TTL index feature for an automatic deletion of the documents when some sort of expiration criteria is satisfied. This is not always possible, but it can be handy when applicable.
if you perform the delete operation as part of some sort of cleanup task on the data, consider scheduling a job performing the data cleanup operation on a regular basis, but outisde of the business hours of your users.
I was recently reading about the Compare And Swap atomic action (CMPXCHG, .NET's Interlocked.CompareExchange, whatever).
I understand how it works internally, and how it's used from a client.
What I can't quite figure out is when would someone use CAS?
Wikipedia says:
CAS is used for implementing synchronization primitives like
semaphores and mutexes, likewise more sophisticated lock-free and
wait-free algorithms.
So, can anyone give me a more generic real-world use case with code and description of CAS usage?
This question is meant to be language-agnostic, so any language will do (C-based or x86 assembly preferred).
Thanks!
This is easy to see by example. Say we want to atomically and concurrently set a bit on a shared variable:
int shared = 0;
void Set(int index) {
while (true) {
if (Interlocked.CompareExchange<int>(ref shared, shared | (1 << index), shared) == shared)
break; //success
}
}
We detect failure if we see that the "old value" (which is the return value) has changed in the meantime.
If this did not happen we did not have a concurrent modification so our own modification went through successfully.
You can realize pretty complex stuff using this technique. The more complex the more performance loss through spinning, though.
I want to emphasize that a key property of CAS is that it can fail and that failure can be detected reliably.
You use CAS to set a value (a bit or a word) atomically in one thread or process, while testing that another thread/process has not already done so. So it's used to acquire a flag or counter in a multi-threaded environment.
Addendum (Feb 2023)
For example, multiple threads could each use a CAS instruction to swap their process-ID into a shared word of memory (which starts out holding a value of zero). The first thread that gets its process-ID stored into the word can then take ownership of whatever resource that shared word is guarding.
When the process is done with the resource, it stores a zero into the word, releasing ownership of the resource and allowing other threads their turn to acquire the resource.
So, can anyone give me a more generic real-world use case with code and description of CAS usage?
This paper uses CAS to implement a thread safe queue without locks.
It has some pseudo code examples in it.
What is most apropriate mutex alg in C#/.NET for this kind of task.
many reads
few incremental changes (up to 3 in "go forward" state machine )
very low collision probability (does collision probability matter?).
I was thinking about simple lock or ReaderWriterLockSlim , but I am not sure which one to choose and if there is something better for this task.
Thanks.
You are going to need to perform your own benchmarks. I think you will find that in most cases a plain old lock will be faster than a ReaderWriterLockSlim even if most of the accesses qualify as read-only. The reason being that the overhead of servicing the lock is a lot higher. It has been awhile since I did the benchmarks, but I believe the ReadWriterLockSlim was about 5x slower than a lock. Obviously, holding the lock longer will reduce the overall impact of the overhead. At some point it stops being the dominating factor. Mileage will vary from one situation to another so benchmarking on your own is about the best advice I can give here.
Although this is an old thread (post, whatever), I came across a unique approach to lock versus MutEx that is beneficial under certain circumstances.
If your collisions by multiple entities desiring access are infrequent, consider the following approach:
Acquire, with lock(s) (MutEx(s) ... whatever), copies of relevant data
Ensure that at least one of the copied items is sufficient to compare
against the original in order to detect whether or not a change has
occurred
Release the lock(s)
Perform the manipulations against the copies, keeping stable your comparison item
Acquire again the lock(s)
Compare your reference item with its original to determine whether or not a change has occurred
If not, apply your results
if so, abandon your results, re-acquire the copies and start over
Release the lock(s)
For example:
Lock checking account and savings account; acquire copies of both balances and the times of latest transactions; unlock
Calculate the changes ... for example, transfer $5 from savings to checking
Lock again the accounts; compare the actual versus your copies of the transaction times- if they match, apply the calculated values and unlock, if not, re-acquire, unlock, and restart
The idea is that if there are any activities that need to lock also but do not affect your code's outcome (admittedly, the example is not a good one for this case), you have not prevented that other code's execution.
There exists some hardware for this but it is no longer "mainstream:" the IBM System 370 had an atomic compare and update instruction for just this situation.
I don't know about ReaderWriterLockSlim especially, but a reader writer mutex can be used
if multiple reads are allowed to access the critical section in parallel. If that assumption is true, depends on our use case. This is often, but not always the case.
If the assumption is meet, a reader write mutex should be a good fit.
What do you mean by "collision probability" in that context? The probability that two threads try to access the critical section concurrently?
I am using Cache in a web service method like this:
var pblDataList = (List<blabla>)HttpContext.Current.Cache.Get("pblDataList");
if (pblDataList == null)
{
var PBLData = dc.ExecuteQuery<blabla>(#"SELECT blabla");
pblDataList = PBLData.ToList();
HttpContext.Current.Cache.Add("pblDataList", pblDataList, null,
DateTime.Now.Add(new TimeSpan(0, 0, 15)),
Cache.NoSlidingExpiration, CacheItemPriority.Normal, null);
}
But I wonder, is this code thread-safe? The web service method is called by multiple requesters. And more then one requester may attempt to retrieve data and add to the Cache at the same time while the cache is empty.
The query takes 5 to 8 seconds. Would introducing a lock statement around this code prevent any possible conflicts? (I know that multiple queries can run simultaneously, but I want to be sure that only one query is running at a time.)
The cache object is thread-safe but HttpContext.Current will not be available from background threads. This may or may not apply to you here, it's not obvious from your code snippet whether or not you are actually using background threads, but in case you are now or decide to at some point in the future, you should keep this in mind.
If there's any chance that you'll need to access the cache from a background thread, then use HttpRuntime.Cache instead.
In addition, although individual operations on the cache are thread-safe, sequential lookup/store operations are obviously not atomic. Whether or not you need them to be atomic depends on your particular application. If it could be a serious problem for the same query to run multiple times, i.e. if it would produce more load than your database is able to handle, or if it would be a problem for a request to return data that is immediately overwritten in the cache, then you would likely want to place a lock around the entire block of code.
However, in most cases you would really want to profile first and see whether or not this is actually a problem. Most web applications/services don't concern themselves with this aspect of caching because they are stateless and it doesn't matter if the cache gets overwritten.
You are correct. The retrieving and adding operations are not being treated as an atomic transaction. If you need to prevent the query from running multiple times, you'll need to use a lock.
(Normally this wouldn't be much of a problem, but in the case of a long running query it can be useful to relieve strain on the database.)
I believe the Add should be thread-safe - i.e. it won't error if Add gets called twice with the same key, but obviously the query might execute twice.
Another question, however, is is the data thread-safe. There is no guarantee that each List<blabla> is isolated - it depends on the cache-provider. The in-memory cache provider stores the objects directly, so there is a risk of collisions if any of the threads edit the data (add/remove/swap items in the list, or change properties of one of the items). However, with a serializing provider you should be fine. Of course, this then demands that blabla is serializable...
I have a ASP.NET C# business webapp that is used internally. One issue we are running into as we've grown is that the original design did not account for concurrency checking - so now multiple users are accessing the same data and overwriting other users changes. So my question is - for webapps do people usually use a pessimistic or optimistic concurrency system? What drives the preference to use one over another and what are some of the design considerations to take into account?
I'm currently leaning towards an optimistic concurrency check since it seems more forgiving, but I'm concerned about the potential for multiple changes being made that would be in contradiction to each other.
Thanks!
Optimistic locking.
Pessimistic is harder to implement and will give problems in a web environment. What action will release the lock, closing the browser? Leaving the session to time out? What about if they then do save their changes?
You don't specify which database you are using. MS SQL server has a timestamp datatype. It has nothing to do with time though. It is mearly a number that will get changed each time the row gets updated. You don't have to do anything to make sure it gets changed, you just need to check it. You can achive similar by using a date/time last modified as #KM suggests. But this means you have to remember to change it each time you update the row. If you use datetime you need to use a data type with sufficient precision to ensure that you can't end up with the value not changing when it should. For example, some one saves a row, then someone reads it, then another save happens but leaves the modified date/time unchanged. I would use timestamp unless there was a requirement to track last modified date on records.
To check it you can do as #KM suggests and include it in the update statement where clause. Or you can begin a transaction, check the timestamp, if all is well do the update, then commit the transaction, if not then return a failure code or error.
Holding transactions open (as suggested by #le dorfier) is similar to pessimistic locking, but the amount of data locked may be more than a row. Most RDBM's lock at the page level by default. You will also run into the same issues as with pessimistic locking.
You mention in your question that you are worried about conflicting updates. That is what the locking will prevent surely. Both optimistic or pessimistic will, when properly implemented prevent exactly that.
I agree with the first answer above, we try to use optimistic locking when the chance of collisions is fairly low. This can be easily implemented with a LastModifiedDate column or incrementing a Version column. If you are unsure about frequency of collisions, log occurrences somewhere so you can keep an eye on them. If your records are always in "edit" mode, having separate "view" and "edit" modes could help reduce collisions (assuming you reload data when entering edit mode).
If collisions are still high, pessimistic locking is more difficult to implement in web apps, but definitely possible. We have had good success with "leasing" records (locking with a timeout)... similar to that 2 minute warning you get when you buy tickets on TicketMaster. When a user goes into edit mode, we put a record into the "lock" table with a timeout of N minutes. Other users will see a message if they try to edit a record with an active lock. You could also implement a keep-alive for long forms by renewing the lease on any postback of the page, or even with an ajax timer. There is also no reason why you couldn't back this up with a standard optimistic lock mentioned above.
Many apps will need a combination of both approaches.
here's a simple solution to many people working on the same records.
when you load the data, get the last changed date, we use LastChgDate on our tables
when you save (update) the data add "AND LastChgDate=previouslyLoadedLastChgDate" to the where clause. If the row count=0 on the update, issue error where "someone else has already saved this data" and rollback everything, otherwise the data is saved.
I generally do the above logic on header tables only and not on the details tables, since they are all in one transaction.
I assume you're experiencing the 'lost update' problem.
To counter this as a rule of thumb I use pessimistic locking when the chances of a collision are high (or transactions are short lived) and optimistic locking when the chances of a collision are low (or transactions are long lived, or your business rules encompass multiple transactions).
You really need to see what applies to your situation and make a judgment call.