I have a .Net 5 Web API project using EF Core and thought about data consistency for my business handlers. I personally think this question is not technology related and applies to other web frameworks as well. The following is just an example (the design might be bad)
Imagine the API provides two endpoints
POST /users/{userId}/documents/{documentId} to link a user to a document
DELETE /users/{userId} to delete a user
The database holds a cross table
User
Document
User1
Document1
User1
Document2
User2
Document2
The logic to link a user to a document might look like
var user = await FetchUserFromDb(userId);
if(user == null)
{
return new NotFoundResponse($"user {userId} not found");
}
var document = await FetchDocumentFromDb(documentId);
if(document == null)
{
return new NotFoundResponse($"document {documentId} not found");
}
var insertResult = await InsertIntoCrossTable(userId, documentId);
return new CreatedResponse(insertResult);
The logic to delete a user might look like
var affectedRows = await DeleteUserFromDb(userId);
if(affectedRows == 0)
{
return new NotFoundResponse($"user {userId} not found");
}
return new NoContentResponse("user deleted successfully");
Currently, I don't know what happens if both requests come in at the same time. The POST request would successfully return the user and then try to retrieve the document from the database. If the user is deleted at this point, the business logic in the POST request will no longer be aware of this. The insert query statement would throw a database exception because the foreign key for the user no longer exists.
Am I wrong here and the requests are processed one after the other? Or do I as a developer have to deal with this issue specifically (by adding additional try/catch statements for database exceptions)?
Or do I as a developer have to deal with this issue specifically (by adding additional try/catch statements for database exceptions)?
You do have to deal with it some way. Either:
Extend your transaction to include both the select and the insert, or
Handle the exception (either with retries that will also retry the select or returning an error code), or
Don't handle the exception and let the user/UI refresh on error.
what if my data source is not a database?
That's an impossibly broad question. So the answer is also broad: "use whatever concurrency techniques are recommended for your data source."
Related
I am trying to think of a best way to architect my back-end rest api. I have the following requirement:
Client makes a post request with to add a resource that contains an ID and meta information
Before I add this resource to the database I need to make a GET request to a third party API with the ID provided to fetch related data.
Then save the original resource and its related data to the database
Currently in the Repository inside the AddAsync method, before I persist the resource to the database, I make a call to the third party API to GET detail information about the resource based on the ID and THEN perform SaveChangesAsync on the model which has the rest of its properties populated by the GET request.
This however feels wrong since I am making a POST request from my client and then a GET request on the backend server. Is there a better way of solving this?
IAccountRepository:
public async Task<SupervisorResult> AddAsync(Account newAccount, CancellationToken ct = default)
{
// GetAccountDataAsync fetches order data that I need to save whenever new Account is added
SupervisorResult result = await GetAccountDataAsync(newAccount, ct);
if(result.Succeeded == false)
{
_logger.Here().Debug("Failed to get new account data.");
return result;
}
Account freshAccount = (Account)result.Value;
_dbContext.Accounts.Add(freshAccount);
await _dbContext.SaveChangesAsync(ct);
result.Succeeded = true;
result.Value = freshAccount;
return result;
}
From my point of view I don't think it's a problem to do a GET call in that situation.
Probably the best approach should be to manage possible exception from the GetAccountDataAsync.
In case a user send all the correct data, but your external system has some problem to create the account, is not responsibility of your user to "retry" the process.
This improvement could be costly, but should be better in the user experience.
I think the approach described from David (raise up one layer) is correct, so do I prefer to segregate the responsibility of the communication with external system to the application/logical layer.
In case of a partial completion of the saving process the user (Account) should be in a sort of waiting state.
I have a C# REST API with an upload endpoint that has the sole purpose to process a binary file and add its metadata (as an Attachment model) to a List<Attachment> property of a different entity.
When I call the endpoint from my web application in a sequential manner like below (pseudo code), the endpoint does as intended and processes each binary file and adds a single Attachment to the provided entity.
const attachments = [Attachment, Attachment, Attachment];
for(const attachment of attachments) {
await this.api.upload(attachment);
}
But when I try to upload the attachments in a parallel manner like below (pseudo code), each binary file gets processed properly, but only one Attachment metadata object gets added to the entity.
const attachments = [Attachment, Attachment, Attachment];
const requests = attachments.map((a) => this.api.upload(a));
await Promise.all(requests);
The endpoint basically does the following (simplified):
var attachment = new Attachment()
{
// Metadata is collected from the binary (FormFile)
};
using (var session = Store.OpenAsyncSession())
{
var entity = await session.LoadAsync<Entity>(entityId);
entity.Attachments.Add(attachment);
await session.StoreAsync(entity);
await session.SaveChangesAsync();
};
I suspect that the problem is that the endpoint is called at the same time. Both request open (at the same time) a database session and query the entity into memory. They each add the Attachment to the entity and update it in the database. The saved attachment you see in the database is from the request that finishes last, e.g. the request that takes the longest.
I've tried to recreate the issue by creating this example. When you open the link, the example runs right away. You can see the created entities on this database server.
Open the Hogwarts database and after that open the contact Harry Potter and you see two attachments added. When you open the contact Hermione Granger you only see the one attachment added (the Second.txt), although it should also have both attachments.
What is the best approach to solve this issue? I prefer not having to send the files as a batch to the endpoint. Appreciate any help!
PS: You might need to run the example manually by clicking on Run. If the database doesn't exist on the server (as the server gets emptied automatically) you can create it manually with the Hogwarts name. And because it looks like a race condition, sometimes both Attachment items are added properly. So you might need to run the example a few times.
That is a a fairly classic example of a race condition in writing to the database, you are correct.
The sequence of event is:
Req 1 load doc Attachments = []
Req 1 load doc Attachments = []
Req 1 Attachments.Push()
Req 2 Attachments.Push()
Req 1 SaveChanges()
Req 2 SaveChanges()
The change in 5 overwrites the change in 4, so you are losing data.
There are two ways to handle this scenario. You can enable optimistic concurrency for this particular scenario, see the documentation on the topic:
https://ravendb.net/docs/article-page/4.2/csharp/client-api/session/configuration/how-to-enable-optimistic-concurrency#enabling-for-a-specific-session
Basically, you can do session.Advanced.UseOptimisticConcurrency = true; to cause the transaction to fail if the document was updated behind the scenes.
You can then retry the transaction to make it work (make sure to create a new session).
Alternatively, you can use the patching API, which will allow you to add an item to the document concurrently safely.
Here is the relevant documentation:
https://ravendb.net/docs/article-page/4.2/csharp/client-api/operations/patching/single-document#add-item-to-array
Note that there is a consideration here, you shouldn't care what the order of the operations are (because they can happen in any order).
If there is a business usecase behind the order, you probably cannot use the patch API easily and need to go with the full transaction route.
I have an implementation of Identity Server 4 that uses Entity Framework Core for persistent storage and ASP.NET Core Identity for users management. Since this IDS will support public applications, we were asked to add a way of completely blocking users - which means not allowing them to sign in and remove their existing logins.
After long research, I've determined that IDS does not support anything like expiring Access Tokens, since that's not part of OpenID Connect. What strikes me as completely odd is that I switched a client to use Reference Tokens, which are correctly stored in the PersistedGrants table, but even clearing that table doesn't invalidate future requests, as the user is still authenticated both to the client application and to Identity Server itself.
Is there any store/service I can re-implement to block all access from a given logged in user?
You'll have to clear the cookies as well.
But you may want to investigate a different approach. Where IdentityServer is used as an authentication service and authorization is outsourced, like the PolicyServer.
That way you can opt-in authorization, making it less important that a user is still authenticated.
In the end, the problem was that someone had changed AccessTokenType from 1 back to 0 since another API didn't work, as it was configured for using Access Tokens rather than Reference Tokens. Thanks #VidmantasBlazevicius for pointing in the right direction of looking at logs for calls to the connect/introspect endpoint.
For reference, this is the code we ended up using (called by admin users, appropriately secured):
[HttpPut("{userId}/Block")]
public async Task<IActionResult> BlockUser(string userId)
{
var user = await _context.Users.FindAsync(userId);
if (user == null || !user.LockoutEnabled || user.LockoutEndDate > DateTime.Now)
{
return BadRequest();
}
var currentTokens = await _context.PersistedGrants
.Where(x => x.SubjectId == user.UserId)
.ToArrayAsync();
_context.PersistedGrants.RemoveRange(currentTokens);
var newLockOutEndDate = DateTime.Now + TimeSpan.FromMinutes(_options.LockOutInMinutes);
user.LockoutEndDate = newLockOutEndDate;
string updater = User.Identity.Name;
user.UpdateTime(updater);
await _context.SaveChangesAsync();
return NoContent();
}
I am using MS Dynamics CRM SDK with C#. In this I have a WCF service method which creates an entity record.
I am using CreateRequest in the method. Client is calling this method with 2 identical requests one after other immediately.
There is a fetch before creating a record. If the record is available we are updating it. However, 2 inserts are happening at the exact time.
So 2 records with identical data are getting created in CRM.
Can someone help to prevent concurrency?
You should force the duplicate detection rule & decide what to do. Read more
Account a = new Account();
a.Name = "My account";
CreateRequest req = new CreateRequest();
req.Parameters.Add("SuppressDuplicateDetection", false);
req.Target = a;
try
{
service.Execute(req);
}
catch (FaultException<OrganizationServiceFault> ex)
{
if (ex.Detail.ErrorCode == -2147220685)
{
// Account with name "My account" already exists
}
else
{
throw;
}
}
As Filburt commented in your question, the preferred approach would be to use an Alternate Key and Upsert requests but unfortunately that's not an option for you if you're working with CRM 2013.
In your scenario, I'd implement a very lightweight cache in the WCF service, probably using the MemoryCache object from the System.Runtime.Caching.dll library (small example). Before executing the query to CRM, you can check if the record exists in the cache and continue with you current processing if it doesn't (remembering to add the record to the cache with a small expiration time for potential concurrent executions) or handle the scenario where the record already exists in the cache (and here you can go from having quite complex checks to detect and prevent potential data loss/unnecessary updates to a simple and stupid Thread.Sleep(1000)).
I do understand that I might be getting this wrong since I am porting asp.net application to asp.net core 2.0 (mainly because the optimizations regarding load speed on pages) but I would ask the question anyway.
So my queries are working properly when I am fetching data only, however, I ran into a problem while having to fetch a file path from the database in order to download it on the client side. Since I don't need the whole model of the file I have 3 field dto on the client side that I fill up with the information regarding the file (etc location, size, filename) the problem is that when I send the async request toward the WCF service on Azure that's hold my entity framework link to the database the code continues further without waiting for the data to be retrieved from the database and throws null reference exception while attempting to fill the dto object that is to be sent further to the client in order to retrieve the file that's marked for downloading
This is my data access on the client side
internal async Task<AnimalDocument> GetAnimalDocument(int id)
{
var data = await _context.GetAnimalDocumentAsync(id);
var result = JsonConvert.DeserializeObject<AnimalDocument>(data);
return result;
}
And this is where I get the null exception
public SeriliazedFile GetFile(int id, int type)
{
var result = new SeriliazedFile();
if (type == 1)
{
var data = _context.GetHumanFile(id);
result.FileName = data.Result.DocumentName;
result.FilePath = data.Result.DocumentLocation;
result.FileSize = data.Result.FileSize.Value;
}
else if (type == 2)
{
var data = _context.GetAnimalDocument(id);
result.FileName = data.Result.DocumentName;
result.FilePath = data.Result.DocumentLocation;
result.FileSize = data.Result.FileSize.Value;
}
return result;
}
Is there a way to force the async request to wait for the result before returning Task that I retrieve from the WCF? I've tried telling _context.GetAnimalDocument(id).Wait(); however, nothing happens it still proceeds further without any result.I've noticed that the trigger to retrieve the data is fired after the ajax request that is sent toward the page returns 200 causing something like a deadlock but I might be wrong. If anyone could give me a work around it would be nice, I am pretty sure that I would figure it out on my own eventually but time is rare anyway, I hope you have a good day.
I am sorry for posting this, it was not a issue with the WCF or the code in any way, async works perfectly fine with asp.net core 2.0 the issue was with me. I am still adapting to the concept of have [FromBody] in front of the types in the functions, it appears to be that I missed one and I was getting id 0 by default (not that I can figure out why I would get 0 instead of null on integer field when there is no data but that doesn't matter anyway.) in my id field and the data layer was returning null value that's why I was getting null reference exception later.