I am using MS Dynamics CRM SDK with C#. In this I have a WCF service method which creates an entity record.
I am using CreateRequest in the method. Client is calling this method with 2 identical requests one after other immediately.
There is a fetch before creating a record. If the record is available we are updating it. However, 2 inserts are happening at the exact time.
So 2 records with identical data are getting created in CRM.
Can someone help to prevent concurrency?
You should force the duplicate detection rule & decide what to do. Read more
Account a = new Account();
a.Name = "My account";
CreateRequest req = new CreateRequest();
req.Parameters.Add("SuppressDuplicateDetection", false);
req.Target = a;
try
{
service.Execute(req);
}
catch (FaultException<OrganizationServiceFault> ex)
{
if (ex.Detail.ErrorCode == -2147220685)
{
// Account with name "My account" already exists
}
else
{
throw;
}
}
As Filburt commented in your question, the preferred approach would be to use an Alternate Key and Upsert requests but unfortunately that's not an option for you if you're working with CRM 2013.
In your scenario, I'd implement a very lightweight cache in the WCF service, probably using the MemoryCache object from the System.Runtime.Caching.dll library (small example). Before executing the query to CRM, you can check if the record exists in the cache and continue with you current processing if it doesn't (remembering to add the record to the cache with a small expiration time for potential concurrent executions) or handle the scenario where the record already exists in the cache (and here you can go from having quite complex checks to detect and prevent potential data loss/unnecessary updates to a simple and stupid Thread.Sleep(1000)).
Related
I am trying to think of a best way to architect my back-end rest api. I have the following requirement:
Client makes a post request with to add a resource that contains an ID and meta information
Before I add this resource to the database I need to make a GET request to a third party API with the ID provided to fetch related data.
Then save the original resource and its related data to the database
Currently in the Repository inside the AddAsync method, before I persist the resource to the database, I make a call to the third party API to GET detail information about the resource based on the ID and THEN perform SaveChangesAsync on the model which has the rest of its properties populated by the GET request.
This however feels wrong since I am making a POST request from my client and then a GET request on the backend server. Is there a better way of solving this?
IAccountRepository:
public async Task<SupervisorResult> AddAsync(Account newAccount, CancellationToken ct = default)
{
// GetAccountDataAsync fetches order data that I need to save whenever new Account is added
SupervisorResult result = await GetAccountDataAsync(newAccount, ct);
if(result.Succeeded == false)
{
_logger.Here().Debug("Failed to get new account data.");
return result;
}
Account freshAccount = (Account)result.Value;
_dbContext.Accounts.Add(freshAccount);
await _dbContext.SaveChangesAsync(ct);
result.Succeeded = true;
result.Value = freshAccount;
return result;
}
From my point of view I don't think it's a problem to do a GET call in that situation.
Probably the best approach should be to manage possible exception from the GetAccountDataAsync.
In case a user send all the correct data, but your external system has some problem to create the account, is not responsibility of your user to "retry" the process.
This improvement could be costly, but should be better in the user experience.
I think the approach described from David (raise up one layer) is correct, so do I prefer to segregate the responsibility of the communication with external system to the application/logical layer.
In case of a partial completion of the saving process the user (Account) should be in a sort of waiting state.
I have a .Net 5 Web API project using EF Core and thought about data consistency for my business handlers. I personally think this question is not technology related and applies to other web frameworks as well. The following is just an example (the design might be bad)
Imagine the API provides two endpoints
POST /users/{userId}/documents/{documentId} to link a user to a document
DELETE /users/{userId} to delete a user
The database holds a cross table
User
Document
User1
Document1
User1
Document2
User2
Document2
The logic to link a user to a document might look like
var user = await FetchUserFromDb(userId);
if(user == null)
{
return new NotFoundResponse($"user {userId} not found");
}
var document = await FetchDocumentFromDb(documentId);
if(document == null)
{
return new NotFoundResponse($"document {documentId} not found");
}
var insertResult = await InsertIntoCrossTable(userId, documentId);
return new CreatedResponse(insertResult);
The logic to delete a user might look like
var affectedRows = await DeleteUserFromDb(userId);
if(affectedRows == 0)
{
return new NotFoundResponse($"user {userId} not found");
}
return new NoContentResponse("user deleted successfully");
Currently, I don't know what happens if both requests come in at the same time. The POST request would successfully return the user and then try to retrieve the document from the database. If the user is deleted at this point, the business logic in the POST request will no longer be aware of this. The insert query statement would throw a database exception because the foreign key for the user no longer exists.
Am I wrong here and the requests are processed one after the other? Or do I as a developer have to deal with this issue specifically (by adding additional try/catch statements for database exceptions)?
Or do I as a developer have to deal with this issue specifically (by adding additional try/catch statements for database exceptions)?
You do have to deal with it some way. Either:
Extend your transaction to include both the select and the insert, or
Handle the exception (either with retries that will also retry the select or returning an error code), or
Don't handle the exception and let the user/UI refresh on error.
what if my data source is not a database?
That's an impossibly broad question. So the answer is also broad: "use whatever concurrency techniques are recommended for your data source."
I have a C# REST API with an upload endpoint that has the sole purpose to process a binary file and add its metadata (as an Attachment model) to a List<Attachment> property of a different entity.
When I call the endpoint from my web application in a sequential manner like below (pseudo code), the endpoint does as intended and processes each binary file and adds a single Attachment to the provided entity.
const attachments = [Attachment, Attachment, Attachment];
for(const attachment of attachments) {
await this.api.upload(attachment);
}
But when I try to upload the attachments in a parallel manner like below (pseudo code), each binary file gets processed properly, but only one Attachment metadata object gets added to the entity.
const attachments = [Attachment, Attachment, Attachment];
const requests = attachments.map((a) => this.api.upload(a));
await Promise.all(requests);
The endpoint basically does the following (simplified):
var attachment = new Attachment()
{
// Metadata is collected from the binary (FormFile)
};
using (var session = Store.OpenAsyncSession())
{
var entity = await session.LoadAsync<Entity>(entityId);
entity.Attachments.Add(attachment);
await session.StoreAsync(entity);
await session.SaveChangesAsync();
};
I suspect that the problem is that the endpoint is called at the same time. Both request open (at the same time) a database session and query the entity into memory. They each add the Attachment to the entity and update it in the database. The saved attachment you see in the database is from the request that finishes last, e.g. the request that takes the longest.
I've tried to recreate the issue by creating this example. When you open the link, the example runs right away. You can see the created entities on this database server.
Open the Hogwarts database and after that open the contact Harry Potter and you see two attachments added. When you open the contact Hermione Granger you only see the one attachment added (the Second.txt), although it should also have both attachments.
What is the best approach to solve this issue? I prefer not having to send the files as a batch to the endpoint. Appreciate any help!
PS: You might need to run the example manually by clicking on Run. If the database doesn't exist on the server (as the server gets emptied automatically) you can create it manually with the Hogwarts name. And because it looks like a race condition, sometimes both Attachment items are added properly. So you might need to run the example a few times.
That is a a fairly classic example of a race condition in writing to the database, you are correct.
The sequence of event is:
Req 1 load doc Attachments = []
Req 1 load doc Attachments = []
Req 1 Attachments.Push()
Req 2 Attachments.Push()
Req 1 SaveChanges()
Req 2 SaveChanges()
The change in 5 overwrites the change in 4, so you are losing data.
There are two ways to handle this scenario. You can enable optimistic concurrency for this particular scenario, see the documentation on the topic:
https://ravendb.net/docs/article-page/4.2/csharp/client-api/session/configuration/how-to-enable-optimistic-concurrency#enabling-for-a-specific-session
Basically, you can do session.Advanced.UseOptimisticConcurrency = true; to cause the transaction to fail if the document was updated behind the scenes.
You can then retry the transaction to make it work (make sure to create a new session).
Alternatively, you can use the patching API, which will allow you to add an item to the document concurrently safely.
Here is the relevant documentation:
https://ravendb.net/docs/article-page/4.2/csharp/client-api/operations/patching/single-document#add-item-to-array
Note that there is a consideration here, you shouldn't care what the order of the operations are (because they can happen in any order).
If there is a business usecase behind the order, you probably cannot use the patch API easily and need to go with the full transaction route.
So I have a few different parse servers setup.
One server is just to capture error logs from various applications (I have a LOT out there) in nice uniformed database.
So I might have a specific standalone data migration tool that if it encounters an error, will write out the exception into this Error_log parse table/class. No problem there.
But, if I have an app that uses a Parse Database for itself, I have not been able to figure out how to let it work on it's own parse server configuration for it's own stuff, but write out error logs to this other Parse server instance.
Yes... I could go through the trouble of writing out something via the REST api just for writing out logs,but I am I trying to avoid that and stick with native parse APIs for the particular platform I am on because of the benefits that the APIs give over REST (like save eventually for the none .NET stuff).
EDIT
Some clarification was requested so here I go...
On the app side of things (c# for this example but the same holds true for iOS etc)… I do the usual initialization of the Parse client as such …
ParseClient.Initialize(new ParseClient.Configuration
{
ApplicationId = "MyAppID",
WindowsKey = "MyDotNetKey",
Server = "www.myparseserver.com/app1"
});
So for all calls to save a parse object go through that parse client connection
But what I need to do would be something like this ….
//Main App cloud database
ParseClient1.Initialize(new ParseClient.Configuration
{
ApplicationId = "MyAppID",
WindowsKey = "MyDotNetKey",
Server = "www.myparseserver.com/app1"
});
ParseClient2.Initialize(new ParseClient.Configuration
{
ApplicationId = "MyAppID",
WindowsKey = "MyDotNetKey",
Server = "www.myparseserver.com/errorcollection"
});
try{
ParseConfig config = null;
config = await ParseConfig.GetAsync().ParseClient1;
} catch (Exception ex){
ParseObject MyError = new ParseObject("Error_Log");
MyError["Application"] = "My First App-App2";
MyError["Error"] = ex.message;
await MyError.Save().ParseClient2;
}
Yes - this is all fake code... my point is I want to be able to have multiple ParseClient instances in one app.
Now... I can simply write a routine that writes out errors that resets the ParseClient.Initialization to the error parse server instance and then redo it back to the original (primary app data) instance when it's done... but that is just asking for trouble in a multi threaded environment and will cause conflicts if some other thread in the app goes to write out parse data right at the moment the error method resets the init.
If ParseClient were IDisposable I could probably do that using :
ParseClient ParseErrorServer = new ParseClient();
ParseErrorServer.ApplicationId = "hmmm";
ParseErrorServer.WindwosKey= "hmmm";
ParseErrorServer.Server= "www.hmmm.com/errorcollection";
using ParseErrorServer {
//Do The Work
}
Is that clear as mud yet? ;P
Without alteration I believe none of the Parse SDKs have the ability to initialise multiple instances.
In the iOS SDK for example, is possible to make a new instance (say with a different server url) upon restarting the app but you cannot have multiple. There has also been discussion on the iOS SDK about being able to change the configuration without restart but no one has implemented this yet.
We would happily review a PR for this, however it would require a major and complex overhaul as you would have to manage cache, users etc across multiple instances.
Is it possible to make asyncs calls, like it was in Crm 4
crmService.UpdateAsync(card, Guid.NewGuid());
in CRM 2011???
I have to do synchronization between CRM and some system with the help of SSIS.
In destination script component I'd like to use Async calls, but I don't want to write async calls on my own.
Thank you!!!
I've mainly worked with CRM 2011 (as opposed to CRM 4) but it sounds like you're about to deploy a plugin. If not, stop reading now. :)
If you are, you can set the type of call to asynchronous while registering your plugin in the PRT. Just click for the option.
You've also got another option. If you're running the newest .NET framework, there's a new keyword - async that executes the method asynchronously. And if you're targeting an older .NET version, don't despair - I used threads for a lengthy update and that worked out quite well too.
It could be answer for this question, but it's possible only in CRM 2011 UR 12
#region Execute Multiple with Results
// Create an ExecuteMultipleRequest object.
requestWithResults = new ExecuteMultipleRequest()
{
// Assign settings that define execution behavior: continue on error, return responses.
Settings = new ExecuteMultipleSettings()
{
ContinueOnError = false,
ReturnResponses = true
},
// Create an empty organization request collection.
Requests = new OrganizationRequestCollection()
};
// Create several (local, in memory) entities in a collection.
EntityCollection input = GetCollectionOfEntitiesToCreate();
// Add a CreateRequest for each entity to the request collection.
foreach (var entity in input.Entities)
{
CreateRequest createRequest = new CreateRequest { Target = entity };
requestWithResults.Requests.Add(createRequest);
}
// Execute all the requests in the request collection using a single web method call.
ExecuteMultipleResponse responseWithResults =
(ExecuteMultipleResponse)_serviceProxy.Execute(requestWithResults);
// Display the results returned in the responses.
foreach (var responseItem in responseWithResults.Responses)
{
// A valid response.
if (responseItem.Response != null)
DisplayResponse(requestWithResults.Requests[responseItem.RequestIndex], responseItem.Response);
// An error has occurred.
else if (responseItem.Fault != null)
DisplayFault(requestWithResults.Requests[responseItem.RequestIndex],
responseItem.RequestIndex, responseItem.Fault);
}
Code from MSDN