Develop Reference

Is this the proper way to reuse connections to an Azure Cosmos instance from a Web Job?

I have a web job in my Azure web app that writes data to an Azure Cosmos instance. This web is triggered from a storage queue. Each trigger spawns a new process to do one insert or one update to the Cosmos instance. With the amount of data coming into that queue, the web job inserts/updates the Azure Cosmos instance around 1000 times every minute.
In a separate, user-facing portal, the users query data from this Azure Cosmos instance. We have been getting a high number of these errors from that public-facing portal:
Only one usage of each socket address (protocol/network address/port) is normally permitted <>
An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full
To me, this is indicative of SNAT port exhaustion. All documentation and help information on this subject, and on these specific error messages point to "ensuring that we are re-using connections to the Cosmos instance", and that we are using best practices. I "believe" we are re-using connections to the Azure Cosmos instance properly, but I am not sure. This is the code:
Program.cs
using Microsoft.Extensions.Hosting;
internal class Program
{
private static async Task Main(string[] args)
{
var builder = new HostBuilder();
builder.ConfigureWebJobs(b =>
{
b.AddAzureStorageQueues();
});
var host = builder.Build();
using (host)
{
await host.RunAsync();
}
}
}
Functions.cs
namespace WebhookMessageProcessor
{
public class RingCentralMessageProcessor
{
private static List<KeyValuePair<string, CosmosClient>> cosmosClients = new List<KeyValuePair<string, CosmosClient>>();
public async static void ProcessQueueMessage([QueueTrigger("<<storage-queue-name>>")] string message, ILogger logger)
{
var model = Newtonsoft.Json.JsonConvert.DeserializeObject<WebHookHandlerModel>(message);
//the intention here is to maintain a list of cosmos clients, as each message from the queue indicates which Cosmos instance to update/insert the data to. For now, however, all messages are going to a single instance. More will be added later.
if (cosmosClients == null) cosmosClients = new List<KeyValuePair<string, CosmosClient>>();
await HandleCallData(model.ownerId, model.body, storageConnectionString);
}
public async static Task HandleCallData(string ownerId, string deserializedData, string storageConnectionString)
{
var model = Newtonsoft.Json.JsonConvert.DeserializeObject<PushModel>(deserializedData);
if (model == null || model.body == null || model.body.sessionId == null)
{
//log error
}
else
{
//the intention here is to maintain a list of cosmos clients, as each message from the queue indicates which Cosmos instance to update/insert the data to. For now, however, all messages are going to a single instance. More will be added later.
var cosmosClient = null;
if (!cosmosClients.Any(x => x.Key == ownerId))
{
cosmosClient = new CosmosClient(cosmosConfig.accountEndpoint, cosmosConfig.accountKey);
cosmosClients.Add(new KeyValuePair<string, CosmosClient>(ownerId, cosmosClient));
}
else
{
cosmosClient = cosmosClients.First(x => x.Key == ownerId).Value;
}
//data building logic here
//...
var cosmosContainer = cosmosClient.GetContainer(cosmosConfig.databaseId, cosmosConfig.containerId);
string etag = null;
if (condition1) // THEN INSERT
{
var task = await cosmosContainer.CreateItemAsync(call, partitionKey: new PartitionKey(partitionKey), requestOptions: new ItemRequestOptions() { IfMatchEtag = etag });
success = true;
}
else if (condition2) // THEN FIND AND REPLACE
{
var response = await cosmosContainer.ReadItemAsync<CallIndex>(call.id, new PartitionKey(partitionKey));
var existingCallIndex = response.Resource;
etag = response.ETag;
await cosmosContainer.ReplaceItemAsync(existingCallIndex, call.id, new PartitionKey(partitionKey), new ItemRequestOptions() { IfMatchEtag = etag });
success = true;
}
else // FIND AND REPLACE BY DEFAULT
{
var response = await cosmosContainer.ReadItemAsync<CallIndex>(call.id, new PartitionKey(partitionKey));
var existingCallIndex = response.Resource;
etag = response.ETag;
await cosmosContainer.ReplaceItemAsync(existingCallIndex, call.id, new PartitionKey(partitionKey), new ItemRequestOptions() { IfMatchEtag = etag });
success = true;
}
}
catch (Exception ex)
{
//handle exception here
}
curTries++;
} while (!success && curTries < maxTries);
}
}
}
}
I am maintaining a list of cosmos clients in a static variable, as the content of the message may indicate writing to a different cosmos instance. However, as of now, there is only one instance, and all data is going to that single instance. There will be more instances in the future. Is this a good/correct way to reuse connections to the Cosmos instance in my web job?
Thanks

This can be technically achieved but there are trade-offs you need to make. Mainly latency (you can't have an unbounded list of Cosmos Clients on Direct mode).
The key of Dictionary should be the account name, that way you don't end up creating multiple clients for the same account even if the "owner" is different. There should be a Singleton client per account your application interacts with.
You should put your client on Gateway mode. This should use less ports, have higher potential latency, but there is no scenario where you can have an unbounded number of client instances on Direct mode, that simply will almost always hit your connection limit. Example on how to change the connection mode.
You are using a List, that is neither concurrent nor handles eviction. You should dispose clients that are not used after some time or define a max number of clients you can handle, it's impossible to write an app that handles an unbounded/infinite number of clients. Maybe MemoryCache is a good option. But you need to define a limit or make sure you can distribute across multiple machines/instances.

Putting Cosmos clients in a List will never work as you can't pool connections for different clients pointing at different accounts. Your single client instance here is likely hitting the 128 port max for your WebJob. For Cosmos you should use a single client per instance. You should also cache the container references too. Not doing this will cause 429s on the master partition (stores all your account meta data) in Cosmos DB due to all the meta data requests that will happen at larger request volumes.
Take a look at this article here on Singleton client, container reference caching and PortReuseMode
Best Practices for .NET SDK
Also see here for Networking Performance Tips for .NET SDK v3

"SqlConnection does not support parallel transactions" in Azure Webjob

I don't use transactions in my C# .NET Core v3.1 with EFCore v3 code explicitly and all works fine.
Except for my Azure Webjob. It listens to a queue. When multiple messages are on the queue and thus the function gets called multiple times in parallel I get transaction errors.
My webjob reads a file from the storage and saves the content to a database table.
I also use the Sharding mechanism: each client has its own database.
I tried using TransactionScope but then I get other errors.
Examples I found use the TransactionScope and opening the connection and doing the saving in one method. I have those parts split into several methods making it unclear to me how to use the TransactionScope.
Here's some code:
ImportDataService.cs:
//using var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled);
await using var tenantContext = await _tenantFactory.GetContextAsync(clientId, true);
await tenantContext.Foo.AddRangeAsync(dboList, cancellationToken);
await tenantContext.SaveChangesAsync(cancellationToken);
//scope.Complete();
TenantFactory.cs:
public async Task<TenantContext> GetContextAsync(int tenantId, bool lazyLoading = false)
{
_tenantConnection = await _sharding.GetTenantConnectionAsync(tenantId);
var optionsBuilder = new DbContextOptionsBuilder<TenantContext>();
optionsBuilder.UseLoggerFactory(_loggerFactory);
if (lazyLoading) optionsBuilder.UseLazyLoadingProxies();
optionsBuilder.UseSqlServer(_tenantConnection,
options => options.MinBatchSize(5).CommandTimeout(60 * 60));
return new TenantContext(optionsBuilder.Options);
}
This code results in SqlConnection does not support parallel transactions.
When enabling TransactionScope I get this error: This platform does not support distributed transactions.
In my ConfigureServices I have
services.AddSingleton<IImportDataService, ImportDataService>();
services.AddTransient <ITenantFactory, TenantFactory>();
services.AddTransient <IShardingService, ShardingService>();
I also tried AddScoped but no change.
Edit: Additional code
ShardingService.cs
public async Task<SqlConnection> GetTenantConnectionAsync(int tenantId)
{
SqlConnection tenantConnection;
try
{
tenantConnection = await _clientShardMap.OpenConnectionForKeyAsync(tenantId, _tenantConnectionString, ConnectionOptions.Validate);
}
catch (Exception e)
{
_logger.LogDebug($"Error getting tenant connection for key {tenantId}. Error: " + e.Message);
throw;
}
if (tenantConnection == null) throw new ApplicationException($"Cannot get tenant connection for key {tenantId}");
return tenantConnection;
}
When the WebJob gets triggered it reads a record from a table. The ID of the record is in the queue message. Before processing the data it first changes the status to processing and when the data is processed it changes the status to processed or error:
var fileImport = await _masterContext.FileImports.FindAsync(fileId);
fileImport.Status = Status.Processing;
await _masterContext.SaveChangesAsync();
if (await _fileImportService.ProcessImportFile(fileImport))
fileImport.Status = Status.Processed;
await _masterContext.SaveChangesAsync();

SqlConnection.OpenAsync issue

I am faced with a peculiar async problem which I can reproduce easily but cannot understand.
My Current Setup
I have a WCF Service which exposes two API's - API1 and API2. Both the service contracts are synchronous. API1, looks up a dictionary in memory, then creates a task using Task.Factory.StartNew to create a new task which fetches data from a SQL server, compares it with the data from the dictionary and writes some logs. In case the SQl Server has connectivity issues, this re-tries SqlConnection.OpenAsync 3 more times. Note that the API call itself returns as soon as it has the data from the dictionary (does not wait for SQl operation to complete)
API2 is much simpler, it just calls a stored procedure on SQL server, gets the data and returns.
The code to open connection is as follows:
public static int OpenSqlConn(SqlConnection connection)
{
return OpenSqlConn(connection).Result;
}
public async static Task<int> OpenSqlConnAsync(SqlConnection connection)
{
return await OpenConnAsync(connection);
}
private static async Task<int> OpenConnAsync(SqlConnection connection)
{
int retryCounter = 0;
TimeSpan? waitTime = null;
while (true)
{
if (waitTime.HasValue)
{
await Task.Delay(waitTime.Value).ConfigureAwait(false);
}
try
{
startTime = DateTime.UtcNow;
await connection.OpenAsync().ConfigureAwait(false);
break;
}
catch (Exception e)
{
if (retryCounter >= 3)
{
SafeCloseConnection(connection);
return retryCounter;
}
retryCounter++;
waitTime = TimeSpan.FromSeconds(6);
}
}
return retryCounter;
}
The API1 code looks like below:
public API1Response API1 (API1Request request)
{
// look up in memory dictionary for the request
API1Response response = getDataFromDictionary(request);
// create a task to get some data from DB
Action action = () =>
{
GetDataFromDb(request);
}
Task.Factory.StartNew(action).ConfigureAwait(false);
// this is called immediately even if DB is not available and above task is retrying.
return API1Response;
}
public void GetDataFromDb(API1Request request)
{
using (var connection = new SqlConnection(...))
{
OpenSqlConn(connection);
/// hangs for long even if db is available
ReadDataFromDb(connection);
}
}
public API2Response API2(API2REquest request)
{
return GetDataFromDbForAPI2(request)
}
public API2Response GetDataFromDbForAPI2(API2Request request)
{
using (var connection = new SqlConnection(...))
{
OpenSqlConn(connection); /// hangs for long even if db is available
ReadDataFromDb(connection);
}
}
The Problem
The service runs into the following problem when the SQL Server is unavailable even for short periods of time, and some client makes just 100 calls to API1:
When my SQL server has connectivity issues, and I get around 100 calls of API1, even though API1 returns to the caller, it has created 100 tasks that will try to open a connection to the bad DB. Each of those tasks hangs in a retry look for some time (which is expected). In my experiments, I can simulate a DB unavailability by using a bad connection string for API1.
Now let's say the DB is back up again and a call to API2 is made to the service. What I find is that when API2 call reaches the OpenAsync portion above, it hangs. Just hangs :(
Some observations
1. When I look at the 'Parallel Stacks' from Visual Studio, I find that there are 100 threads with the API1 stack doing the following stack :
ManualResetEvenSlim.Wait()
Task.SpinThenBlockingWait
Task.InternalWait();
Task<>.GetREsultCore
OpenConn()
There is 1 thread with the API2 stack, which again is in a similar stack as above.
However, if I replace SqlConnection.OpenAsync with SqlConnection.Open(), API2 call returns immediately.
Need Help
What I would like to understand is why does the API2, which can open a DB connection (because DB is available at that time), also hang on OpenAsync. Is there any obvious synchronization issue that I am seeing? When i change SqlConnection.OpenAsync() to SqlConnection.Open() why does the behavior change?

If another request is running a method wait until finish

I'm developing an ASP.NET Web API application with C#, .NET Framework 4.7 and MongoDb.
I have this method:
[HttpPut]
[Route("api/Public/SendCommissioning/{serial}/{withChildren}")]
public HttpResponseMessage SendCommissioning(string serial, bool withChildren)
{
string errorMsg = "Cannot set commissioning.";
HttpResponseMessage response = null;
bool serverFound = true;
try
{
[...]
// Mongo
MongoHelper mgHelper = new MongoHelper();
mgHelper.InsertCommissioning(serial, withChildren);
}
catch (Exception ex)
{
_log.Error(ex.Message);
response = Request.CreateResponse(HttpStatusCode.InternalServerError);
response.ReasonPhrase = errorMsg;
}
return response;
}
Sometimes this method is called very quickly and I get an error here:
// Mongo
MongoHelper mgHelper = new MongoHelper();
mgHelper.InsertCommissioning(serial, withChildren);
Here I'm inserting the serials I received in order, and sometimes I get an error with a duplicated key in MongoDb:
I have a method to get the latest id used in Mongo (the primary key). And two requests get the same id, so when I try to insert it on Mongo I get an invalid key exception.
I thought to use a queue to store the serials and then consume them in the same order that I have received them. But I think I will get the same error when I try to store the serial in MongoDb.
Maybe if I can set a method that if it is running, I have to wait to run it, it will works. This method will have the part of insert the serials into Mongo.
How can I do that? A method that if it is running you can't run it in another Web Api request.
Or, do you know a better option?
By the way, I can't block this method. Maybe I need to run a thread with this synchronized part.

Transactional operations simultaneously mixed with non-transactional ones

I need to perform data import from external source to my database. Because there is a lot of data to download, the import is executing for a long time and I need to persist periodic updates about current importing state to the database (for the user to follow).
Suppose I have 2 tables: Import (storage for imported data) and Status (importing state monitoring table).
The code for data import:
public class Importer
{
public delegate void ImportHandler(string item);
public event ImportHandler ImportStarted;
public void OnStart(string item)
{
ImportStarted(item);
}
public void Execute(string[] items)
{
foreach (var item in items)
{
OnStart(item);
PersistImportedData(Download(item));
}
}
private void PersistImportedData(object data)
{
using (var connection = new SqlConnection()){ /*saving imported data*/ }
}
}
The starter code - for invoking import task and updating its status:
public class Starter
{
public void Process(string[] items)
{
var importer = new Importer();
importer.ImportStarted += UpdateImportState;
importer.Execute(items);
}
private void UpdateImportState(string item)
{
using (var connection = new SqlConnection()){ /*status updates*/ }
}
}
Now everything works fine. Import is executing and user is getting status updates (from Status table) as import goes on.
The problem occurs because such logic is not safe. I have to be sure, that import is an atomic operation. I don't want partially downloaded and saved data. I've used transaction approach as a solution for this (I've wrapped importer.Execute with TransactionScope):
importer.ImportStarted += UpdateImportState;
using (var scope = new TransactionScope())
{
importer.Execute(items);
scope.Complete();
}
Now I have safety - rollback occurs e.g. in case of process abort.
I faced different problem now - the one I want to resolve. I need status updates information for the user to show, but the Status table is not affected by updates, while transaction is not yet completed. Even if I try to use RequiresNew option for creating separate transaction (not ambient one), nothing changes. The Execute function creates its own connection to database and UpdateImportState does the same. The connection is not shared. I don't know why State table cannot be affected, even if TransactionScope covers only logic connected with Import table.
How to preserve consistent import and allow periodic status updates ?

Use TransactionScopeOption.Suppress in UpdateImportState instead of TransactionScopeOption.RequiresNew