Develop Reference

.NET Core API seems to deadlock

We have a .NET Core API running in production which can run stable for days or even weeks and then suddenly freezes. Such a freeze can even happen multiple times a day, completely random. What happens: the code seems to be frozen and doesn't accept any new requests. No new requests are logged, the thread count rises sky-high and the memory rises steadily until it's maxed out.
I created a memory dump to analyze. It tells me that most threads are waiting for a lock to be released at a specific function, looking like a deadlock. I analysed this function and cannot see why this would cause issues. Can someone help me out? Obviously I suspect AsParallel() to be thread unsafe, but the internet says no, it is thread safe.
public async Task<bool> TryStorePropertiesAsync(string sessionId, Dictionary<string, string> keyValuePairs, int ttl = 1500)
{
try
{
await Task.WhenAll(keyValuePairs.AsParallel().Select(async item =>
{
var doc = await _cosmosDbRepository.GetItemByKeyAsync(GetId(sessionId, item.Key), sessionId) ?? new Document();
doc.SetPropertyValue("_partitionKey", sessionId);
doc.SetPropertyValue("key", GetId(sessionId, item.Key));
doc.SetPropertyValue("name", item.Key.ToLowerInvariant());
doc.SetPropertyValue("value", item.Value);
doc.TimeToLive = ttl;
await _cosmosDbRepository.UpsertDocumentAsync(doc, "_partitionKey");
}));
return true;
}
catch
{
ApplicationInsightsLogger.TrackException(ex, new Dictionary<string, string>
{
{ "sessionID", sessionId },
{ "action", "TryStoreItems" }
});
return false;
}
}

The code has serious issues. For eg 100 items, it fires off 100 concurrent operations, 4/8 at a time. The code inside the loop seems to read a document from CosmosDB, set all its properties then call a method named similar to DocumentClient.UpsertDocumentAsync which doesn't need pre-loading anything. Without knowing what _cosmosDbRepository is and what its methods do, one can only guess. It's possible it creates extra conflicts though by trying to lock stuff while the (probably useless) load/update cycle takes place.
For starters, AsParallel() is only meant for data parallelism: partition some data in memory and use as many workers are there are cores to crunch each partition. There's no data here though, just calls to async operations. That's why for 100 items, this code will fire off 100 concurrent tasks.
That could hit any number of CosmosDB throttling limits, even if it doesn't cause concurrency conflicts. It could also lead to networking issues, as the same cable is used for all those concurrent connections.
Not taking CosmosDB into account, the correct way to make lots of calls to a remote service is to queue them and execute them with a limited number of workers. This is very easy to do with .NET's ActionBlock. The code could change to something like this :
class Payload
{
public string SessionKey{get;set;}
public string Key{get;set;}
public string Name{get;set;}
public string Value{get;set;}
public int TTL{get;set;}
}
//Allow only 10 concurrent upserts
var options=new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = 10
};
var upsertBlock=new ActionBlock<Payload>(myPosterAsync,options);
foreach(var payload in payloads)
{
block.Post(pair);
}
//Tell the block we're done
block.Complete();
//Await for all queued operations to complete
await block.Completion;
Where myPosterAsync contains the posting code :
async Task myPosterAsync(Payload item)
{
try
{
var doc = await _cosmosDbRepository.GetItemByKeyAsync(GetId(item.SessionId, item.Key),
item.SessionId)
?? new Document();
doc.SetPropertyValue("_partitionKey", item.SessionId);
doc.SetPropertyValue("key", GetId(sessionId, item.Key));
doc.SetPropertyValue("name", item.Name);
doc.SetPropertyValue("value", item.Value);
doc.TimeToLive = item.TTL;
await _cosmosDbRepository.UpsertDocumentAsync(doc, "_partitionKey");
catch (Exception ex)
{
//Handle the error in some way, eg log it
ApplicationInsightsLogger.TrackException(ex, new Dictionary<string, string>
{
{ "sessionID", item.SessionId },
{ "action", "TryStoreItems" }
});
}
}

Sending 5000 messages in async way in C#

I have to send 10000 messages. At the moment, it happens synchronously and takes up to 20 minutes to send them all.
// sending messages in a sync way
foreach (var message in messages)
{
var result = Send(message);
_logger.Info($"Successfully sent {message.Title}.")
}
To shorten the message sending time, I'd like to use async and await, but my concern is if C# runtime can handle 15000 number of tasks in the worker process.
var tasks = new List<Task>();
foreach (var message in messages)
{
tasks.Add(Task.Run(() => Send(message))
}
var t = Task.WhenAll(tasks);
t.Wait();
...
Also, in terms of memory, I'm not sure if it's a good idea to create a list of 15000 tasks

Since I came home from work, I have played with this a bit and here is my answer.
First of all Parallel.ForEach is bretty cool to use, and with my 8 core runs very fast.
I suggest to limit the CPU usage so you do not use 100% capacity, but that depends on your system, I have made two suggestion for it.
The other things is you need to monitor and be sure that your sender server can eat all these jobs with out getting trouble.
Here is a the implementation:
public void MessMessageSender(List<Message> messages)
{
try
{
var parallelOptions = new ParallelOptions();
_cancelToken = new CancellationTokenSource();
parallelOptions.CancellationToken = _cancelToken.Token;
var maxProc = System.Environment.ProcessorCount;
// this option use around 75% core capacity
parallelOptions.MaxDegreeOfParallelism = Convert.ToInt32(Math.Ceiling(maxProc * 0.75));
// the following option use all cores expect 1
//parallelOptions.MaxDegreeOfParallelism = maxProc - 1;
try
{
Parallel.ForEach(messages, parallelOptions, message =>
{
try
{
Send(message);
//_logger.Info($"Successfully sent {text.Title}.");
}
catch (Exception ex)
{
//_logger.Error($"Something went wrong {ex}.");
}
});
}
catch (OperationCanceledException e)
{
//User has cancelled this request.
}
}
finally
{
//What ever dispose of clients;
}
}
My answer is inspired for this page.
Documentation:
Parallel.Foreach
Environment.ProcessorCount

Force Cancel Task with API that might hang

I am currently working with a Serial Port, and the API I use will some times hang on a read, even when its own time out is set.
This is not a big problem, but i need to do some work when that happens and the hanging thread needs to be shutdown. I have tried that with the following, but it has been giving me problems as the API call is not terminated, but allowed to continue while the rest of the code continues, and the TimeoutException was thrown. How can i use Tasks to be able to cancel a hanging task after a certain amount of time?
CancellationToken token = new CancellationToken();
var task = Task.Factory.StartNew(() =>
{
CallingAPIThatMightHang(); // Example
}, token);
if (!task.Wait(this.TimeToTimeOut, token))
{
throw new TimeoutException("The operation timed out");
}

CancellationToken is of the form of cooperative cancellation. You need to moderate the token while executing your operation and watch if a cancelation has been requested.
From your code block, it seems as you have one long running synchronous operation which you offload to a threadpool thread. If that's the case, see if you can separate that serial call to chunks, where you can poll the token after read chunk. If you can't, cancellation wont be possible.
Note that in order to request cancellation, you'll have to create a CancellationTokenSource, which you'll later be able to call it's Cancel() method.
As a side note, serial port is async IO, You can use naturally async API's instead of offloading a synchronous to a threadpool thread.
Edit:
#HansPassant gave a better idea. Run the third party call inside another process, one which you keep a reference to. Once you need to terminate it, kill the process.
For example:
void Main()
{
SomeMethodThatDoesStuff();
}
void SomeMethodThatDoesStuff()
{
// Do actual stuff
}
And then launch it in a separate process:
private Process processThatDoesStuff;
void Main()
{
processThatDoesStuff = Process.Start(#"SomeLocation");
// Do your checks here.
if (someCondition == null)
{
processThatDoesStuff.Kill();
}
}
If you need to communicate any result between these two processes, you can do those via several mechanisms. One would be writing and reading the Standard Output of the process.

I am sadly not able to use any other framework, and i am not able to just change the API i am calling so it can use a Cancellation Token.
This is how i chose to solve the problem.
class Program
{
static void Main(string[] args)
{
try
{
var result = TestThreadTimeOut();
Console.WriteLine("Result: " + result);
}
catch (TimeoutException exp)
{
Console.WriteLine("Time out");
}
catch (Exception exp)
{
Console.WriteLine("Other error! " + exp.Message);
}
Console.WriteLine("Done!");
Console.ReadLine();
}
public static string TestThreadTimeOut()
{
string result = null;
Thread t = new Thread(() =>
{
while (true)
{
Console.WriteLine("Blah Blah Blah");
}
});
t.Start();
DateTime end = DateTime.Now + new TimeSpan(0, 0, 0, 0, 1500);
while (DateTime.Now <= end)
{
if (result != null)
{
break;
}
Thread.Sleep(50);
}
if (result == null)
{
try
{
t.Abort();
}
catch (ThreadAbortException)
{
// Fine
}
throw new TimeoutException();
}
return result;
}
}

Concurrently running incorrectly, provide more channels than expected

I have a application, which to make concurrently task running. Here we set
MaxDegreeOfParallelism=4, which means at any time at most 4 tasks running concurrently. In this case, I only have 4 channels available. Otherwise an exception
Could not get Channel
will be thrown.
Each task will have an instance of OutboundDial, so at most it will be 4 instances.
public class OutboundDial
{
private ChannelResource m_ChannelResource;
private VoiceResource m_VoiceResource;
private TelephonyServer m_TelephonyServer;
private AppointmentReminderResult m_Result = new AppointmentReminderResult();
public OutboundDial(TelephonyServer telephonyServer)
{
m_TelephonyServer = telephonyServer;
}
internal void RunScript(AppointmentReminder callData)
{
try
{
try
{
m_ChannelResource = m_TelephonyServer.GetChannel();
m_VoiceResource = m_ChannelResource.VoiceResource;
}
catch (Exception ex)
{
Console.WriteLine("Could not get channel: {0}",ex.StackTrace);
return;
}
// a long running process of I/O bound operation
The producer-consumer queue is
public static BufferBlock<AppointmentReminder> m_Queue =
new BufferBlock<AppointmentReminder>(new ExecutionDataflowBlockOptions() { MaxDegreeOfParallelism = 4});
BufferBlock is a TPL class. The TelephontServer was initialized at the very beginning.
public static TelephonyServer ts;
ts = new TelephonyServer(sIpaddress, "username", "password");
In the consumer part, we have:
static async Task Consumer()
{
try
{
while (await m_Queue.OutputAvailableAsync())
{
m_Queue.TryReceive(4, ts); // MaxDegreeOfParallelism = 4
}
}
TryReceive is an extension method.
public static void TryReceive<T>(this BufferBlock<T> bufferBlock, int count, TelephonyServer ts) where T : AppointmentReminder
{
try
{
for (var i = 0; i < count; i++)
{
T item;
if (bufferBlock.TryReceive(out item))
{
Task t = Task.Run(() =>
{
OutboundDial d = new OutboundDial(ts);
d.RunScript<T>((T)item);
});
}
else
{
break;
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.StackTrace);
}
}
My question: I added 10 items to the queue in the producer part and I set a breaking point at the constructor.
I found the code run 10 times in the constructor then 10 times RunScript, which indicated 10 tasks run together rather than 4. But I only want 4(MaxDegreeOfParallelism). Therefore I did't have enough channels available, an exception was thrown.
Why in my extension method the concurrent running was not working?

BufferBlock doesn't really execute anything, so it doesn't make sense to specify its MaxDegreeOfParallelism. It works only because ExecutionDataflowBlockOptions inherits from DataflowBlockOptions, which is what the BufferBlock constructor expects.
Your Consumer() does this: take up to 4 items and execute them, take up to 4 items and execute them, take up to 4 items and execute them, etc. Since your never wait for those executions to complete, you're not actually limiting the degree of parallelism this way.
If you want to limit the degree of parallelism to 4, you could use ActionBlock, instead of your combination of BufferBlock and Consumer():
new ActionBlock<AppointmentReminder>(
reminder => new OutboundDial(ts).RunScript(reminder),
new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 4});
What this does is to execute the lambda for each reminder, but at most 4 at the same time, which seems to be what you're asking for. Though I'm not sure it's what you need, since you don't seem to release (dispose) the channel after usage, to be used for the next reminder.

It is not clear where the problem is, but it seems that it is not that the cinstructor is called after the method.
Iwould advise you to change the constructor code to:
public OutboundDial(TelephonyServer telephonyServer)
{
m_TelephonyServer = telephonyServer;
Console.WriteLine(m_Telephonyserver);
}
And then you will see for sure that the constructor is complete.
Also, add some Console.WriteLine with useful informatino after each line in RunScript - then you will see where the error is coming from.

Logging exceptions for each item in Parallel.ForEach and Task.Factory.StartNew from it

I am trying to use Parallel.ForEach on a list and for each item in the list, trying to make a database call. I am trying to log each item with or without error. Just wanted to check with experts here If I am doing thinsg right way. For this example, I am simulating the I/O using the File access instead of database access.
static ConcurrentQueue<IdAndErrorMessage> queue = new ConcurrentQueue<IdAndErrorMessage>();
private static void RunParallelForEach()
{
List<int> list = Enumerable.Range(1, 5).ToList<int>();
Console.WriteLine("Start....");
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
Parallel.ForEach(list, (tempId) =>
{
string errorMessage = string.Empty;
try
{
ComputeBoundOperationTest(tempId);
try
{
Task[] task = new Task[1]
{
Task.Factory.StartNew(() => this.contentFactory.ContentFileUpdate(content, fileId))
};
}
catch (Exception ex)
{
this.tableContentFileConversionInfoQueue.Enqueue(new ContentFileConversionInfo(fileId, ex.ToString()));
}
}
catch (Exception ex)
{
errorMessage = ex.ToString();
}
if (queue.SingleOrDefault((IdAndErrorMessageObj) => IdAndErrorMessageObj.Id == tempId) == null)
{
queue.Enqueue(new IdAndErrorMessage(tempId, errorMessage));
}
}
);
Console.WriteLine("Stop....");
Console.WriteLine("Total milliseconds :- " + stopWatch.ElapsedMilliseconds.ToString());
}
Below are the helper methods :-
private static byte[] FileAccess(int id)
{
if (id == 5)
{
throw new ApplicationException("This is some file access exception");
}
return File.ReadAllBytes(Directory.GetFiles(Environment.SystemDirectory).First());
//return File.ReadAllBytes("Files/" + fileName + ".docx");
}
private static void ComputeBoundOperationTest(int tempId)
{
//Console.WriteLine("Compute-bound operation started for :- " + tempId.ToString());
if (tempId == 4)
{
throw new ApplicationException("Error thrown for id = 4 from compute-bound operation");
}
Thread.Sleep(20);
}
private static void EnumerateQueue(ConcurrentQueue<IdAndErrorMessage> queue)
{
Console.WriteLine("Enumerating the queue items :- ");
foreach (var item in queue)
{
Console.WriteLine(item.Id.ToString() + (!string.IsNullOrWhiteSpace(item.ErrorMessage) ? item.ErrorMessage : "No error"));
}
}

There is no reason to do this:
/*Below task is I/O bound - so do this Async.*/
Task[] task = new Task[1]
{
Task.Factory.StartNew(() => FileAccess(tempId))
};
Task.WaitAll(task);
By scheduling this in a separate task, and then immediately waiting on it, you're just tying up more threads. You're better off leaving this as:
/*Below task is I/O bound - but just call it.*/
FileAccess(tempId);
That being said, given that you're making a logged value (exception or success) for every item, you might want to consider writing this into a method and then just calling the entire thing as a PLINQ query.
For example, if you write this into a method that handles the try/catch (with no threading), and returns the "logged string", ie:
string ProcessItem(int id) { // ...
You could write the entire operation as:
var results = theIDs.AsParallel().Select(id => ProcessItem(id));

You might want to remove Console.WriteLine from thread code. Reason being there can be only one console per windows app. So if two or more threads going to write parallel to console, one has to wait.
In replacement to your custom error queue you might want to see .NET 4's Aggregate Exception and catch that and process exceptions accordingly. The InnerExceptions propery will give you the necessary list of exceptions. More here
And a general code review comment, don't use magic numbers like 4 in if (tempId == 4) Instead have some const defined which tells what 4 stands for. e.g. if (tempId == Error.FileMissing)

Parallel.ForEach runs an action/func concurrently up to a certain number of simultaneous instances. If what each of those iterations is doing is not inherently independent on one another, you're not getting any performance gains. And, likely are reducing performance by introducing expensive context switching and contention. You say that you want to do a "database call" and simulating it with a file operation. If each iteration uses the same resource (same row in a database table, for example; or try to write to the same file in the same location) then they're not really going to be run in parallel. only one will be running at a time, the others will simply be "waiting" to get a hold of the resource--needlessly making your code complex.
You haven't detailed what you want to do for each iteration; but when I've encountered situations like this with other programmers, they almost always aren't really doing things in parallel and they've simply gone through and replaced foreachs with Parallel.ForEach in the hopes of magically gaining performance or magically making use of multi-CPU/Core processors.