Develop Reference

Understanding the await or timeout: How to send multiple emails concurrently without blocking the rest of the program

New to this parallel coding.
I am trying to fire off a list of tasks (in this case, emails to send).
The below code does work, however I am unsure, if ONE email was to fail sending, or that task not finish for whatever reason etc.
Would me code just hang on the await line?
What the solution here, or is it a non problem?
I need to await OR intercept each one when it finishes as I need to (mark the email as sent correctly or retry etc).
However is awaiting all safe, or should I have an event to intercept the response?
(I do not really care if the odd one takes a while, My key is not to block up the rest of the program)
Over all this will be in a 5 mins loop, and I wouldn't want the email in the future to be blocked up is a timeout per task, as option.
Any help would be great, thank You.
// Create a list of emails to send
List<Task<string>> tasks = new List<Task<string>>();
foreach (var item in lstEmailList) // loop the list of emails to send
{
tasks.Add(EmailLib.SendEmailOldSmtpAuth(item)); // add each email as a task
}
// now run the list as parallel tasks
await Task.Run(() => Parallel.ForEach(tasks, s =>
{
//EmailLib.SendEmailOldSmtpAuth(item);
}));
foreach (var task in tasks)
{
var result = ((Task<string>)task).Result;
if (result.Contains("PH_OK") == true)
{
string strFindId = result.Replace("PH_OK=", "").ToString();
int EmailID = Int32.Parse(strFindId);
DbFunc.MarkEmailAsSent(ref SQLConnX, EmailID);
}
}
In case it is helpful:
public async Task<string> SendEmailOldSmtpAuth(Data_PendingEmails objEmail)
{
try
{
var emailMessage = new MimeMessage();
string SendToName = "";
if (objEmail.CarerID > 0)
{
SendToName = objEmail.carForename + " " + objEmail.carSurname;
}
else
{
SendToName = objEmail.cliForename + " " + objEmail.cliSurname;
}
if (SendToName == "")
{
SendToName = "User";
}
emailMessage.From.Add(new MailboxAddress(objEmail.EMailYourName, objEmail.EMailAddress));
emailMessage.To.Add(new MailboxAddress(SendToName, objEmail.ToEmailAddress));
emailMessage.Subject = objEmail.EmailSubject;
emailMessage.Body = new TextPart("html") { Text = objEmail.EmailMessage };
try
{
var client = new SmtpClient();
await client.ConnectAsync(objEmail.SMTPServer, Int32.Parse(objEmail.SMTPPort), SecureSocketOptions.SslOnConnect);
await client.AuthenticateAsync(objEmail.SMTPUserName, objEmail.SMTPPassword);
await client.SendAsync(emailMessage);
await client.DisconnectAsync(true);
return "PH_OK=" + objEmail.EmailID.ToString();
}
catch (Exception ex)
{
var e = ex;
return e.Message;
}
}
catch (Exception SendEmailOldSmtpAuthOverAll)
{
return SendEmailOldSmtpAuthOverAll.Message.ToString();
}
}

I will try to simulate your scenario with a much simpler example but the main idea will be the same. I do it this way to simulate the potential exception.
First af all you need concurrent calls to a web service, so the best way is to use Task.WhenAll because this is an I/O operation (as #Charlieface already mentioned)
Lets say that we have a list of emails:
var emailList = new List<string>() { "1", "2", "3", "4", "5", "6" };
then we need to create an ienumerable of tasks:
var tasks = emailList.Select(async email =>
{
var response = await SendEmailAsync(email);
Console.WriteLine(response);
});
Then we "mock" an exception and append it to task list in order to simulate an exception:
var problematicTask = ThrowExceptionAsync("Error from initial task");
var allTasks = tasks.Append(problematicTask);
Now, in order to fire the tasks and to catch the exceptions we need to do this:
var aggregateTasks = Task.WhenAll(allTasks);
try
{
await aggregateTasks;
}
catch
{
AggregateException aggregateException = aggregateTasks.Exception!;
foreach (var ex in aggregateException.InnerExceptions)
{
Console.WriteLine(ex.Message);
}
}
and as helper methods for this example i created these two arbitrary methods:
async Task<string> SendEmailAsync(string email)
{
await Task.Delay(1500);
if (email.Equals("7") || email.Equals("10"))
{
await ThrowExceptionAsync("Error from sendEmail");
}
return $"Email: {email} sent";
}
async Task ThrowExceptionAsync(string msg)
{
throw new Exception(msg);
}
If you run this simple example you would inspect all exceptions may thrown in each call..
Now in regards to your particular example i think you need to remove all try/catch blocks in order to catch the exceptions as aggregate exception and not blocking your app.

Task does not always return upon completion

So I am trying to build a program to control a machine. Communications with said machine is via a serial port for which I have written a driver. Continuous polling to the machine is necessary for status feedback etc. In my program I have a dedicated ExecutionEngine() class to handle serial send and receive. I also need to have two separate control sequences running, which I have put into methods RunSequenceA() and RunSequenceB() respectively. During normal operation, all three methods need to run until both control sequences finish, at which point the StopSequence() method is called. My issue is that sometimes, for whatever reason, the StopSequence() method is never called, leaving my ExecutionEngine() method in an infinite loop!
Code for ExecutionEngine():
private static void ExecutionEngine()
{
// Clear both lists in case they have old data
_commandList.Clear();
_pollingList.Clear();
// Poll while user has not yet clicked "STOP"
while (!_cTokenSource.Token.IsCancellationRequested)
{
// If there are commands to be sent, send them first
if (_commandList.Count > 0)
{
Command[] tempCommandArray;
lock (_commandList)
tempCommandArray = _commandList.ToArray();
foreach (var c in tempCommandArray)
{
if (_cTokenSource.Token.IsCancellationRequested)
break;
var response = SerialDriver.ComCycle(c.CommandBytes, _serialPort);
var success = CheckErrorReturn(response, false);
if (success)
{
AddPolling(c);
RemoveCommand(c);
}
}
}
// Do polling operation on applicable controllers
if (_pollingList.Count > 0)
{
Command[] tempPollingArray;
lock (_pollingList)
tempPollingArray = _pollingList.ToArray();
foreach (var c in tempPollingArray)
{
if (_cTokenSource.Token.IsCancellationRequested)
break;
var response = SerialDriver.ComCycle(c.PollBytes, _serialPort);
var success = ProcessPollReturn(response);
if (success)
{
c.FlagDone();
RemovePolling(c);
}
}
}
if (_commandList.Count + _pollingList.Count == 0)
{
// Will get stuck here if neither list gets new items added
Console.WriteLine("Bad place");
Thread.Sleep(500);
}
}
// Cancellation has been requested
lock (_commandList)
_commandList.Clear();
lock (_pollingList)
_pollingList.Clear();
ResetTriggers();
var endCommand = new Command("GL_SYSCMD", 0);
SerialDriver.ComCycle(endCommand.CommandBytes, _serialPort);
_serialPort.Close();
_vm.SequenceRunning = false;
return;
}
Code for running sequences:
private static async Task RunSequencesAsync()
{
var taskArray = new Task[2];
var a = new Action(RunSequenceA);
var b = new Action(RunSequenceB);
taskArray[0] = Task.Run(a);
taskArray[1] = Task.Run(b);
await Task.WhenAll(taskArray).ConfigureAwait(continueOnCapturedContext: false);
// Sometimes this never fires, WHY?
UpdateStatus("All done!");
StopSequence();
}
// Run A sequence
internal static void RunSequenceA()
{
if (_sequenceA1 != null && _sequenceA1.Count > 0)
{
foreach (var s in _sequenceA1)
{
if (_cTokenSource.Token.IsCancellationRequested)
return;
s.Execute();
if (s.Reference != null && TriggerStepCompleted != null)
TriggerStepCompleted(s, EventArgs.Empty);
}
// This part always fires
Console.WriteLine("Sequence A finished");
return;
}
else
return;
}
And finally, the methods to start and stop everything:
private static async Task StartSequenceAsync()
{
_serialPort.PortName = _vm.SelectedComPort;
_serialPort.Open();
_serialPort.DiscardInBuffer();
_serialPort.DiscardOutBuffer();
// Start
_cTokenSource = new CancellationTokenSource();
_vm.SequenceRunning = true;
var taskArray = new Task[2];
taskArray[0] = Task.Run(() => ExecutionEngine());
Thread.Sleep(50);
taskArray[1] = Task.Run(() => RunSequencesAsync());
await Task.WhenAll(taskArray).ConfigureAwait(continueOnCapturedContext: false);
}
private static void StopSequence()
{
_cTokenSource.Cancel();
}
To reiterate, the problem doesn't happen every time. In fact, most times the program runs fine. It seems that problems only arise if I manually call the StopSequence() method half way through execution. Then it's 50/50 as to whether the problem shows up. I'm pretty sure my issue is threading related, but not sure exactly what is going wrong. Any help pointing me in the right direction will be greatly appreciated!

Multithreaded c# console app to scrape data from sites

I have written an app that goes through our own properties and scraps the data. To make sure I don't run through the same URLs, I am using a MySQL database to store the URL, flag it once its processed. All this was being done in a single thread and it's fine if I had only few thousand entries. But I have few hundred thousand entries that I need to parse so I need to make changes in the code (I am newbie in multithreading in general). I found an example and was trying to copy the style but doesn't seem to work. Anyone know what the issue is with the following code?
EDIT: Sorry didn't mean to make people guess the issue but was stupid of me to include the exception. Here is the exception
"System.InValidCastException: 'Specified cast is not valid.'"
When I start the process it collects the URLs from the database and then never hits DoWork method
//This will get the entries from the database
List<Mappings> items = bot.GetUrlsToProcess(100);
if (items != null)
{
var tokenSource = new CancellationTokenSource();
var token = tokenSource.Token;
Worker.Done = new Worker.DoneDelegate(WorkerDone);
foreach (var item in items)
{
urls.Add(item.Url);
WaitingTasks.Enqueue(new Task(id => new Worker().DoWork((int)id, item.Url, token), item.Url, token));
}
LaunchTasks();
}
static async void LaunchTasks()
{
// keep checking until we're done
while ((WaitingTasks.Count > 0) || (RunningTasks.Count > 0))
{
// launch tasks when there's room
while ((WaitingTasks.Count > 0) && (RunningTasks.Count < MaxRunningTasks))
{
Task task = WaitingTasks.Dequeue();
lock (RunningTasks) RunningTasks.Add((int)task.AsyncState, task);
task.Start();
}
UpdateConsole();
await Task.Delay(300); // wait before checking again
}
UpdateConsole(); // all done
}
static void UpdateConsole()
{
Console.Write(string.Format("\rwaiting: {0,3:##0} running: {1,3:##0} ", WaitingTasks.Count, RunningTasks.Count));
}
static void WorkerDone(int id)
{
lock (RunningTasks) RunningTasks.Remove(id);
}
public class Worker
{
public delegate void DoneDelegate(int taskId);
public static DoneDelegate Done { private get; set; }
public async void DoWork(object id, string url, CancellationToken token)
{
if (token.IsCancellationRequested) return;
Content obj;
try
{
int tries = 0;
bool IsUrlProcessed = true;
DateTime dtStart = DateTime.Now;
string articleDate = string.Empty;
try
{
ScrapeWeb bot = new ScrapeWeb();
SearchApi searchApi = new SearchApi();
SearchHits searchHits = searchApi.Url(url, 5, 0);
if (searchHits.Hits.Count() == 0)
{
obj = await bot.ReturnArticleObject(url);
if (obj.Code != HttpStatusCode.OK)
{
Console.WriteLine(string.Format("\r Status is {0}", obj.Code));
tries = itemfound.UrlMaxTries + 1;
IsUrlProcessed = false;
itemfound.HttpCode = obj.Code;
}
else
{
string title = obj.Title;
string content = obj.Contents;
string description = obj.Description;
Articles article = new Articles();
article.Site = url.GetSite();
article.Content = content;
article.Title = title;
article.Url = url.ToLower();
article.Description = description;
string strThumbNail = HtmlHelper.GetImageUrl(url, obj.RawResponse);
article.Author = HtmlHelper.GetAuthor(url, obj.RawResponse);
if (!string.IsNullOrEmpty(strThumbNail))
{
//This condition needs to be added to remove ?n=<number> from EP thumbnails
if (strThumbNail.Contains("?"))
{
article.ImageUrl = strThumbNail.Substring(0, strThumbNail.IndexOf("?")).Replace("http:", "https:");
}
else
article.ImageUrl = strThumbNail.Replace("http:", "https:");
}
else
{
article.ImageUrl = string.IsNullOrEmpty(strThumbNail) ? article.Url.GetDefaultImageUrls() : strThumbNail.Replace("http:", "https:");
}
articleDate = HtmlHelper.GetPublishDate(url, obj.RawResponse);
if (string.IsNullOrEmpty(articleDate))
article.Pubdate = DateTime.Now;
else
article.Pubdate = DateTime.Parse(articleDate);
var client = new Index(searchApi);
var result = client.Upsert(article);
itemfound.HttpCode = obj.Code;
if (result)
{
itemfound.DateCreated = DateTime.Parse(articleDate);
itemfound.DateModified = DateTime.Parse(articleDate);
UpdateItem(itemfound);
}
else
{
tries = itemfound.UrlMaxTries + 1;
IsUrlProcessed = false;
itemfound.DateCreated = DateTime.Parse(articleDate);
itemfound.DateModified = DateTime.Parse(articleDate) == null ? DateTime.Now : DateTime.Parse(articleDate);
UpdateItem(itemfound, tries, IsUrlProcessed);
}
}
}
else
{
tries = itemfound.UrlMaxTries + 1;
IsUrlProcessed = true;
itemfound.HttpCode = HttpStatusCode.OK;
itemfound.DateCreated = DateTime.Parse(articleDate);
itemfound.DateModified = DateTime.Parse(articleDate) == null ? DateTime.Now : DateTime.Parse(articleDate);
}
}
catch (Exception e)
{
tries = itemfound.UrlMaxTries + 1;
IsUrlProcessed = false;
itemfound.DateCreated = DateTime.Parse(articleDate);
itemfound.DateModified = DateTime.Parse(articleDate) == null ? DateTime.Now : DateTime.Parse(articleDate);
}
finally
{
DateTime dtEnd = DateTime.Now;
Console.WriteLine(string.Format("\r Total time taken to process items is {0}", (dtEnd - dtStart).TotalSeconds));
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
Done((int)id);
}
}
All this code is based from Best multi-thread approach for multiple web requests this link. Can someone tell me how to get this approach running?

I think the problem is in the way you're creating your tasks:
new Task(id => new Worker().DoWork((int)id, item.Url, token), item.Url, token)
This Task constructor overload expected Action<object> delegate. That means id will be typed as object and you need to cast it back to something useful first.
Parameters
action
Type: System.Action<Object>
The delegate that represents the code to execute in the task.
state
Type: System.Object
An object representing data to be used by the action.
cancellationToken
Type: System.Threading.CancellationToken
-The CancellationToken that that the new task will observe.
You decided to cast it to int by calling (int)id, but you're passing item.Url as the object itself. I can't tell you 100% what the type of Url is but I don't expect Url-named property to be of type int.

Based on what #MarcinJuraszek said I just went back to my code and added an int as I couldn't find another way to resolve it. Here is the change I made
int i=0
foreach (var item in items)
{
urls.Add(item.Url);
WaitingTasks.Enqueue(new Task(id => new Worker().DoWork((string)id, item.Url, token), item.Url, token));
i++;
}

Azure Service Bus Not Detecting Duplicates

I have a process that reads a message from an Azure Service Bus Queue and converts that message to a Video to be Encoded by Azure Media Services. I noticed that if the process is kicked off very quickly in a row, the same video was being encoded right after another. Here is my code that adds the Video to the Queue
public class VideoManager
{
string _connectionString = ConfigurationManager.AppSettings["Microsoft.ServiceBus.ConnectionString"];
string _queueName = ConfigurationManager.AppSettings["ServiceBusQueueName"];
QueueClient _client;
public VideoManager()
{
var conStringBuilder = new ServiceBusConnectionStringBuilder(_connectionString)
{
OperationTimeout = TimeSpan.FromMinutes(120)
};
var messagingFactory = MessagingFactory.CreateFromConnectionString(conStringBuilder.ToString());
_client = messagingFactory.CreateQueueClient(_queueName);
}
public void Approve(Video video)
{
// Set video to approved.
video.ApprovalStatus = ApprovalStatus.Approved;
var message = new BrokeredMessage(new VideoMessage(video, VideoMessage.MessageTypes.Approve, string.Empty));
message.MessageId = video.RowKey;
_client.Send(message);
}
}
And the process that reads from the Queue
class Program
{
static QueueClient client;
static void Main(string[] args)
{
VideoManager videoManager = new VideoManager();
var connectionString = ConfigurationManager.AppSettings["Microsoft.ServiceBus.ConnectionString"];
var conStringBuilder = new ServiceBusConnectionStringBuilder(connectionString)
{
OperationTimeout = TimeSpan.FromMinutes(120)
};
var messagingFactory = MessagingFactory.CreateFromConnectionString(conStringBuilder.ToString());
client = messagingFactory.CreateQueueClient(ConfigurationManager.AppSettings["ServiceBusQueueName"]);
Console.WriteLine("Starting: Broadcast Center Continuous Video Processing Job");
OnMessageOptions options = new OnMessageOptions
{
MaxConcurrentCalls = 25,
AutoComplete = false
};
client.OnMessageAsync(async message =>
{
bool shouldAbandon = false;
try
{
await HandleMessage(message);
}
catch (Exception ex)
{
shouldAbandon = true;
Console.WriteLine(ex.Message);
}
if (shouldAbandon)
{
await message.AbandonAsync();
}
}, options);
while (true) { }
}
async static Task<int> HandleMessage(BrokeredMessage message)
{
VideoMessage videoMessage = message.GetBody<VideoMessage>();
Console.WriteLine(String.Format("Message body: {0}", videoMessage.Video.Title));
Console.WriteLine(String.Format("Message id: {0}", message.MessageId));
VideoProcessingService vp = new VideoProcessingService(videoMessage.Video);
Task task;
switch (videoMessage.MessageType)
{
case VideoMessage.MessageTypes.CreateThumbnail:
task = new Task(() => vp.ProcessThumbnail(videoMessage.TimeStamp));
task.Start();
while (!task.IsCompleted)
{
await Task.Delay(15000);
message.RenewLock();
}
await task;
Console.WriteLine(task.Status.ToString());
Console.WriteLine("Processing Complete");
Console.WriteLine("Awaiting Message");
break;
case VideoMessage.MessageTypes.Approve:
task = new Task(() => vp.Approve());
task.Start();
while (!task.IsCompleted)
{
await Task.Delay(15000);
message.RenewLock();
}
await task;
Console.WriteLine(task.Status.ToString());
Console.WriteLine("Processing Complete");
Console.WriteLine("Awaiting Message");
break;
default:
break;
}
return 0;
}
}
What I see in the Console Window is the following if I kick off the process 3 times in a row
Message id: 76aca19a-0698-449b-bf58-a24876fc4314
Message id: 76aca19a-0698-449b-bf58-a24876fc4314
Message id: 76aca19a-0698-449b-bf58-a24876fc4314
I thought maybe I did not have the settings correct, but they are there
I am really at a loss here, as I would expect this to be fairly out of the box behavior. Does duplicate detection only work if the message has been completed, so I can't use OnMessageAsync()?

The issue is not the completion (as it was in the code), but the fact that you have in essence multiple consumers (25 concurrent callbacks) and it seems like the LockDuration is elapsing faster than the processing takes. As a result of that, message re-appears and re-processed. As a result of that you see the same message ID logged more than once.
Possible solutions are (as I've outlined in a comment above):
Let OnMessage API manage timeout extension for you (example)
Manually renew the lock as you've done using BrokeredMessage.RenewLock

There is a line of code missing from your HandleMessage code.
async static Task<int> HandleMessage(BrokeredMessage message)
{
VideoMessage videoMessage = message.GetBody<VideoMessage>();
message.CompleteAsync(); // This line...
Console.WriteLine(String.Format("Message id: {0}", message.MessageId));
// Processes Message
}
So yes you have to mark the message with either, Complete, Defer etc..
Also see this Answer, also found this which may be useful in how duplicate detection works

Not reach the code as expected

I have a telephony application, in which I want to invoke simultaneous calls,. Each call will occupy a channel or port. So I added all channels to a BlockingCollection. The application is a windows service.
Let's see the code.
public static BlockingCollection<Tuple<ChannelResource, string>> bc = new BlockingCollection<Tuple<ChannelResource, string>>();
public static List<string> list = new List<string>();// then add 100 test items to it.
The main application has the code:
while (true)
{
ThreadEvent.WaitOne(waitingTime, false);
lock (SyncVar)
{
Console.WriteLine("Block begin");
for (int i = 0; i < ports; i++)
{
var firstItem = list.FirstOrDefault();
if (bc.Count >= ports)
bc.CompleteAdding();
else
{
ChannelResource cr = OvrTelephonyServer.GetChannel();
bc.TryAdd(Tuple.Create(cr, firstItem));
list.Remove(firstItem);
}
}
pc.SimultaneousCall();
Console.WriteLine("Blocking end");
if (ThreadState != State.Running) break;
}
Now for the simultaneous call code:
public void SimultaneousCall()
{
Console.WriteLine("There are {0} channels to be processed.", bc.Count);
var workItemBlock = new ActionBlock<Tuple<ChannelResource, string>>(
workItem =>
{
ProcessEachChannel(workItem);
});
foreach (var workItem in bc.GetConsumingEnumerable())
{
bool result = workItemBlock.SendAsync(workItem).Result;
}
workItemBlock.Complete();
}
private void ProcessEachChannel(Tuple<ChannelResource, string> workItem)
{
ChannelResource cr = workItem.Item1;
string sipuri = workItem.Item2;
VoiceResource vr = workItem.Item1.VoiceResource;
workItem.Item1.Disconnected += new Disconnected(workItemItem1_Disconnected);
bool success = false;
try
{
Console.WriteLine("Working on {0}", sipuri);
DialResult dr = new DialResult();
// blah blah for calling....
}
catch (Exception ex)
{
Console.WriteLine("Exception: {0}", ex.Message);
}
finally
{
if (cr != null && cr.VoiceResource != null)
{
cr.Disconnect();
cr.Dispose();
cr = null;
Console.WriteLine("Release channel for item {0}.", sipuri);
}
}
}
The question was when I tested the application with 4 ports, I thought the code should reach at
Console.WriteLine("Blocking end");
However it was not. Please see the snapshot.
The application is just hanging on after releasing the last channel. I guess that I may use the blockingcollection incorrectly. Thanks for help.
UPDATE:
Even I changed the code by using POST action as below, the situation is still unchanged.
private bool ProcessEachChannel(Tuple<ChannelResource, string> workItem)
{
// blah blah to return true or false respectively.
public void SimultaneousCall()
{
Console.WriteLine("There are {0} channels to be processed.", bc.Count);
var workItemBlock = new ActionBlock<Tuple<ChannelResource, string>>(
workItem =>
{
bool success = ProcessEachChannel(workItem);
});
foreach (var workItem in bc.GetConsumingEnumerable())
{
workItemBlock.Post(workItem);
}
workItemBlock.Complete();
}

I believe the problem is that you never call bc.CompleteAdding(): the if means it would be called in ports + 1-th iteration of the loop, but the loop iterates only ports-times. Because of this, GetConsumingEnumerable() returns a sequence that never ends, which means the foreach inside SimultaneousCall() blocks forever.
I think the right solution is to call bc.CompleteAdding() after the for loop, not in an impossible condition inside it.