I have a process that reads a message from an Azure Service Bus Queue and converts that message to a Video to be Encoded by Azure Media Services. I noticed that if the process is kicked off very quickly in a row, the same video was being encoded right after another. Here is my code that adds the Video to the Queue
public class VideoManager
{
string _connectionString = ConfigurationManager.AppSettings["Microsoft.ServiceBus.ConnectionString"];
string _queueName = ConfigurationManager.AppSettings["ServiceBusQueueName"];
QueueClient _client;
public VideoManager()
{
var conStringBuilder = new ServiceBusConnectionStringBuilder(_connectionString)
{
OperationTimeout = TimeSpan.FromMinutes(120)
};
var messagingFactory = MessagingFactory.CreateFromConnectionString(conStringBuilder.ToString());
_client = messagingFactory.CreateQueueClient(_queueName);
}
public void Approve(Video video)
{
// Set video to approved.
video.ApprovalStatus = ApprovalStatus.Approved;
var message = new BrokeredMessage(new VideoMessage(video, VideoMessage.MessageTypes.Approve, string.Empty));
message.MessageId = video.RowKey;
_client.Send(message);
}
}
And the process that reads from the Queue
class Program
{
static QueueClient client;
static void Main(string[] args)
{
VideoManager videoManager = new VideoManager();
var connectionString = ConfigurationManager.AppSettings["Microsoft.ServiceBus.ConnectionString"];
var conStringBuilder = new ServiceBusConnectionStringBuilder(connectionString)
{
OperationTimeout = TimeSpan.FromMinutes(120)
};
var messagingFactory = MessagingFactory.CreateFromConnectionString(conStringBuilder.ToString());
client = messagingFactory.CreateQueueClient(ConfigurationManager.AppSettings["ServiceBusQueueName"]);
Console.WriteLine("Starting: Broadcast Center Continuous Video Processing Job");
OnMessageOptions options = new OnMessageOptions
{
MaxConcurrentCalls = 25,
AutoComplete = false
};
client.OnMessageAsync(async message =>
{
bool shouldAbandon = false;
try
{
await HandleMessage(message);
}
catch (Exception ex)
{
shouldAbandon = true;
Console.WriteLine(ex.Message);
}
if (shouldAbandon)
{
await message.AbandonAsync();
}
}, options);
while (true) { }
}
async static Task<int> HandleMessage(BrokeredMessage message)
{
VideoMessage videoMessage = message.GetBody<VideoMessage>();
Console.WriteLine(String.Format("Message body: {0}", videoMessage.Video.Title));
Console.WriteLine(String.Format("Message id: {0}", message.MessageId));
VideoProcessingService vp = new VideoProcessingService(videoMessage.Video);
Task task;
switch (videoMessage.MessageType)
{
case VideoMessage.MessageTypes.CreateThumbnail:
task = new Task(() => vp.ProcessThumbnail(videoMessage.TimeStamp));
task.Start();
while (!task.IsCompleted)
{
await Task.Delay(15000);
message.RenewLock();
}
await task;
Console.WriteLine(task.Status.ToString());
Console.WriteLine("Processing Complete");
Console.WriteLine("Awaiting Message");
break;
case VideoMessage.MessageTypes.Approve:
task = new Task(() => vp.Approve());
task.Start();
while (!task.IsCompleted)
{
await Task.Delay(15000);
message.RenewLock();
}
await task;
Console.WriteLine(task.Status.ToString());
Console.WriteLine("Processing Complete");
Console.WriteLine("Awaiting Message");
break;
default:
break;
}
return 0;
}
}
What I see in the Console Window is the following if I kick off the process 3 times in a row
Message id: 76aca19a-0698-449b-bf58-a24876fc4314
Message id: 76aca19a-0698-449b-bf58-a24876fc4314
Message id: 76aca19a-0698-449b-bf58-a24876fc4314
I thought maybe I did not have the settings correct, but they are there
I am really at a loss here, as I would expect this to be fairly out of the box behavior. Does duplicate detection only work if the message has been completed, so I can't use OnMessageAsync()?
The issue is not the completion (as it was in the code), but the fact that you have in essence multiple consumers (25 concurrent callbacks) and it seems like the LockDuration is elapsing faster than the processing takes. As a result of that, message re-appears and re-processed. As a result of that you see the same message ID logged more than once.
Possible solutions are (as I've outlined in a comment above):
Let OnMessage API manage timeout extension for you (example)
Manually renew the lock as you've done using BrokeredMessage.RenewLock
There is a line of code missing from your HandleMessage code.
async static Task<int> HandleMessage(BrokeredMessage message)
{
VideoMessage videoMessage = message.GetBody<VideoMessage>();
message.CompleteAsync(); // This line...
Console.WriteLine(String.Format("Message id: {0}", message.MessageId));
// Processes Message
}
So yes you have to mark the message with either, Complete, Defer etc..
Also see this Answer, also found this which may be useful in how duplicate detection works
Related
Following this example I have now therefore been required to update the MQTT.NET from version 3 (that works thanks the provided help) to version 4.
A very basic set of capabilities would be enough:
Connect to an adress with a timeout
Check if the connection has gone well
Receive messages
check disconnection
that was extremely easy in version 3
MqttClientOptionsBuilder builder = new MqttClientOptionsBuilder()
.WithClientId("IoApp" + HelperN.MQTT.GetClientID(true))
.WithTcpServer("localhost", 1883);
ManagedMqttClientOptions options = new ManagedMqttClientOptionsBuilder()
.WithAutoReconnectDelay(TimeSpan.FromSeconds(60))
.WithClientOptions(builder.Build())
.Build();
mqttClient = new MqttFactory().CreateManagedMqttClient();
mqttClient.ConnectedHandler = new MqttClientConnectedHandlerDelegate(OnConnected);
mqttClient.DisconnectedHandler = new MqttClientDisconnectedHandlerDelegate(OnDisconnected);
mqttClient.ConnectingFailedHandler = new ConnectingFailedHandlerDelegate(OnConnectingFailed);
mqttClient.SubscribeAsync(...);
mqttClient.SubscribeAsync(...);
mqttClient.StartAsync(options).GetAwaiter().GetResult();
mqttClient.UseApplicationMessageReceivedHandler(args => { OnMessageReceived(args); });
but when it comes to version 4 if I have to relay on those examples I have problems.
Let's start from the connection
public static async Task Connect_Client_Timeout()
{
/*
* This sample creates a simple MQTT client and connects to an invalid broker using a timeout.
*
* This is a modified version of the sample _Connect_Client_! See other sample for more details.
*/
var mqttFactory = new MqttFactory();
strError = String.Empty;
using (var mqttClient = mqttFactory.CreateMqttClient())
{
var mqttClientOptions = new MqttClientOptionsBuilder().WithTcpServer("aaaa127.0.0.1",1883).Build();
try
{
using (var timeoutToken = new CancellationTokenSource(TimeSpan.FromSeconds(5)))
{
await mqttClient.ConnectAsync(mqttClientOptions, timeoutToken.Token);
}
}
catch (OperationCanceledException exc)
{
strError = "Connect_Client_Timeout exc:" + exc.Message;
}
}
}
And I call this task from the main awaiting the result.
var connectTask = Connect_Client_Timeout();
connectTask.Wait();<-----never ends
Since I put a wrong address "aaaa127.0.0.1" I expect a failure after 5 seconds. But the connectTask.Wait never end. But even if I put the right address "127.0.0.1" it never exits.
So perhaps the error stands in the connectTask.Wait();.
Thanks
The solution is here
In short you have to do this:
static async Task Connect()
{
IManagedMqttClient _mqttClient = new MqttFactory().CreateManagedMqttClient();
// Create client options object
MqttClientOptionsBuilder builder = new MqttClientOptionsBuilder()
.WithClientId("behroozbc")
.WithTcpServer("localhost");
ManagedMqttClientOptions options = new ManagedMqttClientOptionsBuilder()
.WithAutoReconnectDelay(TimeSpan.FromSeconds(60))
.WithClientOptions(builder.Build())
.Build();
// Set up handlers
_mqttClient.ConnectedAsync += _mqttClient_ConnectedAsync;
_mqttClient.DisconnectedAsync += _mqttClient_DisconnectedAsync;
_mqttClient.ConnectingFailedAsync += _mqttClient_ConnectingFailedAsync;
// Connect to the broker
await _mqttClient.StartAsync(options);
// Send a new message to the broker every second
while (true)
{
string json = JsonSerializer.Serialize(new { message = "Hi Mqtt", sent = DateTime.UtcNow });
await _mqttClient.EnqueueAsync("behroozbc.ir/topic/json", json);
await Task.Delay(TimeSpan.FromSeconds(1));
}
Task _mqttClient_ConnectedAsync(MqttClientConnectedEventArgs arg)
{
Console.WriteLine("Connected");
return Task.CompletedTask;
};
Task _mqttClient_DisconnectedAsync(MqttClientDisconnectedEventArgs arg)
{
Console.WriteLine("Disconnected");
return Task.CompletedTask;
};
Task _mqttClient_ConnectingFailedAsync(ConnectingFailedEventArgs arg)
{
Console.WriteLine("Connection failed check network or broker!");
return Task.CompletedTask;
}
}
and then just call Connect() and rely on the subscribed examples
New to this parallel coding.
I am trying to fire off a list of tasks (in this case, emails to send).
The below code does work, however I am unsure, if ONE email was to fail sending, or that task not finish for whatever reason etc.
Would me code just hang on the await line?
What the solution here, or is it a non problem?
I need to await OR intercept each one when it finishes as I need to (mark the email as sent correctly or retry etc).
However is awaiting all safe, or should I have an event to intercept the response?
(I do not really care if the odd one takes a while, My key is not to block up the rest of the program)
Over all this will be in a 5 mins loop, and I wouldn't want the email in the future to be blocked up is a timeout per task, as option.
Any help would be great, thank You.
// Create a list of emails to send
List<Task<string>> tasks = new List<Task<string>>();
foreach (var item in lstEmailList) // loop the list of emails to send
{
tasks.Add(EmailLib.SendEmailOldSmtpAuth(item)); // add each email as a task
}
// now run the list as parallel tasks
await Task.Run(() => Parallel.ForEach(tasks, s =>
{
//EmailLib.SendEmailOldSmtpAuth(item);
}));
foreach (var task in tasks)
{
var result = ((Task<string>)task).Result;
if (result.Contains("PH_OK") == true)
{
string strFindId = result.Replace("PH_OK=", "").ToString();
int EmailID = Int32.Parse(strFindId);
DbFunc.MarkEmailAsSent(ref SQLConnX, EmailID);
}
}
In case it is helpful:
public async Task<string> SendEmailOldSmtpAuth(Data_PendingEmails objEmail)
{
try
{
var emailMessage = new MimeMessage();
string SendToName = "";
if (objEmail.CarerID > 0)
{
SendToName = objEmail.carForename + " " + objEmail.carSurname;
}
else
{
SendToName = objEmail.cliForename + " " + objEmail.cliSurname;
}
if (SendToName == "")
{
SendToName = "User";
}
emailMessage.From.Add(new MailboxAddress(objEmail.EMailYourName, objEmail.EMailAddress));
emailMessage.To.Add(new MailboxAddress(SendToName, objEmail.ToEmailAddress));
emailMessage.Subject = objEmail.EmailSubject;
emailMessage.Body = new TextPart("html") { Text = objEmail.EmailMessage };
try
{
var client = new SmtpClient();
await client.ConnectAsync(objEmail.SMTPServer, Int32.Parse(objEmail.SMTPPort), SecureSocketOptions.SslOnConnect);
await client.AuthenticateAsync(objEmail.SMTPUserName, objEmail.SMTPPassword);
await client.SendAsync(emailMessage);
await client.DisconnectAsync(true);
return "PH_OK=" + objEmail.EmailID.ToString();
}
catch (Exception ex)
{
var e = ex;
return e.Message;
}
}
catch (Exception SendEmailOldSmtpAuthOverAll)
{
return SendEmailOldSmtpAuthOverAll.Message.ToString();
}
}
I will try to simulate your scenario with a much simpler example but the main idea will be the same. I do it this way to simulate the potential exception.
First af all you need concurrent calls to a web service, so the best way is to use Task.WhenAll because this is an I/O operation (as #Charlieface already mentioned)
Lets say that we have a list of emails:
var emailList = new List<string>() { "1", "2", "3", "4", "5", "6" };
then we need to create an ienumerable of tasks:
var tasks = emailList.Select(async email =>
{
var response = await SendEmailAsync(email);
Console.WriteLine(response);
});
Then we "mock" an exception and append it to task list in order to simulate an exception:
var problematicTask = ThrowExceptionAsync("Error from initial task");
var allTasks = tasks.Append(problematicTask);
Now, in order to fire the tasks and to catch the exceptions we need to do this:
var aggregateTasks = Task.WhenAll(allTasks);
try
{
await aggregateTasks;
}
catch
{
AggregateException aggregateException = aggregateTasks.Exception!;
foreach (var ex in aggregateException.InnerExceptions)
{
Console.WriteLine(ex.Message);
}
}
and as helper methods for this example i created these two arbitrary methods:
async Task<string> SendEmailAsync(string email)
{
await Task.Delay(1500);
if (email.Equals("7") || email.Equals("10"))
{
await ThrowExceptionAsync("Error from sendEmail");
}
return $"Email: {email} sent";
}
async Task ThrowExceptionAsync(string msg)
{
throw new Exception(msg);
}
If you run this simple example you would inspect all exceptions may thrown in each call..
Now in regards to your particular example i think you need to remove all try/catch blocks in order to catch the exceptions as aggregate exception and not blocking your app.
My telegram bot is necessary so that the user can answer questions in order and save these answers in the same order for a specific user in parallel.
static readonly ConcurrentDictionary<int, string[]> Answers = new ConcurrentDictionary<int, string[]>();
static void Main(string[] args)
{
try
{
Task t1 = CreateHostBuilder(args).Build().RunAsync();
Task t2 = BotOnMessage();
await Task.WhenAll(t1, t2);
}
catch (Exception ex)
{
Console.WriteLine("Error" + ex);
}
}
here is my BotOnMessage() method to receive and process messages from users
async static Task BotOnMessage()
{
int offset = 0;
int timeout = 0;
try
{
await bot.SetWebhookAsync("");
while (true)
{
var updates = await bot.GetUpdatesAsync(offset, timeout);
foreach (var update in updates)
{
var message = update.Message;
if (message.Text == "/start")
{
Registration(message.Chat.Id.ToString(), message.Chat.FirstName.ToString(), createdDateNoTime.ToString("yyyy-MM-dd HH:mm:ss.fff", CultureInfo.InvariantCulture));
var replyKeyboard = new ReplyKeyboardMarkup
{
Keyboard = new[]
{
new[]
{
new KeyboardButton("eng"),
new KeyboardButton("ger")
},
}
};
replyKeyboard.OneTimeKeyboard = true;
await bot.SendTextMessageAsync(message.Chat.Id, "choose language", replyMarkup: replyKeyboard);
}
switch (message.Text)
{
case "eng":
var replyKeyboardEN = new ReplyKeyboardMarkup
{
Keyboard = new[]
{
new[]
{
new KeyboardButton("choice1"),
new KeyboardButton("choice2")
},
}
};
replyKeyboardEN.OneTimeKeyboard = true;
await bot.SendTextMessageAsync(message.Chat.Id, "Enter choice", replyMarkup: replyKeyboardEN);
await AnonymEN();
break;
case "ger":
var replyKeyboardGR = new ReplyKeyboardMarkup
{
Keyboard = new[]
{
new[]
{
new KeyboardButton("choice1.1"),
new KeyboardButton("choice2.2")
},
}
};
replyKeyboardGR.OneTimeKeyboard = true;
await bot.SendTextMessageAsync(message.Chat.Id, "Enter choice", replyMarkup: replyKeyboardGR);
await AnonymGR();
break;
}
offset = update.Id + 1;
}
}
}
catch (Exception ex)
{
Console.WriteLine("Error" + ex);
}
}
and AnonymEN() method for eng case in switch. The problem appears here when I call this method from switch case in BotOnMessage(). Until switch (message.Text) multiple users can asynchronously send messages and get response. When first user enters AnonymEN() second user can't get response from this method until first user will finish it till the end. Also I call BotOnMessage() in the end of AnonymEN() to get back for initial point with possibility to start bot again. For the ordered structure of questions and answers I used ConcurrentDictionary way from here Save user messages sent to bot and send finished form to other user. Any suggestion and solution how to edit code to make this bot available for multiple users at one time?
async static Task AnonymEN()
{
int offset = 0;
int timeout = 0;
try
{
await bot.SetWebhookAsync("");
while (true)
{
var updates = await bot.GetUpdatesAsync(offset, timeout);
foreach (var update in updates)
{
var message = update.Message;
int userId = (int)message.From.Id;
if (message.Type == MessageType.Text)
{
if (Answers.TryGetValue(userId, out string[] answers))
{
var title = message.Text;
if (answers[0] == null)
{
answers[0] = message.Text;
await bot.SendTextMessageAsync(message.Chat, "Enter age");
}
else
{
SaveMessage(message.Chat.Id.ToString(), "anonym", "anonym", "anonym", answers[0].ToString(), title.ToString(), createdDateNoTime.ToString("yyyy-MM-dd HH:mm:ss.fff", CultureInfo.InvariantCulture));
Answers.TryRemove(userId, out string[] _);
await bot.SendTextMessageAsync(message.Chat.Id, "ty for request click /start");
await BotOnMessage();
}
}
else if (message.Text == "choice1")
{
Answers.TryAdd(userId, new string[1]);
await bot.SendTextMessageAsync(message.Chat.Id, "Enter name");
}
}
offset = update.Id + 1;
}
}
}
catch (Exception ex)
{
Console.WriteLine("Error" + ex);
}
}
I can see multiple issues with your code:
It is hard to read. While this is a personal preference I strongly advise to write short concise methods that have 1 responsibility. This will make it easier to understand and maintain your code. https://en.wikipedia.org/wiki/Single-responsibility_principle
Everything is static. This makes it very hard to keep track of any state such as language that should be tracked per user.
Using infinite loops and recursion with no escape. I highly doubt this was intended but you could get an infinite chain of calls like this BotOnMessage -> AnonymEN -> BotOnMessage -> AnonymEN. I think you want to exit the AnonymEN function using either a return, break or while(someVar) approach instead of calling the BotOnMessage function.
If two users are sending messages you get mixed responses. Example message flow user1: /start, user1: eng, user2: hello. The bot will now give an english response to user2. I'm sure this is not intended
The code below is a minimal example that addresses the issues I mentioned. It is not perfect code but should help you get started.
private Dictionaty<string, UserSession> userSessions = new ();
async Task BotOnMessage()
{
try
{
while(true)
{
var message = await GetMessage(timeout);
var userSession = GetUserSession(message.user);
userSession.ProcessMessage(message);
}
}
catch(){}
}
async void GetUserSession(string user)
{
if(!userSessions.HasKey(user))
{
userSessions[user](new Session());
}
return userSessions[user];
}
public class UserSession
{
public async Task ProcessMessage(message)
{
// Existing message processing code goes here.
// Do not use a loop or recursion.
// Instead track the state (e.g. languge) using fields.
}
}
I wrote a code to read 1000 messages one shot from an Azure Service Bus queue. I read the messages with the line: await receiver.ReceiveMessagesAsync(1000); but only a subset of the messages are received.
I took the code from the sample: https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/servicebus/Azure.Messaging.ServiceBus/tests/Samples/Sample01_HelloWorld.cs, the SendAndReceiveMessageSafeBatch() method
This is my code:
public class Program
{
static void Main(string[] args)
{
SendAndReceiveMessage().GetAwaiter().GetResult();
}
public static async Task SendAndReceiveMessage()
{
var connectionString = "myconnectionstring";
var queueName = "myqueue";
// since ServiceBusClient implements IAsyncDisposable we create it with "await using"
await using var client = new ServiceBusClient(connectionString);
// create the sender
var sender = client.CreateSender(queueName);
IList<ServiceBusMessage> messages = new List<ServiceBusMessage>();
for (var i = 0; i < 1000; i++)
{
messages.Add(new ServiceBusMessage($"Message {i}"));
}
// send the messages
await sender.SendMessagesAsync(messages);
// create a receiver that we can use to receive the messages
var options = new ServiceBusReceiverOptions()
{
ReceiveMode = ServiceBusReceiveMode.ReceiveAndDelete
};
ServiceBusReceiver receiver = client.CreateReceiver(queueName, options);
// the received message is a different type as it contains some service set properties
IReadOnlyList<ServiceBusReceivedMessage> receivedMessages = await receiver.ReceiveMessagesAsync(1000);
Console.WriteLine($"Received {receivedMessages.Count} from the queue {queueName}");
foreach (ServiceBusReceivedMessage receivedMessage in receivedMessages)
{
var body = receivedMessage.Body.ToString();
Console.WriteLine(body);
}
Console.WriteLine("END");
Console.ReadLine();
}
}
Do you have any suggestion how to read all 1000 messages one shot?
This is expected behaviour with Azure Service Bus. The number of messages to receive, maxMessages is a maximum number that is not guaranteed.
I have created a task which creates an XML string. The task can last multiple seconds. When the task isn't finished after 5 seconds, I want to cancel the Task 'smootly' and continue with writing the rest of the XML. So I built in cancellation inside my task. But although I see the following message in the Log:
ProcessInformationTask timed out
I also see this line in my log
Adding the process information took 10001 ms
I wonder why this can happen because I want to cancel the Task after 5 seconds (if not finished). So I expect the task to last 5 seconds max. How can I solve this? Probably the cancelation isn't setup properly?
Code where I call my task
string additionalInformation = null;
var contextInfo = new StringBuilder();
var xmlWriterSettings = new XmlWriterSettings()
{
OmitXmlDeclaration = true,
ConformanceLevel = ConformanceLevel.Fragment
};
var tokenSource = new CancellationTokenSource();
var token = tokenSource.Token;
using (XmlWriter xmlWriter = XmlWriter.Create(contextInfo, xmlWriterSettings))
{
try
{
xmlWriter.WriteStartElement("AdditionalInformation");
//Write xml (not long running)
var watch = System.Diagnostics.Stopwatch.StartNew();
string processInformation = AddProcessesInformation(xmlWriterSettings);
watch.Stop();
var elapsedMs = watch.ElapsedMilliseconds;
Log.Info("Adding the process information took : " + elapsedMs + " ms");
if (!string.IsNullOrEmpty(processInformation))
{
xmlWriter.WriteRaw(processInformation);
}
//Write xml (not long running)
xmlWriter.WriteEndElement();
additionalInformation = contextInfo.ToString();
}
catch (Exception e)
{
Log.Info("An exception occured during writing the additional information: " + e.Message);
return false;
}
return true;
}
Task method
private static string AddProcessesInformation(XmlWriterSettings xmlWriterSettings)
{
var tokenSource = new CancellationTokenSource();
var token = tokenSource.Token;
var contextInfo = new StringBuilder();
var processInformationTask = Task<string>.Factory.StartNew(() =>
{
if (token.IsCancellationRequested)
{
Log.Info("Cancellation request for the ProcessInformationTask");
}
Thread.Sleep(10000);
return "Ran without problems'";
}, token);
if (!processInformationTask.Wait(5000, token))
{
Log.Info("ProcessInformationTask timed out");
tokenSource.Cancel();
}
return processInformationTask?.Result;
}
I think you should check if cancellation is requested multiple Times, after each step in your method. Here is example using for loop:
private static string AddProcessesInformation(XmlWriterSettings xmlWriterSettings)
{
var tokenSource = new CancellationTokenSource();
var token = tokenSource.Token;
var contextInfo = new StringBuilder();
var processInformationTask = Task<string>.Factory.StartNew(() =>
{
for(int i = 0; i < 10; i++)
{
if (token.IsCancellationRequested)
{
Console.WriteLine("Cancellation request for the ProcessInformationTask");
return string.Empty;
}
Thread.Sleep(1000);
}
return "Ran without problems'";
}, token);
if (!processInformationTask.Wait(5000, token))
{
Console.WriteLine("ProcessInformationTask timed out");
tokenSource.Cancel();
}
return processInformationTask?.Result;
}
If inside task method you call other methods, you can pass cancellation token to them and use method token.ThrowIfCancellationRequested() inside.
var processInformationTask = Task<string>.Factory.StartNew(() =>
{
try
{
Method1(token);
Thread.Sleep(1000);
Method2(token);
Thread.Sleep(1000);
}
catch(OperationCanceledException ex)
{
return string.Empty;
}
return "Ran without problems'";
}, token);
private static void Method1(CancellationToken token)
{
token.ThrowIfCancellationRequested();
// do more work
}