We are using Azure Service Bus in our project and while reading messages from service bus topic/subscription.
We are using subscriptionClient.OnMessageAsync event in conjunction with onMessageOptions.ExceptionReceived.
Let me write down the steps we followed to reproduce the issue we are facing.
Create a service bus namespace with default config in the azure portal
Create a topic inside it with default config in the azure portal
Create a subscription inside it with default config in the azure portal
Create a console app and paste the code added below
Connect the service bus using Service Bus Explorer
Run the console app
Send a few test messages from service bus explorer & watch the console app window
Though the messages are processed successfully every time the control is going inside the ExceptionReceived method
Here's the code:
class Program
{
static void Main()
{
var subscriptionClient = SubscriptionClient.CreateFromConnectionString
(
"servicebusendpointaddress",
"topicname",
"subscriptionname",
ReceiveMode.PeekLock
);
var onMessageOptions = new OnMessageOptions();
onMessageOptions.ExceptionReceived += OnMessageError;
subscriptionClient.OnMessageAsync(OnMessageReceived, onMessageOptions);
Console.ReadKey();
}
private static void OnMessageError(object sender, ExceptionReceivedEventArgs e)
{
if (e != null && e.Exception != null)
{
Console.WriteLine("Hey, there's an error!" + e.Exception.Message + "\r\n\r\n");
}
}
private static async Task OnMessageReceived(BrokeredMessage arg)
{
await arg.CompleteAsync();
Console.WriteLine("Message processing done!");
}
}
Are we missing something here?
Also one point to mention is that is we enable ‘autocomplete’ and remove the await arg.CompleteAsync(); then this is not happening.
var onMessageOptions = new OnMessageOptions() { AutoComplete = true};
In both the cases the messages are being processed successfully & removed from the subscription immediately.
You might be getting this because you are debugging and stepping though the code i.e. the lock expires. The LockDuration by default is 60 seconds.
You can try setting your OnMessageOptions() like this to test:
var onMessageOptions = new OnMessageOptions() { AutoRenewTimeout = TimeSpan.FromMinutes(1) };
Related
Just starting to work with Azure.
Have a simple C# .NET Core app, which connects to Azure ServiceBus, reads messages, and writes them to Azure SQL database.
Works locally just fine - connects to remote Azure Service Bus Queue, reads messages, connect to remote Azure SQL db, writes records.
Same exact app, when deployed to Azure as a WebApp, appears to "run", but no longer reads messages from Services Bus, and no longer writes anything to Azure SQL.
Here is the entire app (i.e. Program.cs):
using System;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Azure.ServiceBus;
using System.Data.SqlClient;
namespace ServiceBusReader
{
class Program
{
const string ServiceBusConnectionString = "SB_CONNECTION_STRING";
const string QueueName = "BasicQueue";
static IQueueClient queueClient;
static SqlConnection connection = null;
public static async Task Main(string[] args)
{
System.Diagnostics.Trace.TraceError("Inside Main function...");
queueClient = new QueueClient(ServiceBusConnectionString, QueueName);
Console.WriteLine("======================================================");
System.Diagnostics.Trace.TraceError("======================================================");
Console.WriteLine("Press ENTER key to exit after receiving all the messages.");
Console.WriteLine("======================================================");
SqlConnectionStringBuilder builder = new SqlConnectionStringBuilder();
builder.DataSource = "XXXX.database.windows.net";
builder.UserID = "USERID";
builder.Password = "PASSWORD";
builder.InitialCatalog = "mySampleDatabase";
connection = new SqlConnection(builder.ConnectionString);
connection.Open();
// Register the queue message handler and receive messages in a loop
RegisterOnMessageHandlerAndReceiveMessages();
Console.ReadKey();
await queueClient.CloseAsync();
}
static void RegisterOnMessageHandlerAndReceiveMessages()
{
// Configure the message handler options in terms of exception handling, number of concurrent messages to deliver, etc.
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandler)
{
// Maximum number of concurrent calls to the callback ProcessMessagesAsync(), set to 1 for simplicity.
// Set it according to how many messages the application wants to process in parallel.
MaxConcurrentCalls = 1,
// Indicates whether the message pump should automatically complete the messages after returning from user callback.
// False below indicates the complete operation is handled by the user callback as in ProcessMessagesAsync().
AutoComplete = false
};
// Register the function that processes messages.
queueClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
}
static async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
// Process the message.
Console.WriteLine($"Received message: SequenceNumber:{message.SystemProperties.SequenceNumber} Body:{Encoding.UTF8.GetString(message.Body)}");
string query = "INSERT INTO [SalesLT].[messages] (message) VALUES(#Message)";
SqlCommand cmd = new SqlCommand(query, connection);
System.Diagnostics.Trace.TraceError(Encoding.UTF8.GetString(message.Body));
cmd.Parameters.AddWithValue("#Message", Encoding.UTF8.GetString(message.Body));
cmd.ExecuteNonQuery();
Console.WriteLine("Records Inserted Successfully...");
System.Diagnostics.Trace.TraceError("Records Inserted Successfully...");
// Complete the message so that it is not received again.
// This can be done only if the queue Client is created in ReceiveMode.PeekLock mode (which is the default).
await queueClient.CompleteAsync(message.SystemProperties.LockToken);
// Note: Use the cancellationToken passed as necessary to determine if the queueClient has already been closed.
// If queueClient has already been closed, you can choose to not call CompleteAsync() or AbandonAsync() etc.
// to avoid unnecessary exceptions.
}
// Use this handler to examine the exceptions received on the message pump.
static Task ExceptionReceivedHandler(ExceptionReceivedEventArgs exceptionReceivedEventArgs)
{
Console.WriteLine($"Message handler encountered an exception {exceptionReceivedEventArgs.Exception}.");
var context = exceptionReceivedEventArgs.ExceptionReceivedContext;
Console.WriteLine("Exception context for troubleshooting:");
Console.WriteLine($"- Endpoint: {context.Endpoint}");
Console.WriteLine($"- Entity Path: {context.EntityPath}");
Console.WriteLine($"- Executing Action: {context.Action}");
return Task.CompletedTask;
}
}
}
Should I do anything differently in this app in order to make it work in Azure?
It seems that you are trying to deploy a webjob to web app. May I know if you have set it to Continuous type? If you have set it to Continuous, the webjob will automatically run after deployment.
By default, the type is Triggered and you need to manually start the webjob from portal.
I have a problem with MassTransit (Using MassTransit 3.5.7 (via Nuget), RabbitMQ 3.6.10, Erlang 19.0)
Looks like that MassTransit not clear up RabbitMQ channels, when bus is failed to start.
Here is my test program
using System;
using System.Threading;
using MassTransit;
namespace TestSubscriber
{
class Program
{
static void Main()
{
IBusControl busControl = null;
var failCount = 0;
var busNotInitialised = true;
//Keep RabbitMQ switched off for a few iterations of this loop, then switch it on.
while (busNotInitialised)
{
busControl = Bus.Factory.CreateUsingRabbitMq(sbc =>
{
var host = sbc.Host(new Uri("rabbitmq://localhost/"), h =>
{
h.Username("guest");
h.Password("guest");
});
sbc.ReceiveEndpoint(host, "some_queue", endpoint =>
{
endpoint.Handler<string>(async context =>
{
await Console.Out.WriteLineAsync($"Received: {context.Message}");
});
});
});
try
{
busControl.Start();
busNotInitialised = false;
}
catch (Exception)
{
Console.WriteLine($"Attempt:{++failCount} failed.");
//wait some time
Thread.Sleep(5000);
}
}
//At this point, using RabbitMq's management web page, you will see failCount + 1 channels.
busControl.Stop();
//At this point, using RabbitMq's management web page, you will see failCount channels.
Console.ReadLine();
}
}
}
It continuously tries to create a service bus that uses RabbitMQ.
The program breaks out of the loop as soon as the service bus is created successfully.
After running the program for a couple of minutes (with stopped RabbitMQ) and stopping on a breakpoint, I can see a lot of worker threads (one for each failed service bus creation attempt).
After startup RabbitMQ, all of those "dangling" connect threads will connect to RabbitMQ.
If I try to shut down the bus, the latest connection ( that belongs to the bus that was successfully created), is closed. All of the other dangling connections are still connected to RabbitMQ.
Problem here is that those dangling threads, when connected, reads data from the queue which leads to data loss.
Is there any way to get around this problem?
I have a job that imports files into a system. Everytime a file is imported, we create a blob in azure and we send a message with instructions to a queue so that the data is persisted in SQL accordingly. We do this using azure-webjobs and azure-webjobssdk.
We experienced an issue in which after the messages failed more than 7 times, they didn't move to the poision queue as expected. The code is the following:
Program.cs
public class Program
{
static void Main()
{
//Set up DI
var module = new CustomModule();
var kernel = new StandardKernel(module);
//Configure JobHost
var storageConnectionString = AppSettingsHelper.Get("StorageConnectionString");
var config = new JobHostConfiguration(storageConnectionString) { JobActivator = new JobActivator(kernel), NameResolver = new QueueNameResolver() };
config.Queues.MaxDequeueCount = 7;
config.UseTimers();
//Pass configuration to JobJost
var host = new JobHost(config);
host.RunAndBlock();
}
}
Functions.cs
public class Functions
{
private readonly IMessageProcessor _fileImportQueueProcessor;
public Functions(IMessageProcessor fileImportQueueProcessor)
{
_fileImportQueueProcessor = fileImportQueueProcessor;
}
public async void FileImportQueue([QueueTrigger("%fileImportQueueKey%")] string item)
{
await _fileImportQueueProcessor.ProcessAsync(item);
}
}
_fileImportQueueProcessor.ProcessAsync(item) threw an exception and the message's dequeue count was properly increased and re-processed. However, it was never moved to the poison-queue. I attached a screenshot of the queues with the dequeue counts at over 50.
After multiple failures the webjob was stuck in a Pending Restart state and I was unable to either stop or start and I ended up deleting it completely. After running the webjob locally, I saw messages being processed (I assumed that the one with a dequeue count of over 7 should've been moved to the poison queue).
Any ideas on why this is happening and what can be done to have the desired behavior.
Thanks,
Update
Vivien's solution below worked.
Matthew has kind enough to do a PR that will address this. You can check out the PR here.
Fred,
The FileImportQueue method being an async void is the source of your problem.
Update it to return a Task:
public class Functions
{
private readonly IMessageProcessor _fileImportQueueProcessor;
public Functions(IMessageProcessor fileImportQueueProcessor)
{
_fileImportQueueProcessor = fileImportQueueProcessor;
}
public async Task FileImportQueue([QueueTrigger("%fileImportQueueKey%")] string item)
{
await _fileImportQueueProcessor.ProcessAsync(item);
}
}
The reason for the dequeue count to be over 50 is because when _fileImportQueueProcessor.ProcessAsync(item) threw an exception it will crash the whole process. Meaning the WebJobs SDK can't execute the next task that will move the message to the poison queue.
When the message is available again in the queue the SDK will process it again and so on.
I have created a azure webjob which will send a strongly typed message to a service bus queue and it successfully sends.
I want to create another webjob which should be triggered whenever there is a message in the servicebus queue. Please find below the code i am trying. For some reason, though there are messages in the servicebus queue, the webjob is not getting triggered and getting an error when i run the webjob locally.
Error:
System.InvalidOperationException
{"Missing value for trigger parameter 'blobIinfo'."}
Code:
public static void Main()
{
var config = new JobHostConfiguration
{
NameResolver = new QueueNameResolver(),
ServiceBusConnectionString = ApplicationSettings.ServiceBusConnectionString
};
var host = new JobHost(config);
host.Call(typeof(BankLineFileProcessorWebJob).GetMethod("ProcessQueueMessage"));
}
[NoAutomaticTrigger]
public static void ProcessQueueMessage(
TextWriter log,
[ServiceBusTrigger("testsftppollingqueue")] SftpQueueMessage blobIinfo
)
{
while (true)
{
log.WriteLine("Queue message refers to blob: " + blobIinfo.BlobUri);
Thread.Sleep(TimeSpan.FromMinutes(PollingInterval));
}
}
Can anyone help me how to solve this?
Thanks
You have to use
host.RunAndBlock();
instead of
host.Call(typeof(BankLineFileProcessorWebJob).GetMethod("ProcessQueueMessage"));
Also, please take out the NoAutomaticTrigger attribute.
I developed a http server via console application in C# and decided to turn it into a Windows service to be able to initialize it without the need to login the machine.
I followed all the steps in How to create Windows Service and chose the account as "Local System", but when I install in my server machine and push the start button it takes a while and gives the following error:
Erro 1053: The service did not respond to the start or control request in timely fashion.
After that, the service status stays stuck in "starting" and the application don't work and I can't even stop the service anymore.
Trying to work around this problem, I changed it to "Network Service", so it started normally, but the application was not listening in the port I set when I checked in the prompt with the command "netstat -an". But the application listens normally if i run it as a console application.
So I am looking for an answer to one of these two questions:
What should I do to make the service starts properly with a Local System account?
If I decide to use Network service account, what should I care about to guarantee that my service works properly as a server?
When I converted my console application to windows service I simply put my code directly in the OnStart method. However, I realized the OnStart method should start the service, but needs to end some time to the service indeed start. So I created a thread that runs my service and let the OnStart method finish. I tested and the service worked just fine. Here is how it was the code:
protected override void OnStart(string[] args)
{
Listener(); // this method never returns
}
Here is how it worked:
protected override void OnStart(string[] args)
{
Thread t = new Thread(new ThreadStart(Listener));
t.Start();
}
But I still don't understand why the service ran (passed the "starting" status, but didn't work) when I used network service account. If anyone knows, I'll be glad to know the reason.
If you have a service that is not responding or showing pending in Windows services that you are unable to stop, use the following directions to force the service to stop.
Start -> Run or Start -> type services.msc and press Enter
Look for the service and check the Properties and identify its service name
Once found, open a command prompt. Type sc queryex [servicename]
Identify the PID (process ID)
In the same command prompt type taskkill /pid [pid number] /f
Find PID of Service
sc queryex <SERVICE_NAME>
Give result's below
SERVICE_NAME: Foo.Services.Bar TYPE : 10 WIN32_OWN_PROCESS STATE : 2 0 START_PENDING (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN) WIN32_EXIT_CODE : 0 (0x0) SERVICE_EXIT_CODE : 0 (0x0) CHECKPOINT : 0x0 WAIT_HINT : 0x0 PID : 3976 FLAGS :
Now Kill the Service:
taskkill /f /pid 3976
SUCESS: The process with PID 3976 has been terminated.
Check the Windows Application event log, it could contain some entries from your service's auto generated event source (which should have the same name of the service).
For me it was a while loop that looked at an external queue. The while-loop continued running until the queue was empty. Solved it by calling a timer event directly only when Environment.UserInteractive. Therefore the service could be debugged easily but when running as a service it would wait for the timers ElapsedEventHandler event.
Service:
partial class IntegrationService : ServiceBase
{
private static Logger logger = LogManager.GetCurrentClassLogger();
private System.Timers.Timer timer;
public IntegrationService()
{
InitializeComponent();
}
protected override void OnStart(string[] args)
{
try
{
// Add code here to start your service.
logger.Info($"Starting IntegrationService");
var updateIntervalString = ConfigurationManager.AppSettings["UpdateInterval"];
var updateInterval = 60000;
Int32.TryParse(updateIntervalString, out updateInterval);
var projectHost = ConfigurationManager.AppSettings["ProjectIntegrationServiceHost"];
var projectIntegrationApiService = new ProjectIntegrationApiService(new Uri(projectHost));
var projectDbContext = new ProjectDbContext();
var projectIntegrationService = new ProjectIntegrationService(projectIntegrationApiService, projectDbContext);
timer = new System.Timers.Timer();
timer.AutoReset = true;
var integrationProcessor = new IntegrationProcessor(updateInterval, projectIntegrationService, timer);
timer.Start();
}
catch (Exception e)
{
logger.Fatal(e);
}
}
protected override void OnStop()
{
try
{
// Add code here to perform any tear-down necessary to stop your service.
timer.Enabled = false;
timer.Dispose();
timer = null;
}
catch (Exception e)
{
logger.Fatal(e);
}
}
}
Processor:
public class IntegrationProcessor
{
private static Logger _logger = LogManager.GetCurrentClassLogger();
private static volatile bool _workerIsRunning;
private int _updateInterval;
private ProjectIntegrationService _projectIntegrationService;
public IntegrationProcessor(int updateInterval, ProjectIntegrationService projectIntegrationService, Timer timer)
{
_updateInterval = updateInterval;
_projectIntegrationService = projectIntegrationService;
timer.Elapsed += new ElapsedEventHandler(OnTimedEvent);
timer.Interval = _updateInterval;
//Don't wait for first elapsed time - Should not be used when running as a service due to that Starting will hang up until the queue is empty
if (Environment.UserInteractive)
{
OnTimedEvent(null, null);
}
_workerIsRunning = false;
}
private void OnTimedEvent(object source, ElapsedEventArgs e)
{
try
{
if (_workerIsRunning == false)
{
_workerIsRunning = true;
ProjectInformationToGet infoToGet = null;
_logger.Info($"Started looking for information to get");
//Run until queue is empty
while ((infoToGet = _projectIntegrationService.GetInformationToGet()) != null)
{
//Set debugger on logger below to control how many cycles the service should run while debugging.
var watch = System.Diagnostics.Stopwatch.StartNew();
_logger.Info($"Started Stopwatch");
_logger.Info($"Found new information, updating values");
_projectIntegrationService.AddOrUpdateNewInformation(infoToGet);
_logger.Info($"Completed updating values");
watch.Stop();
_logger.Info($"Stopwatch stopped. Elapsed seconds: {watch.ElapsedMilliseconds / 1000}. " +
$"Name queue items: {infoToGet.NameQueueItems.Count} " +
$"Case queue items: {infoToGet.CaseQueueItems.Count} " +
$"Fee calculation queue items: {infoToGet.FeeCalculationQueueItems.Count} " +
$"Updated foreign keys: {infoToGet.ShouldUpdateKeys}");
}
_logger.Info($"Nothing more to get from integration service right now");
_workerIsRunning = false;
}
else
{
_logger.Info($"Worker is already running! Will check back again after {_updateInterval / 1000} seconds");
}
}
catch (DbEntityValidationException exception)
{
var newException = new FormattedDbEntityValidationException(exception);
HandelException(newException);
throw newException;
}
catch (Exception exception)
{
HandelException(exception);
//If an exception occurs when running as a service, the service will restart and run again
if (Environment.UserInteractive)
{
throw;
}
}
}
private void HandelException(Exception exception)
{
_logger.Fatal(exception);
_workerIsRunning = false;
}
}
You can try to increase the windows service timeout with a key in the registry
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control
"ServicesPipeTimeout"=dword:300000 (300 seconds or 5 minutes)
If it doesn't exists it has to be created.