Develop Reference

C# Azure ServiceBus / Azure SQL - works locally, does not seem to work when deployed to Azure

Just starting to work with Azure.
Have a simple C# .NET Core app, which connects to Azure ServiceBus, reads messages, and writes them to Azure SQL database.
Works locally just fine - connects to remote Azure Service Bus Queue, reads messages, connect to remote Azure SQL db, writes records.
Same exact app, when deployed to Azure as a WebApp, appears to "run", but no longer reads messages from Services Bus, and no longer writes anything to Azure SQL.
Here is the entire app (i.e. Program.cs):
using System;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Azure.ServiceBus;
using System.Data.SqlClient;
namespace ServiceBusReader
{
class Program
{
const string ServiceBusConnectionString = "SB_CONNECTION_STRING";
const string QueueName = "BasicQueue";
static IQueueClient queueClient;
static SqlConnection connection = null;
public static async Task Main(string[] args)
{
System.Diagnostics.Trace.TraceError("Inside Main function...");
queueClient = new QueueClient(ServiceBusConnectionString, QueueName);
Console.WriteLine("======================================================");
System.Diagnostics.Trace.TraceError("======================================================");
Console.WriteLine("Press ENTER key to exit after receiving all the messages.");
Console.WriteLine("======================================================");
SqlConnectionStringBuilder builder = new SqlConnectionStringBuilder();
builder.DataSource = "XXXX.database.windows.net";
builder.UserID = "USERID";
builder.Password = "PASSWORD";
builder.InitialCatalog = "mySampleDatabase";
connection = new SqlConnection(builder.ConnectionString);
connection.Open();
// Register the queue message handler and receive messages in a loop
RegisterOnMessageHandlerAndReceiveMessages();
Console.ReadKey();
await queueClient.CloseAsync();
}
static void RegisterOnMessageHandlerAndReceiveMessages()
{
// Configure the message handler options in terms of exception handling, number of concurrent messages to deliver, etc.
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandler)
{
// Maximum number of concurrent calls to the callback ProcessMessagesAsync(), set to 1 for simplicity.
// Set it according to how many messages the application wants to process in parallel.
MaxConcurrentCalls = 1,
// Indicates whether the message pump should automatically complete the messages after returning from user callback.
// False below indicates the complete operation is handled by the user callback as in ProcessMessagesAsync().
AutoComplete = false
};
// Register the function that processes messages.
queueClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
}
static async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
// Process the message.
Console.WriteLine($"Received message: SequenceNumber:{message.SystemProperties.SequenceNumber} Body:{Encoding.UTF8.GetString(message.Body)}");
string query = "INSERT INTO [SalesLT].[messages] (message) VALUES(#Message)";
SqlCommand cmd = new SqlCommand(query, connection);
System.Diagnostics.Trace.TraceError(Encoding.UTF8.GetString(message.Body));
cmd.Parameters.AddWithValue("#Message", Encoding.UTF8.GetString(message.Body));
cmd.ExecuteNonQuery();
Console.WriteLine("Records Inserted Successfully...");
System.Diagnostics.Trace.TraceError("Records Inserted Successfully...");
// Complete the message so that it is not received again.
// This can be done only if the queue Client is created in ReceiveMode.PeekLock mode (which is the default).
await queueClient.CompleteAsync(message.SystemProperties.LockToken);
// Note: Use the cancellationToken passed as necessary to determine if the queueClient has already been closed.
// If queueClient has already been closed, you can choose to not call CompleteAsync() or AbandonAsync() etc.
// to avoid unnecessary exceptions.
}
// Use this handler to examine the exceptions received on the message pump.
static Task ExceptionReceivedHandler(ExceptionReceivedEventArgs exceptionReceivedEventArgs)
{
Console.WriteLine($"Message handler encountered an exception {exceptionReceivedEventArgs.Exception}.");
var context = exceptionReceivedEventArgs.ExceptionReceivedContext;
Console.WriteLine("Exception context for troubleshooting:");
Console.WriteLine($"- Endpoint: {context.Endpoint}");
Console.WriteLine($"- Entity Path: {context.EntityPath}");
Console.WriteLine($"- Executing Action: {context.Action}");
return Task.CompletedTask;
}
}
}
Should I do anything differently in this app in order to make it work in Azure?

It seems that you are trying to deploy a webjob to web app. May I know if you have set it to Continuous type? If you have set it to Continuous, the webjob will automatically run after deployment.
By default, the type is Triggered and you need to manually start the webjob from portal.

RabbitMQ channel leak when bus was failed to start

I have a problem with MassTransit (Using MassTransit 3.5.7 (via Nuget), RabbitMQ 3.6.10, Erlang 19.0)
Looks like that MassTransit not clear up RabbitMQ channels, when bus is failed to start.
Here is my test program
using System;
using System.Threading;
using MassTransit;
namespace TestSubscriber
{
class Program
{
static void Main()
{
IBusControl busControl = null;
var failCount = 0;
var busNotInitialised = true;
//Keep RabbitMQ switched off for a few iterations of this loop, then switch it on.
while (busNotInitialised)
{
busControl = Bus.Factory.CreateUsingRabbitMq(sbc =>
{
var host = sbc.Host(new Uri("rabbitmq://localhost/"), h =>
{
h.Username("guest");
h.Password("guest");
});
sbc.ReceiveEndpoint(host, "some_queue", endpoint =>
{
endpoint.Handler<string>(async context =>
{
await Console.Out.WriteLineAsync($"Received: {context.Message}");
});
});
});
try
{
busControl.Start();
busNotInitialised = false;
}
catch (Exception)
{
Console.WriteLine($"Attempt:{++failCount} failed.");
//wait some time
Thread.Sleep(5000);
}
}
//At this point, using RabbitMq's management web page, you will see failCount + 1 channels.
busControl.Stop();
//At this point, using RabbitMq's management web page, you will see failCount channels.
Console.ReadLine();
}
}
}
It continuously tries to create a service bus that uses RabbitMQ.
The program breaks out of the loop as soon as the service bus is created successfully.
After running the program for a couple of minutes (with stopped RabbitMQ) and stopping on a breakpoint, I can see a lot of worker threads (one for each failed service bus creation attempt).
After startup RabbitMQ, all of those "dangling" connect threads will connect to RabbitMQ.
If I try to shut down the bus, the latest connection ( that belongs to the bus that was successfully created), is closed. All of the other dangling connections are still connected to RabbitMQ.
Problem here is that those dangling threads, when connected, reads data from the queue which leads to data loss.
Is there any way to get around this problem?

How can I both Send and Receive from a Router socket in ZeroMQ or NetMQ?

I have a Dealer <--> Router setup in NetMQ v4 where I can asynchronously send and receive messages in any direction with no problems.
I now want to formalize that into an abstraction where the server (Router) listens for any incoming message but it also need to on demand broadcasts messages to any of the connected clients (Dealers).
I am trying to avoid using Pub <--> Sub sockets as I need the subscribers to also send messages to the server. The closest pattern to what I am trying to achieve is a WebSocket client-server communication.
The first part of listening to the client messages are done in something like:
using (var server = new RouterSocket("#tcp://*:80"))
{
var addresses = new HashSet<string>();
while (true)
{
var msg = server.ReceiveMultipartMessage();
var address = Encoding.UTF8.GetString(msg[0].Buffer);
var payload = Encoding.UTF8.GetString(msg[2].Buffer);
Console.WriteLine("[Server] - Client: {0} Says: {1}", address, payload);
var contains = addresses.Contains(address);
if (!contains) { addresses.Add(address); }
msg.Clear();
msg.Append(address);
msg.AppendEmptyFrame();
msg.Append("Reply for: " + address);
server.SendMultipartMessage(msg);
}
}
Now given that the sockets are not Thread-Safe, I am stuck on finding a way to broadcasts messages (coming from a different thread on demand) to all the clients.
I can probably use the TryReceiveMultipartMessage method in the loop instead with a set timeout after which I can check a queue for any broadcast messages and then loop through each client sending such message. Something like:
using (var server = new RouterSocket("#tcp://*:80"))
{
var addresses = new HashSet<string>();
var msg = new NetMQMessage();
while (true)
{
var clientHasMsg = server.TryReceiveMultipartMessage(TimeSpan.FromSeconds(1), ref msg);
if (!clientHasMsg)
{
// Check any incoming broacast then loop through all the clients
// sending each the brodcast msg
var broadMsg = new NetMQMessage();
foreach (var item in addresses)
{
broadMsg.Append(item);
broadMsg.AppendEmptyFrame();
broadMsg.Append("This is a broadcast");
server.SendMultipartMessage(broadMsg);
broadMsg.Clear();
}
// Go back into the loop waiting for client messages
continue;
}
var address = Encoding.UTF8.GetString(msg[0].Buffer);
var payload = Encoding.UTF8.GetString(msg[2].Buffer);
Console.WriteLine("[Server] - Client: {0} Says: {1}", address, payload);
var contains = addresses.Contains(address);
if (!contains) { addresses.Add(address); }
msg.Clear();
msg.Append(address);
msg.AppendEmptyFrame();
msg.Append("Reply for: " + address);
server.SendMultipartMessage(msg);
}
}
This somehow does not feel right mainly due to:
What value for the timeout is a good value? 1 sec, 100 ms etc;
Is this the most efficient/performing solution as this program will be used to have 100k+ clients connected with each sending thousands of messages per second.
Any pointers on what's the best approach to this is very much appreciated.

You can use netmqqueue, it multi producer single consumer queue. You can add it to NetMQPoller and enqueue from multiple threads without lock.

I think PUB/SUB is a proper approach for your requirements of 100k+ clients. Nevertheless, it doesn't mean that you can't communicate back to server: use DEALER/ROUTER. Why do you think this solution is not acceptable?

How can I consume RabbitMq messages with different type of consumers in .Net?

MessageType: "PublishX"
Consumers:
Type1ConsumerX
Type2ConsumerX
Type3ConsumerX
All of the consumers must catch messages immediately, but consume synchronously inside themselves..
For example there are 100 "PublishX" messages in the queue. Type1ConsumerX consumed 30 messages (synchronously), Type2ConsumerX consumed 50 messages(synchronously) , Type3ConsumerX consumed 100 messages(synchronously).
How can I know the message is consumed by "all type of consumers" ?
Could RabbitMQ/MassTransit PUSH messages to consumers?
Could RabbitMQ/MassTransit push messages (merging them) with intervals (1s) for decrease network traffic?
Could RabbitMQ/MassTransit push same messages to the different type of Consumers?

If I've understood the question correctly you just need to set up a basic pub/sub pattern. This will allow you to deliver the same message to multiple consumers.
Example publisher:
public static void PublishMessageToFanout()
{
var factory = new ConnectionFactory { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.ExchangeDeclare("messages", "fanout");
var message = new Message { Text = "This is a message to send" };
var json = JsonConvert.SerializeObject(message);
var body = Encoding.UTF8.GetBytes(json);
channel.BasicPublish("messages", string.Empty, null, body);
}
}
Example consumers:
SubscribeToMessages("sms-messages", (s) => Console.WriteLine("SMS Message: {0}", s));
SubscribeToMessages("email-messages", (s) => Console.WriteLine("Email Message: {0}", s));
public static void SubscribeToMessages(string queueName, Action<string> messageAction)
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.ExchangeDeclare("messages", "fanout");
channel.QueueDeclare(queueName, true, false, false, null);
channel.QueueBind(queueName, "messages", string.Empty);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume(queueName, true, consumer);
while (true)
{
var ea = consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
messageAction(message);
}
}
}
If you run the SubscribeToMessages loops in separate processes or console apps you'll see that they both print out the message whenever you call the PublishMessageToFanout. You'll also see that both queues exist in RabbitMQ Management under Queues.

Regarding the MassTransit part of your question
RabbitMQ/MassTransit PUSH messages to consumers?
Yes, MassTransit publishes messages to the bus, and then a consumer processes them
Could RabbitMQ/MassTransit push messages (merging them) with intervals (1s) for decrease network traffic?
Don't know if there is a feature for this, you could write your own but you would have to be very careful about losing the messages.
Could RabbitMQ/MassTransit push same messages to the different type of Consumers?
Yes, multiple consumers can consume the same type of message.
I've written a simple hello world app that shows the basics - http://nodogmablog.bryanhogan.net/2015/04/mass-transit-with-rabbitmq-hello-world/

Websphere MQ Queue Depth with XMS.Net

Since we are experiencing some trouble with IBM's Websphere MQ using XMS.net (Windows service that sometimes seems to give up listening for messages on a queue) we would like to create a simple application to monitor the depths of some queues (or number of messages on the queue) to be able to alert someone when the queue depth exceeds a certain threshold. This application would be launched by the task scheduler on a specific interval and would "read out" for X queues their queue depth (and maybe some other statistics).
Our windows service is using the following code and I was hoping I could reuse that same "knowledge" for our "monitoring" application.
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);
//Read config values
string QueueManager = ConfigurationManager.AppSettings["queuemanager"];
string Channel = ConfigurationManager.AppSettings["channel"];
string Queue = ConfigurationManager.AppSettings["queue"];
string HostIP = ConfigurationManager.AppSettings["host"];
int Port = int.Parse(ConfigurationManager.AppSettings["port"]);
//Create connection
var factoryfactory = XMSFactoryFactory.GetInstance(XMSC.CT_WMQ);
var connectionfactory = factoryfactory.CreateConnectionFactory();
connectionfactory.SetStringProperty(XMSC.WMQ_QUEUE_MANAGER, QueueManager);
connectionfactory.SetStringProperty(XMSC.WMQ_HOST_NAME, HostIP);
connectionfactory.SetIntProperty(XMSC.WMQ_PORT, Port);
connectionfactory.SetStringProperty(XMSC.WMQ_CHANNEL, Channel);
connectionfactory.SetIntProperty(XMSC.WMQ_BROKER_VERSION, XMSC.WMQ_BROKER_V2);
connectionfactory.SetIntProperty(XMSC.WMQ_CONNECTION_MODE, XMSC.WMQ_CM_CLIENT_UNMANAGED);
Console.WriteLine("Creating connection");
var connection = connectionfactory.CreateConnection();
connection.ExceptionListener = new ExceptionListener(OnXMSExceptionReceived);
//Create a_session
Console.WriteLine("Creating sessions");
var session = connection.CreateSession(false, AcknowledgeMode.ClientAcknowledge);
//Create queue
Console.WriteLine("Creating queue");
var queue = session.CreateQueue(string.Format("queue://{0}/{1}", QueueManager, Queue));
I have browsed through the properties of session, queue etc. but, ofcourse, there are no "current queue depth" properties. I could use GetIntProperty() or GetLongProperty() on these objects but I don't know which constant to use for that (I have seen IBM.XMS.MQC.MQIA_CURRENT_Q_DEPTH but that contains an int and Get...Property() expects a string as parameter).
Long story short: how would I go about retrieving a queues depth with the above code as a starting-point? Is it at all possible using XMS.Net?

I was able to solve it using, as Shashi suggested, the MQ API. For this you need to reference amqmdnet.dll (C:\Program Files (x86)\IBM\WebSphere MQ\bin\amqmdnet.dll) and use the following (example) code. Please note that this is a simple example, no exception handling etc. is included.
using System;
using System.Collections;
using System.Configuration;
using IBM.WMQ;
namespace Test
{
class Program
{
static void Main(string[] args)
{
//Connection properties
var properties = new Hashtable();
properties.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_CLIENT);
properties.Add(MQC.CHANNEL_PROPERTY, "SOME.CHANNEL.TCP");
properties.Add(MQC.HOST_NAME_PROPERTY, "12.34.56.78");
properties.Add(MQC.PORT_PROPERTY, 1416);
var qmgr = new MQQueueManager("MYQMGR", properties);
Console.WriteLine("Local : {0}", GetQueueDepth(qmgr, "FOO.LOCALQ"));
Console.WriteLine("Report : {0}", GetQueueDepth(qmgr, "FOO.REPORTQ"));
}
public static int GetQueueDepth(MQQueueManager queuemgr, string queue)
{
return queuemgr.AccessQueue(queue,
MQC.MQOO_INPUT_AS_Q_DEF +
MQC.MQOO_FAIL_IF_QUIESCING +
MQC.MQOO_INQUIRE).CurrentDepth;
}
}
}
This performs way better than my initial "workaround".

Using XMS .NET queue depth can't be determined. Queue depth is specific to messaging providers and not JMS/XMS, so you will need to use MQ APIs to get the queue depth. You could use MQ .NET API to find the queue depth. MQQueue.CurrentDepth will give the number of message in the queue.
IMO it would be good to investigate why XMS .NET service stopped listening for messages than write another program to monitor queue depth.