Develop Reference

Azure Service Bus Receive Messages continuously when ever new message placed in web application [duplicate]

I am using Azure.Messaging.ServiceBus nuget package to work with Azure service bus. We have created a topic and a subscription. The subscription has 100+ messages. We want to read all the message and continue to read message as they arrive.
Microsoft.Azure.ServiceBus package (deprecated now) provided RegisterMessageHandler which use to process every incoming message. I am not able to find similar option under Azure.Messaging.ServiceBus nuget package.
I am able to read one message at a time but I have to call await receiver.ReceiveMessageAsync(); every time manually.

To receive multiple messages (a batch), you should use ServiceBusReceiver.ReceiveMessagesAsync() (not plural, not singular 'message'). This method will return whatever number of messages it can send back. To ensure you retrieve all 100+ messages, you'll need to loop until no messages are available.
If you'd like to use a processor, that's also available in the new SDK. See my answer to a similar question here.

As suggested by #gaurav Mantri, I used ServiceBusProcessor class to implement event based model for processing messages
public async Task ReceiveAll()
{
string connectionString = "Endpoint=sb://sb-test-today.servicebus.windows.net/;SharedAccessKeyName=manage;SharedAccessKey=8e+6SWp3skB3Aedsadsadasdwz5DU=;";
string topicName = "topicone";
string subscriptionName = "subone";
await using var client = new ServiceBusClient(connectionString, new ServiceBusClientOptions
{
TransportType = ServiceBusTransportType.AmqpWebSockets
});
var options = new ServiceBusProcessorOptions
{
// By default or when AutoCompleteMessages is set to true, the processor will complete the message after executing the message handler
// Set AutoCompleteMessages to false to [settle messages](https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-transfers-locks-settlement#peeklock) on your own.
// In both cases, if the message handler throws an exception without settling the message, the processor will abandon the message.
AutoCompleteMessages = false,
// I can also allow for multi-threading
MaxConcurrentCalls = 1
};
await using ServiceBusProcessor processor = client.CreateProcessor(topicName, subscriptionName, options);
processor.ProcessMessageAsync += MessageHandler;
processor.ProcessErrorAsync += ErrorHandler;
await processor.StartProcessingAsync();
Console.ReadKey();
}
public async Task MessageHandler(ProcessMessageEventArgs args)
{
string body = args.Message.Body.ToString();
Console.WriteLine(body);
// we can evaluate application logic and use that to determine how to settle the message.
await args.CompleteMessageAsync(args.Message);
}
public Task ErrorHandler(ProcessErrorEventArgs args)
{
// the error source tells me at what point in the processing an error occurred
Console.WriteLine(args.ErrorSource);
// the fully qualified namespace is available
Console.WriteLine(args.FullyQualifiedNamespace);
// as well as the entity path
Console.WriteLine(args.EntityPath);
Console.WriteLine(args.Exception.ToString());
return Task.CompletedTask;
}

How to switch a ConnectionHandler to UDP

My ASP.Net Core application provides a TCP listener, implemented with a custom ConnectionHandler, to receive binary data from another process (let's call it Datasource) on another host. This data is then sent to the browser through a WebSocket (called DataSink in the code).
Since the process Datasource has changed from a single TCP connection to UDP datagrams, I need to adapt (its internals are out of my reach).
How can I switch the current implementation to an UDP listener? Is there a canonical way how this is done with ASP.Net Core?
public class MySpecialConnectionHandler : ConnectionHandler
{
private readonly IMyDataSink DataSink;
public MySpecialConnectionHandler(IMyDataSink dataSink)
{
DataSink = dataSink;
}
public override async Task OnConnectedAsync(ConnectionContext context)
{
TransportConnection connection = context as TransportConnection;
Console.WriteLine("new connection: " + connection.RemoteAddress + ":" + connection.RemotePort);
while (true)
{
var result = await connection.Transport.Input.ReadAsync().ConfigureAwait(false);
var buffer = result.Buffer;
foreach (var segment in buffer)
{
await DataSink.RelayData(segment.Span.ToArray()).ConfigureAwait(false);
}
if (result.IsCompleted)
{
break;
}
connection.Transport.Input.AdvanceTo(buffer.End);
}
Console.WriteLine(connection.ConnectionId + " disconnected");
}
}
The UDP listener must be available while the ASP.Net Core application is running.
EDIT:
Order and reliability of the datagram transmission is not that important (perhaps not at all), since the transmitted data is a MPEG1-stream multiplexed into MPEG-TS. The data source is on the first host, the ASP.Net Core application is on a second host and the receiver / consumer is a third host. The host creating the stream and the receiving process on the third host are in separate networks. The ASP.Net Core application acts as a relay. The sender is sending all time, but does not care about whether the data is received or not.
EDIT 2:
The main problem right now is where to put the UdpClient. The previous implementation (back when we used TCP) configured the Kestrel server for additional TCP listening and used the already presented ConnectionHandler:
return WebHost.CreateDefaultBuilder(args)
.UseStartup<Startup>()
.ConfigureKestrel((_, options) =>
{
// HTTP
options.Listen(networkInterface, httpPort);
// HTTPS
options.Listen(networkInterface, httpsPort, builder => builder.UseHttps());
// stream sink
options.Listen(streamInterface, streamPort, builder => builder.UseConnectionHandler<MySpecialConnectionHandler >());
});
The ConnectionHandler accepts the incoming TCP connection and then forwards the streaming data to a number of connected WebSockets. Since this is not usable with UDP datagrams, I need to place the UdpClient somewhere where it continuously (i.e. while(true)) receives datagrams and forwards them to the WebSockets. My problem is that I don't know where to put it, run the background thread and have the communication span threads without having any problems with this inter-thread data flow (like race conditions).

So, to conclude this:
We used a combination of a BackgroundWorker with an UdpClient. The BackgroundWorker is only instantiated when there is at least one receiver:
StreamReceiver = new BackgroundWorker();
StreamReceiver.DoWork += ReceiveStream;
StreamReceiver.RunWorkerAsync();
ReceiveStream is a private method that establishes the UdpClient and then waits for incoming data that needs to be relayed.
private async void ReceiveStream(object sender, DoWorkEventArgs e)
{
// DataSinkPort is a const int
UdpClient datasource = new UdpClient(_DataSinkPort);
while (true)
{
var rec = await datasource.ReceiveAsync();
await RelayData(rec.Buffer);
if (_CancellationToken.IsCancellationRequested)
{
return;
}
}
}
The method RelayData just uses the outgoing TCP connection of each subscribed receiver.

Correct pattern for ZeroMQ Server/Client type

I have a server that needs to get instructions to run processes for clients on another machine.
The clients send a job message, the Server processes the job and later sends the back results.
I tried using the NetMQ Request-Response pattern (see below)
This works nicely for 1 client, BUT if a second client sends a request before previous client job is finished - I get an error.
I really need to be able to receive ad-hoc messages from clients, and send results when they are completed. Clearly, I am using the wrong pattern, but reading the ZeroMQ docs has not highlighted a more appropriate one.
namespace Utils.ServerMQ
{
class ServerMQ
{
public static void Go()
{
using (var responseSocket = new ResponseSocket("#tcp://*:393"))
{
while (true)
{
Console.WriteLine("Server waiting");
var message = responseSocket.ReceiveFrameString();
Console.WriteLine("Server Received '{0}'", message);
//System.Threading.Thread.Sleep(1000);
var t2 = Task.Factory.StartNew(() =>
{
RunProcMatrix(message, responseSocket);
});
}
}
}
public static void RunProcMatrix(object state, ResponseSocket responseSocket)
{
var process = new Process
{
StartInfo = new ProcessStartInfo
{
FileName = Path.Combine(#"H:\Projects\Matrix\Matrix\bin\Debug\", "Matrix001.exe"),
Arguments = (string)state,
WindowStyle = ProcessWindowStyle.Normal,
CreateNoWindow = false
}
};
process.Start();
process.WaitForExit();
responseSocket.SendFrame((string)state);
}
}
}

You want a ROUTER socket on the server side, so it can receive multiple requests at a time. (Guide) REQ sockets on the client side are still fine unless the server may arbitrarily push data to them, then they need to be DEALER sockets.
Note that for sockets beyond REQ/RESP you need to manually handle the message envelope (the first frame of the message indicating its destination). Guide
The 0MQ docs are incredibly dense... I don't blame you for not intuiting this from them :)
This example from the NetMQ docs is full ROUTER-DEALER: https://netmq.readthedocs.io/en/latest/router-dealer/#router-dealer, you can take just the router side and it should work the same though.

Request-Response logic to allow non-response messages to be processed

I have a socket application that allows thousands of clients to connect. It stores them in a ConcurrentDictionary<int, Socket> and operates solely for request-response situations:
When I need data, I find the relevant socket and send a request, asking for the data I need.
After sending the request, I receive bytes until it sends the response. Then I stop receiving.
Like this:
public Task<Message> Request(int clientId, Message message)
{
Socket client;
return Clients.TryGetValue(clientId, out client)
? RequestInternal(client, message);
: _EmptyTask;
}
public async Task<Message> RequestInternal(Socket client, Message message)
{
await SendAsync(client, message).ConfigureAwait(false);
return await ReceiveOneAsync(client).ConfigureAwait(false);
}
Now I need to change this application to allow clients to send me anything, anytime; even without I make a request for it. Which -I think- will require constantly receiving from the sockets and a completely different approach.
Questions:
What are the bets-known approaches (best-practices) for this kind of applications?
Any gotchas you can tell me about or any guides you can point me to?
What I have in mind:
Disclaimer: This part is a little long and completely hypothetical. You can skip that if you have an answer to the above questions.
What I have in mind:
Receiving bytes constantly and adding the assembled PDUs to a BlockingCollection<Message>.
Creating a thread, dedicated to process received messages using BlockingCollection's GetConsumingEnumerable method.
The processing thread will do this:
foreach (var message in Messages.GetConsumingEnumerable())
ProcessMessage(message);
With this I can receive and process everything the clients send but distinguishing the messages that are sent to reply my requests from the messages that are sent because the client needed to would be an issue.
I think I can send a unique identifier byte (unique to that particular client) with the request. Then the client can send that identifier back to me in its response and I can use it to distinguish the response.
ProcessMessage(Message msg)
{
// msg is a message from msg.Sender.
if (msg.Id == 0)
{
// msg is not a response, do processing.
}
else
{
// msg is a response to the message that's sent with msg.Id.
// Find the request that:
// * ...is made to msg.Sender
// * ...and has the msg.Id as identifier.
// And process the response according to that.
}
}
This means I also have to store the requests.
Here is an hypothetical version of RequestInternal:
Edit: Replaced Wait calls with awaits after Stephen Cleary's answer.
private async Task RequestInternal(Socket client, Message message)
{
var request = new Request(client, message);
Requests.Add(request);
await SendAsync(client, message).ConfigureAwait(false);
return await request.Source.Task.ConfigureAwait(false);
}
And the Request class:
private sealed class Request
{
public readonly byte Id;
public readonly Socket Client;
public readonly Message Message;
public readonly TaskCompletionSource<Message> Source;
public Request(Socket client, Message message)
{
Client = client;
Message = message;
Source = new TaskCompletionSource<Message>();
// Obtain a byte unique to that socket...
Id = GetId(client);
}
}
And ProcessMessage becomes this:
ProcessMessage(Message msg)
{
if (msg.Id == 0)
OnReceived(msg); // To raise an event.
else
{
// Method to find a request using msg.Sender and msg.Id
var request = Requests.Find(msg);
if (request != null)
request.Source.SetResult(msg);
}
}
Although I have no idea what kind of collection type Requests would be.
Edit: I've used a ConcurrentDictionary<Key, Request> where Key is a private struct with an Int32 (id of the socket) and a Byte (id of the message) fields. it also implements IEquatable<T>.

I wrote a TCP/IP .NET Sockets FAQ a few years ago that addresses some common problems (such as message framing, continuous reading, and explanations of common errors). The code samples all use the Socket class, but the same concepts apply to all TCP/IP sockets.
Regarding your protocol design and request/response matching, the overall approach sounds good. You'll need to ensure you're threadsafe (e.g., Requests would probably be a ConcurrentDictionary). Also, you should await SendAsync rather than calling Wait.
An alternative approach that I've played around with but haven't put into production is based on TPL Dataflow. You can create one block that represents the "output" for each client and another block for the "input". Then you can layer your message framing on that, and layer your request/response matching on that, and then send any remaining (unsolicited) messages to a single shared BufferBlock.
So your "end-user" API would end up looking like this:
// Send a request and asynchronously receive a matching response.
Task<Message> RequestAsync(int clientId, Message message);
// Endpoint for unsolicited messages.
IReceivableSourceBlock<Tuple<int, Message>> UnsolicitedMessages { get; }
You could then hook up an ActionBlock to UnsolicitedMessages to execute a delegate whenever one comes in.

WebSockets - How to create different messages?

I am creating a websocket chat application and I managed to relay chat messages to other browsers connected. I have a console application listening on one port.
My question is... If one person logs on to the system I want everybody to know that, how can I do that? I'm using Linq to map the DB but if the logging is ok how do I send that message, that user X has logged in?
FINALLY I was able to create a chatroom using websockets, here is the final product, thanks for the orientation!
http://ukchatpoint.no-ip.org/Chatpoint/Pages/Uklobby.aspx

First make sure you're sending messages as JSON (JavaScript Object Notation) as this allows structured data to be sent back and forth, and client & server can differentiate between a chat message and an instruction (e.g. someone new logged in). For instance on the client:
mySocket.onmessage = function(event) {
var command = JSON.parse(event.data);
if(command.type === 'message') {
var message = command.message;
// handle chat message
}
else if (command.type === 'newUser') {
var username = command.username;
// handle new user
}
};
On the server in ASP.NET C# you'd send them as:
public class ChatHandler : WebSocketHandler
{
private JavaScriptSerializer serializer = new JavaScriptSerializer();
private static WebSocketCollection chatapp = new WebSocketCollection();
public override void OnMessage(string message)
{
var m = serializer.Deserialize<Message>(message);
switch (m.Type)
{
case MessageType.NewUser:
chatapp.Broadcast(serializer.Serialize(new
{
type = "newUser",
username = m.username
}));
break;
case MessageType.Message:
chatapp.Broadcast(serializer.Serialize(new
{
type = "message",
message = m.message
}));
break;
default:
return;
}
}
}
As Hightechrider says, you'll need to keep track of a list of connected clients, that's what WebSocketCollection class does in the code listing above.
Check out Paul Batum's WebSocket chat example on github here (https://github.com/paulbatum/BUILD-2011-WebSocket-Chat-Samples/blob/master/BasicAspNetChat/ChatHandler.cs)
Also he did a presentation at the recent MS BUILD conference here (http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-807T)

You would need to track the connections at the application level so you can send to all of them. But take a look at SignalR instead where a lot of the work involved with webSockets and long polling is being written for you. With SignalR you can use GetClients to get all the clients connected to a Hub.

When using PostgreSQL, you could use NOTIFY from within the database to notify the application layer, which could generate messages sent via WebSockets.