I have a MQTT calls inside a loop and in each iteration, it should return a response from the subscriber so that I could use the value being forwarded after I published. But the problem is I don't know how would I do it.
I hope you have an idea there or maybe if I'm just not implementing it right, may you guide me through this. Thanks.
Here's my code:
// MyClientMgr
class MyClientMgr{
public long CurrentOutput { get; set; }
public void GetCurrentOutput(MyObjectParameters parameters, MqttClient client)
{
MyMessageObject msg = new MyMessageObject
{
Action = MyEnum.GetOutput,
Data = JsonConvert.SerializeObject(parameters)
}
mq_GetCurrentOutput(msg, client);
}
private void mq_GetCurrentOutput(MyMessageObject msg, MqttClient client)
{
string msgStr = JsonConvert.SerializeObject(msg);
client.Publish("getOutput", Encoding.UTF8.GetBytes(msgStr),
MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, false);
client.MqttMsgPublishReceived += (sender, e) =>{
MyObjectOutput output = JsonConvert.DeserializeObject<MyObjectOutput>(Encoding.UTF8.GetString(e.Message));
CurrentOutput = output;
};
}
}
// MyServerMgr
class MyServerMgr
{
public void InitSubscriptions()
{
mq_GetOutput();
}
private void mq_GetOutput()
{
MqttClient clientSubscribe = new MqttClient(host);
string clientId = Guid.NewGuid().ToString();
clientSubscribe.Connect(clientId);
clientSubscribe.Subscribe(new string[] { "getOutput" }, new byte[] { MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE });
MqttClient clientPublish = new MqttClient(host);
string clientIdPub = Guid.NewGuid().ToString();
clientPublish.Connect(clientIdPub);
clientSubscribe.MqttMsgPublishReceived += (sender, e) => {
MyMessageObj msg = JsonConvert.DeserializeObject<MyMessageObj>(Encoding.UTF8.GetString(e.Message));
var output = msg.Output;
clientPublish.Publish("getOutput", Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(output)), MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, false);
}
}
}
// MyCallerClass
class MyCallerClass
{
var host = "test.mqtt.org";
var myServer = new MyServerMgr(host);
var myClient = new MyClientMgr();
myServer.InitSubscriptions();
MqttClient client = new MqttClient(host);
for(int i = 0; i < 10; i++)
{
long output = 0;
MyObjectParameters parameters = {};
myClient.GetCurrentOutput(parameters, client) // here I call the method from my client manager
// to publish the getting of the output and assigned
// below for use, but the problem is the value doesn't
// being passed to the output variable because it is not
// yet returned by the server.
// Is there a way I could wait the process to
// get the response before assigning the output?
output = myClient.CurrentOutput; // output here will always be null
// because the response is not yet forwarded by the server
}
}
I have a loop in my caller class to call the mqtt publish for getting the output, but I have no idea how to get the output before it was assigned, I want to wait for the response first before going to the next.
I've already tried doing a while loop inside like this:
while(output == 0)
{
output = myClient.CurrentOutput;
}
Yes, I can get the output here, but it will slow down the process that much. And sometimes it will fail.
Please help me. Thanks.
It looks like you are trying to do synchronous communication over an asynchronous protocol (MQTT).
By this I mean you want to send a message and then wait for a response, this is not how MQTT works as there is no concept of a reply to a message at the protocol level.
I'm not that familiar with C# so I'll just give an abstract description of possible solution.
My suggestion would be to use a publishing thread, wait/pulse (Look at the Monitor class) to have this block after each publish and have the message handler call pulse when it has received the response.
If the response doesn't contain a wait to identify the original request you will also need a state machine variable to record which request is in progress.
You may want to look at putting a time out on the wait in case the other end does not respond for some reasons.
You can use AutoResetEvent class that has WaitOne() and Set() methods. Using WaitOne() after publish will wait until the message is published and using Set() under client_MqttMsgPublishReceived event will release the wait when the subscriber received the message he subscribed for.
QueueClient.OnMessage takes a callback, Action<BrokeredMessaged>, as an argument that will be executed by an internal message pump that's constantly polling a queue (or subscription) when a message is available.
I've been looking at the BrokeredMessage type in Reflector but can't find a way to get the queue name that the message came from the BrokeredMessage object (that last part is key). If this is possible, how can it be pulled out?
Finally figured out a solution using reflection:
public void OnMessageCallback(BrokeredMessage message) {
var context = message.GetType().GetProperty("ReceiveContext", BindingFlags.NonPublic | BindingFlags.Instance).GetValue(message);
var receiver = (MessageReceiver)context.GetType().GetProperty("MessageReceiver", BindingFlags.Public | BindingFlags.Instance).GetValue(context);
var queueName = receiver.Path;
}
If you are using the QueueClient.OnMessage, you can do something like that:
var client = QueueClient.CreateFromConnectionString("MyConnectionString");
client.OnMessage(message =>
{
// You always have access to the queue path
var queueName = client.Path;
});
If you don't want to use anonymous function you can pass the queueName to the function that is going to process you message:
public void ProcessMessage(BrokeredMessage message, string queueName)
{
}
And call you function like that:
var client = QueueClient.CreateFromConnectionString("MyConnectionString");
client.OnMessage(message =>
{
ProcessMessage(message , client.Path);
});
EDIT : Using a MessageReceiver
Azure ServiceBus SDK provides an abstraction to receive messages from queues or subscriptions:
var messagingFactory = MessagingFactory.CreateFromConnectionString("MyConnectionString");
var messageReceiver = messagingFactory.CreateMessageReceiver("MyQueueName");
messageReceiver.OnMessage(message =>
{
// You always have access to the queue path
var queueName = messageReceiver.Path;
}, new OnMessageOptions());
I want to check the azure servicebus/iothub constantly for messages. However, when I do it like this I get the following error
"An exception of type 'Amqp.AmqpException' occurred in mscorlib.dll but was not handled in user code
Additional information: Operation 'Receive' is not valid under state: End."
Any ideas how I should implement constant pulling of messages and/or resolve this error?
var connection = new Connection(address);
var session = new Session(connection);
var entity = Fx.Format("/devices/{0}/messages/deviceBound", _deviceId);
var receiveLink = new ReceiverLink(session, "receive-link", entity);
while (true)
{
await Task.Delay(1000);
var message = await receiveLink.ReceiveAsync();
if (message == null) continue;
//else do things with message
}
From the endpoint you're using it looks like you're talking about receiving cloud-to-device (c2d) messages, in other words, the code you're writing runs on the device, and is meant to receive messages sent through the service to this device, right?
The simplest way of doing this is using the DeviceClient class of the C# SDK. An example of how to use this class is provided in the DeviceClientAmqpSample project.
Once you create your DeviceClient instance using your device connection string, the DeviceClient class has a ReceiveAsync method that can be used to receive messages.
var deviceClient = DeviceClient.CreateFromConnectionString("<DeviceConnectionString>");
while(true)
{
await Task.Delay(1000);
var receivedMessage = await deviceClient.ReceiveAsync(TimeSpan.FromSeconds(1));
if (receivedMessage != null)
{
var messageData = Encoding.ASCII.GetString(receivedMessage.GetBytes());
Console.WriteLine("\t{0}> Received message: {1}", DateTime.Now.ToLocalTime(), messageData);
await deviceClient.CompleteAsync(receivedMessage);
}
}
I have some code which rolls through a list of ports and sets up a stream containing the UDP Buffer contents which can be subscribed to.
I'm pretty new to RX and this was built with some help in response to another stackoverflow question.
This code seemed to be OK in early tests and handled all the packets we could send it but now it's in soak test the subscription seems to fail after a while and it stops sending events to the event hub.
I'm particularly worried that I have an exception somewhere in the subscription code which causes me to lose the information but isn't caught.
Here is the code, any comments or suggestions for simplification or improvement would be greatly appreciated:
var receiveStream =
_deviceTypeProvider.GetDeviceTypes().ToObservable().Trace("GetDeviceTypes")
.SelectMany(devicePortMapping => Observable
.Using(() => UdpListener(devicePortMapping),
client =>
Observable
.FromAsync(client.ReceiveAsync)
.Repeat()).Trace("UdpListener"),
(devicePortMapping, stream) =>
{
Log
.ForContext("Raw", stream.Buffer.ToPrintByteArray())
.Verbose("Received Incoming {DeviceType} Message from {Device} on Port {Port}",
devicePortMapping.DeviceType, stream.RemoteEndPoint.Address, devicePortMapping.Port);
try
{
var timeZoneOffset =
_deviceTimeZoneOffsetProvider.GetDeviceTimeZoneOffset(devicePortMapping.Port);
var tenant = _deviceTenantProvider.GetDeviceTenant(devicePortMapping.Port);
if (tenant == null || timeZoneOffset == null)
{
Log
.Error(
"Tenant or TimeOffset Missing for Port: {Port}, cannot continue processing this message",
devicePortMapping.Port);
return null;
}
var message =
new DeviceMessage(new Device(stream.RemoteEndPoint.Address.ToIPAddressNum(),
stream.RemoteEndPoint.Port, devicePortMapping.DeviceType, tenant.TenantId,
timeZoneOffset.Offset))
{
MessageUid = Guid.NewGuid(),
Timestamp = DateTime.UtcNow,
Raw = stream.Buffer,
};
message.Information("Received Incoming Message");
return message;
}
catch (Exception ex)
{
Log.Error(ex, "Exception whilst receiving incoming message");
throw;
}
}).Trace("SelectMany").Select(Task.FromResult).Trace("Select");
if (_settings.TestModeEnabled)
{
Log
.Warning("Test Mode is Enabled");
receiveStream = receiveStream
.Select(async message =>
await _testModeProvider.InjectTestModeAsync(await message)).Trace("TestMode");
}
_listener = receiveStream.Subscribe(async messageTask =>
{
var message = await messageTask;
if (message == null)
{
Log
.Warning("Message is null, returning");
return;
}
Log
.ForContext("Raw", message.Raw.ToPrintByteArray(), true)
.ForContext("Device", message.Device, true)
.Verbose("Publishing Message {MessageUid} from {#Device}", message.MessageUid, message.Device);
await _messagePublisher.Publish(message).ConfigureAwait(false);
}, error => { Log.Error(error, "Exception whilst publishing message"); });
Here is the InjectTestMode method:
public async Task<DeviceMessage> InjectTestModeAsync(DeviceMessage deviceMessage)
{
try
{
var imei = GetImei(deviceMessage.Raw);
if (string.IsNullOrEmpty(imei))
{
Log
.ForContext("DeviceMessage",deviceMessage,true)
.Error("Error while getting IMEI value from message raw data in Test Mode");
return null;
}
//var dummyIpAddress = DummyIPfromIMEI(imei).ToIPAddressNum();
var mapping = await _mappingService.GetIPMappingAsync(deviceMessage.Device.IPAddress);
if (mapping == null)
{
Log
.ForContext("DeviceMessage", deviceMessage, true)
.Warning("Test Mode updated IP Address mapping with IPAddress: {IPAddress} for IMEI: {IMEI}", deviceMessage.Device.IPAddress.ToIPAddressString(), imei);
await _mappingService.UpdateIPMappingAsync(deviceMessage.Device.IPAddress, imei);
}
// deviceMessage.Device.IPAddress = dummyIpAddress;
return deviceMessage;
}
catch (Exception ex)
{
Log
.ForContext("DeviceMessage",deviceMessage,true)
.Error("Exception raised whilst injecting Test Mode", ex);
return null;
}
}
Here is the UdpListener method:
private UdpClient UdpListener(DeviceTypeMap deviceTypeMap)
{
Log.Information("Listening for Device Type: {DeviceType} messages on Port: {Port}", deviceTypeMap.DeviceType,
deviceTypeMap.Port);
var udpClient = new UdpClient();
var serverEndpoint = new IPEndPoint(IPAddress.Any, deviceTypeMap.Port);
udpClient.Client.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
udpClient.Client.Bind(serverEndpoint);
return udpClient;
}
[Update] - 16/11/2015
So I've done some research and it would seem I have more than one code smell in this so I've updated the code and it's now running but I thought I'd share the intent of this code and well as the revised code to see if someone can suggest a more elegant solution.
The Intent
Listen to several UDP Ports and send the traffic to Azure Event Hubs along with some meta based on the port it was received on.
This meta would depend on whether the system was in a 'Test Mode' or not.
The Implementation
For each Port and Device Type create a UDPListener and Observable Collection from the ReceiveAsync event, combine these Observable Collections into a single collection which could then be subscribed from component which would publish the data to EventHubs.
The Problems
The InjectMode was async, and this could return a null if there was a problem, this seemed to be killing the sequence. I think that this probably should have been some sort of Observable Extension Method which allow as to modify or remove the device message from the sequence but I couldn't figure this out.
Originally the publish to EventHub was in the subscription until I read that that you shouldn't use async inside a subscription as it generates a async void which is bad. All the research seemed to point to pushing this into a SelectMany which did'nt make any sense since this was the destination of the observed sequence not part of the process but I went with this. The subscription in effect became redundant.
I not sure that all of the try catch blocks were required but I was convinced that I had a problem which was disrupted the sequence. As noted by Enigmativity I made these all Exception catches and logged then and re-throw, nothing ever appeared from these log entries.
Retry().Do() doesn't feel right, I could get SelectMany() as suggested in many other posts to work so I had no choice.
Here's the code which is running now :
public void Start()
{
Log.Information("InboundUdpListener is starting");
var receiveStream =
_deviceTypeProvider.GetDeviceTypes().ToObservable().Trace("GetDeviceTypes")
.SelectMany(devicePortMapping => Observable
.Using(() => UdpListener(devicePortMapping),
client =>
Observable
.FromAsync(client.ReceiveAsync)
.Repeat()).Trace("UdpListener"),
(devicePortMapping, stream) =>
{
Log
.ForContext("Raw", stream.Buffer.ToPrintByteArray())
.Verbose("Received Incoming {DeviceType} Message from {Device} on Port {Port}",
devicePortMapping.DeviceType, stream.RemoteEndPoint.Address, devicePortMapping.Port);
try
{
var timeZoneOffset =
_deviceTimeZoneOffsetProvider.GetDeviceTimeZoneOffset(devicePortMapping.Port);
var tenant = _deviceTenantProvider.GetDeviceTenant(devicePortMapping.Port);
if (tenant == null || timeZoneOffset == null)
{
Log
.Error(
"Tenant or TimeOffset Missing for Port: {Port}, cannot continue processing this message",
devicePortMapping.Port);
return null;
}
var message =
new DeviceMessage(new Device(stream.RemoteEndPoint.Address.ToIPAddressNum(),
stream.RemoteEndPoint.Port, devicePortMapping.DeviceType, tenant.TenantId,
timeZoneOffset.Offset))
{
MessageUid = Guid.NewGuid(),
Timestamp = DateTime.UtcNow,
Raw = stream.Buffer,
};
message.Information("Received Incoming Message");
return message;
}
catch (Exception ex)
{
Log.Error(ex, "Exception whilst receiving incoming message");
throw;
}
}).Trace("SelectMany");
receiveStream = receiveStream.Retry().Do(async message =>
{
try
{
if (_testModeEnabled && message != null)
{
message = await _testModeProvider.InjectTestModeAsync(message);
}
if (message != null)
{
await _messagePublisher.Publish(message);
}
}
catch (Exception ex)
{
Log.Error(ex, "Exception whilst publishing incoming message");
throw;
}
}).Trace("Publish");
_listener = receiveStream.Retry().Subscribe(OnMessageReceive, OnError, OnComplete);
Log.Information("InboundUdpListener is started");
}
Can anyone see any issues with this code or suggest any improvements. I really would appreciate some help with this.
[Update Following Lee's Comment]
I totally agree that it was a mess and to show I'm willing to learn take on board people's help this is my next attempt
public void Start()
{
_listener = _deviceTypeProvider.GetDeviceTypes().ToObservable()
.SelectMany(CreateUdpListener, CreateMessage)
.SelectMany(InjectTestMode)
.SelectMany(PublishMessage)
.Retry()
.Subscribe(OnMessageReceive, OnError, OnComplete);
}
private IObservable<UdpReceiveResult> CreateUdpListener(DeviceTypeMap deviceType)
{
return Observable.Using(() => UdpListener(deviceType),
client => Observable.FromAsync(client.ReceiveAsync).Repeat());
}
private DeviceMessage CreateMessage(DeviceTypeMap deviceTypeMap, UdpReceiveResult receiveResult)
{
var timeZoneOffset =
_deviceTimeZoneOffsetProvider.GetDeviceTimeZoneOffset(deviceTypeMap.Port);
var tenant = _deviceTenantProvider.GetDeviceTenant(deviceTypeMap.Port);
if (tenant == null || timeZoneOffset == null)
{
Log
.Error(
"Tenant or TimeOffset Missing for Port: {Port}, cannot continue processing this message",
deviceTypeMap.Port);
return null;
}
var message =
new DeviceMessage(new Device(receiveResult.RemoteEndPoint.Address.ToIPAddressNum(),
receiveResult.RemoteEndPoint.Port, deviceTypeMap.DeviceType, tenant.TenantId,
timeZoneOffset.Offset))
{
MessageUid = Guid.NewGuid(),
Timestamp = DateTime.UtcNow,
Raw = receiveResult.Buffer,
};
message.Information("Received Incoming Message");
return message;
}
private async Task<DeviceMessage> InjectTestMode(DeviceMessage message)
{
if (_testModeEnabled && message != null)
{
message = await _testModeProvider.InjectTestModeAsync(message);
}
return message;
}
private async Task<DeviceMessage> PublishMessage(DeviceMessage message)
{
await _messagePublisher.Publish(message);
return message;
}
private void OnComplete()
{
throw new NotImplementedException();
}
private void OnError(Exception ex)
{
throw new NotImplementedException();
}
private void OnMessageReceive(object o)
{
throw new NotImplementedException();
}
[Final Update]
This is what we finally ended up with; an IObservable>
var listeners = Observable.Defer(() => _deviceTypeProvider.GetDeviceTypes()
.ToObservable()
.Select(UdpListener)
.SelectMany(listener =>
{
return Observable.Defer(() => Observable
.FromAsync(listener.UdpClient.ReceiveAsync)
.Where(x => x.Buffer.Length > 0)
.Repeat()
.Select(result => CreateMessage(listener.DeviceType, result))
.SelectMany(InjectTestMode)
.OfType<DeviceMessage>()
.Do(async message => await PublishMessage(message)))
.Retry();
})).Retry();
_listener = listeners.Subscribe(OnMessageReceive, OnError, OnComplete);
So I can post some code and be constructive, this is what I think the query in your Start method should look like:
_listener = _deviceTypeProvider.GetDeviceTypes().ToObservable()
.SelectMany(CreateUdpListener, CreateMessage)
.Retry()
.Subscribe(OnMessageReceive, OnError, OnComplete);
Now the poor guy to come along and support this code has a chance of understanding it. :-)
I think you're missing the fact that RX sequences do terminate on exceptions. You need to use Observable.Catch if you want to continue sequence when exception occur.
See this http://www.introtorx.com/content/v1.0.10621.0/11_AdvancedErrorHandling.html
I have a Message Publishing Server class which creates a PUB socket at construction, with the following code:
this.context = NetMQContext.Create();
this.pubSocket = this.context.CreatePublisherSocket();
var portNumber = this.installerSettings.PublisherPort;
this.pubSocket.Bind("tcp://127.0.0.1:" + portNumber);
Sending a message using messagePublishingServer.Publish(message) executes:
this.pubSocket.SendMoreFrame(string.Empty).SendFrame(message);
The following xBehave test...
[Scenario]
public void PublishMessageScenario()
{
MessagePublishingServer messagePublishingServer = null;
NetMQContext context;
NetMQ.Sockets.SubscriberSocket subSocket = null;
string receivedMessage = null;
"Given a running message publishing server"._(() =>
{
var installerSettingsManager = A.Fake<IInstallerSettingsManager>();
var settings = new InstallerSettings { PublisherPort = "5348" };
A.CallTo(() => installerSettingsManager.Settings).Returns(settings);
messagePublishingServer = new MessagePublishingServer(installerSettingsManager);
});
"And a subscriber connected to the publishing server"._(() =>
{
context = NetMQContext.Create();
subSocket = context.CreateSubscriberSocket();
subSocket.Options.ReceiveHighWatermark = 1000;
subSocket.Connect("tcp://127.0.0.1:5348");
subSocket.Subscribe(string.Empty);
});
"When publishing a message"._(() =>
{
messagePublishingServer.Publish("test message");
// Receive the topic
subSocket.ReceiveFrameString();
// and the message
receivedMessage = subSocket.ReceiveFrameString();
});
"Then the subscriber must have received it"._(() =>
{
receivedMessage.Should().NotBeNullOrEmpty();
receivedMessage.Should().Be("test message");
});
}
... blocks in the first subSocket.ReceiveFrameString() which I find unexpected. Shouldn't the subscriber socket have queued the published message until the receive is called?
Publisher is like radio, if you was not connected and subscribed when the publisher published you miss the message. My tip is to put 100ms sleep after subscriber connect (only for testing).
From the source (ReceivingSocketExtensions.cs
):
/// Receive a single frame from socket, blocking until one arrives, and decode as a string using ...
public static string ReceiveFrameString([NotNull] this IReceivingSocket socket)
and
/// If no message is immediately available, return <c>false</c>.
public static bool TryReceiveFrameString([NotNull] this IReceivingSocket socket, out string frameString)