I Am trying to play with MQTT and RabbitMQ but with some problems.
Introduction
I have several physical machines that generate data and I want to do something with this data, for simplicity imagine that I just want to write those messages to a separate file for each machine identified by an ID. (do not focus on the details, it is just an example)
What I have
I have an MQTT broker (i.e. mosquitto broker) that handles several messages coming from those machines.
I have a (complex) windows service written in C#. An object instantiation exists for each machine (i.e. machine with id = 1000 leads to an object that represents the physical machine in the service program and so on). This machine object has a RabbitMQ queue of messages that contains every message that is delivered by the machine.
The problem
How can I populate this queue?
I thought that there was the possibility, of using the rabbitMQTT plugin, to instantiate an exchange or something similar to listen for the MQTT topics and to forward the received messages to the appropriate queue as usual, but I cannot find anything on the net.
I hope I've been clear enough in proposing the problem that I Am facing.
Assume that I MUST use RabbitMQ since it is already used for the communication of the different modules of the service.
Hope you can help me understand if there is any possibility to use RabbitMQ to listen for messages from an external MQTT broker that I cannot decide on and then push those messages in an exchange that will route the message based on the routing key extracted from the message.
Practical case:
Real-world machine 10000 produces a message A
Real-world machine 10000 publishes on topic machines/10000 the message A
The service (that is subscribed to the machine/# topic) gets the message A
The service publishes message A in the exchange with routing key machines.10000
The machine 10000's callback processes the message A and does something
Thanks, please be as clear as possible since I need to understand the entire process (if possible)
Related
I am trying to implement the publisher subscriber pattern using Nservice bus , what i am trying to do is as follow :
I have web application , user can write news and add documents to that Application using his account .
I have a Windows forms desktop application that running on users machines , they can login this desktop app using the same credentials they are using to access the web application.
what i need to to is when users on web applications add news or documnets , the installed desktop app should receive notifications to informing it about this added news .....
i imagin that my desktop application will be the subscriber that will subscribe on events that will be published from the server
what i need to know is
when multiple users for example 1000 user , install this desktop app on different machines to start using it , in this case Nservice bus will consider them as multiple subscribers and will send a copy of message to each one of them or what ???
As described in the documentation, it works like this:
If you use Send(ICommand command), you get message sent directly to the matching target. In this case, if you have multiple consumers, they will compete to get this message and only one gets it. This is usually used to scale applications.
If you do Publish(IEvent #event), your message will get delivered to all subscribers.
When Pub/Sub happens in NServiceBus, there is one logical publisher that sends an event to multiple logical subscribers. Each logical subscriber will only receive one copy of each event that is published.
If many subscribers represent the same logical entity then each event will be processed by only one of the subscribers within that logical entity. Two subscribers are considered to represent the same logical entity if they share an Endpoint Name.
In this case, if every instance of the desktop application have the same endpoint name then they are considered to represent the same logical entity. When an event is published, then only one of those desktop application instances will get a copy of the event. That is probably not what you want.
If you give each instance of the desktop application a different endpoint name then they each represent their own logical entity. When an event is published, all of the desktop instances will get a copy of the event. That is also probably not what you want.
Another thing to consider is pub/sub in NSB is persistent and reliable. If you start a desktop application and it sets up a subscription, that subscription is in effect even if the desktop application is offline. Messages may be queued up waiting for the Desktop Application to come back online to be processed, even if you never run the desktop application again. Does your desktop application need these or do you only care about events that occur while you are connected?
If you want to notify a specific subscriber or subset of subscribers that are online when an event occurs on the server then I would use SignalR before using NSB.
Additional Info
MSMQ and SQL Server do not support pub-sub out of the box and so they use a feature called message driven-subscription. Message driven subscriptions work by having the subscriber send a message to the publisher when it starts up (or calls Subscribe()). This message contains a Reply To header which the publisher stores and uses to send copies of each message that is published.
For SQL Server transport, the Reply To address will be the name of the Endpoint. This in turn is mapped to a table in the database that represents the queue of messages. If multiple subscribers share the same Endpoint name (and hence the same table) then they will get one copy of the event between them and will be competing to see which subscriber pulls it from the queue.
For MSMQ, the Reply To address will be the name of the Endpoint and the name of the machine separated by an # symbol. This means that each subscription will be for a different address. When the event is published, each subscriber will get it's own copy of it. If multiple subscribers represent the same logical entity (i.e. share an Endpoint name), then you should have a distributor in front of them. When you do this then the Reply To header on outgoing subscribe messages will be the address of the Distributor. When an event is published, it will always go to the Distributor which will hand it off to one (and only one) of it's workers for processing.
Rabbit MQ and Azure Service Bus have built-in mechanisms for pub/sub but they will behave in a manner similar to SQL Transport. One instance of the logical subscriber will get a copy of the event.
i am developing an android application and i want to broadcast some message to all my clients with RabbitMQ. is there any way wiht RabbitMQ, that supports offline clients - by offline clients I mean that if i have server submitting messages to the clients,and some clients are unreachable,the client will receive messages after it gets connected again.
Is there some open source solution for that? It seems that rabbitMQ does not do this,i might be wrong tho.
Any help would be appreciated.
You can create a unique queue for each client. Bind those queues to a common exchange. Then every time your client will come online it will create a connection to rabbit (earlier defined queue) and will consume messages.
I think you would better benefit from topic exchange (https://www.rabbitmq.com/tutorials/tutorial-five-python.html)
And bind them to a common exchange with desired routing keys.
I am looking for a message broker API to use it with c#.
Normally the things are quite simple. I have a server that knows what jobs are to do and I have some clients that need to get these jobs.
And here are the special requirements I have:
If a client got a job but fails to answer within a specific time, then another client should do the work.
More than one queue and priorities
If possible it needs to work with big message queues (this way I could just load all jobs sometimes a month and forget about it
secured communications would be good.
API for talking with the broker from c#. How much work is done? What is still to do?
Delete some jobs...
If available replication to another broker would be good.
The broker needs to run on windows
What is not an issue:
low latency (there is no problem when a message needs minutes)
Do you know such a message broker that is free to use?
RabbitMQ and several other AMQP implementations satisfy most of (if not all of) these requirements.
RabbitMQ allows clients to acknowledge receipt and/or processing of messages. As per http://www.rabbitmq.com/tutorials/amqp-concepts.html#message-acknowledge:
If a consumer dies without sending an acknowledgement the AMQP broker
will redeliver it to another consumer or, if none are available at the
time, the broker will wait until at least one consumer is registered
for the same queue before attempting redelivery.
Many queues (and in fact many brokers) are supported, in a variety of different configurations
It scales particularly well, even for very large message queues: http://www.rabbitmq.com/faq.html#performance
Encryption is supported: http://www.rabbitmq.com/faq.html#channel-encryption
There is a .NET Client Users Guide and API docs: http://www.rabbitmq.com/documentation.html
There is live failover if a broker dies: http://www.rabbitmq.com/clustering.html
It runs on Windows, Linux, and probably anything else that has an Erlang implementation
I'm trying to implement publish/subscribe through WCF. My problem is that multiple clients originating from 1 computer work as expected, but clients on another box on the same network don't get the messages.
I'm following this blog tutorial: http://chakkaradeep.wordpress.com/2007/07/30/coding-wcf-publishersusbcriber-model/
I'm hosting the WCF service on an IIS box on my network, and running multiple instances of the test client on 2 separate network machines. 1 machine can bounce messages off the server without problem, but the other machine's clients should be receiving these messages and they just aren't. Is there a specific config flag I need to be setting to allow what it is I want?
You can take a look at the code for my project nvents, or simply use it as is.
How do you 'verify' that a message sent using MSMQ to a private local queue was actually delivered? I'm especially thinking of a scenario where the listener (a C# service in my case) is not running and therefore delivery can't be successful.
You can only guarantee that it can get to the queue without taking extra steps. To deal with the "not running receiver" scenario, you would need to code the receiver to send a message back to the server when it processes the message. The original sender would be responsible for tracking the sent messages and verifying that the client has recieved them.
That's one of the decisions you should be taking when deciding whether or not to use MSMQ as opposed to a remoting or a web service scenario. For example, we had a project used for notifying all of our retail locations when an emergency occurred (such as a product recall/food safety issue.) We needed to know immediately if the store's listener was running so we chose remoting, and when the sender received an error indicating one of the listeners was not listenting, we would need to pick up the phone and call.
Just something to keep in mind.
Edit - clarification
I was really giving out two options above.
Code the client to send a message back to the sender when it receives a message.
Use another option, such as remoting, where you can detect if the client is running and receives the message.
It's always sent to the queue.
If your service isn't running to receive it, it just sits there, waiting patiently, until someone receives it.
You know it's been sent to the queue because .Send() returns without crashing.
You can probably pull this info out using administrative queues
When you send a message you can specify the AcknowledgeType which will allow you find out (through positive or negative acknowledgement) whether the message reached the queue and/or was received from the queue. Acknowledgements are sent as messages, by MSMQ, to the AdministrativeQueue so make sure you assign that property on the Message object.
You can check the administrative queue for acknowledgements by correlation ID which is ID of the original message.