Based on the documentation wildcards support does exist, but I can't seem to find any other info on whether it's just supposed to work or if it's configured on the server or whether the producers or consumers need to configure it.
I'm assuming as a publisher I would just send messages to a topic named /patient/2/goal/ and when a consumer subscribed to a topic called /patient/*/goal/ it would still receive the message, however nothing shows up. What am I missing?
Please note that if I publish a message to /patient/*/goal/ and subscribe to /patient/*/goal/ then I receive the message. However, that only confirms my message bus is working, not that wildcard support is working.
Producer test:
var connectUri = new Uri("...");
var factory = new NMSConnectionFactory(connectUri);
var connection = factory.CreateConnection();
session = connection.CreateSession();
var destination = session.GetTopic("/patient/1/goal/");
producer = session.CreateProducer(destination);
...
Consumer:
var topic = _session.GetTopic("/patient/*/goal/");
var consumer = _session.CreateConsumer(topic);
...
Using / as the the path separator needs to be configured through a plugin. Switching to . made it work as expected.
Related
I'm trying to receive watch notification for Gmail mailboxes in my .NET C# program.
I have the following code:
WatchRequest body = new WatchRequest()
{
LabelIds = labelIds,
TopicName = PrivateKeys.GoogleGmailTopic
};
WatchResponse response = new UsersResource.WatchRequest(_gmailService, body, "me")
{
Key = PrivateKeys.GoogleApiKey,
OauthToken = AccessToken
}
.Execute();
... and it seems to work, since I get a response.
But where do I receive these notifications inside my program? How can I configure the event entry point?
Assuming you did the preliminary work to set up a Cloud Pub/Sub client (described here: https://developers.google.com/gmail/api/guides/push#initial_cloud_pubsub_setup), you should have a Pub/Sub subscription - either PULL or PUSH.
In order to get the notifications, your application should initiate a subscriber, you can follow the instructions in https://cloud.google.com/pubsub/docs/pull for PULL or in https://cloud.google.com/pubsub/docs/push for PUSH. Configuring the event entry point should be done as part of the subscriber set up.
I'm building a Kafka consumer for my .net proyect. I'm using kafka-net (A native c# client fro Apache Kafka mady by James Roland).
The problem I have is that this code (based on the documentation) fetches all messages from the beginning by default:
private void StartKafkaConsumer(string ipKafka, string portKafka, string topicKafka)
{
string topic = topicKafka;
Uri uri = new Uri($"http://{ipKafka}:{portKafka}");
var options = new KafkaOptions(uri);
using (var router = new BrokerRouter(options))
{
using (var consumer = new Consumer(new ConsumerOptions(topic, router)))
{
foreach (var message in consumer.Consume())
{
Console.WriteLine(Encoding.UTF8.GetString(message.Value));
}
}
}
}
...
StartKafkaConsumer("localhost", "9092", "test"); //this fetches messages sent weeks ago, since the creation of the 'test' topic
Basically, this code does the same than this command:
.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
All I want to do is to fetch messages from the time the client gets connected to the Apache server, not from the beginning. I know it's possible since I tried the last command without the "--from-beginning" part and it worked.
Any suggestion would be appreciated.
Look at the ConsumerOptions methods. There should be an option to set a property/config value:
auto.offset.reset
Set the above property/config to latest. At that point, anytime you connect with an unknown/new consumer group id, the consumer will default to start at the latest offset.
If, however, the consumer group id was already known (i.e., had already consumed at least once from that topic/partition), it will attempt to get the last committed offset + 1. If that offset is unavailable, because maybe it has gone past the retention threshold, then it will default to latest.
Here's more detailed documentation:
https://kafka.apache.org/documentation/#newconsumerconfigs
I'm trying to create a new Topic (and then a subscription) with the method CreateTopicAsync of the library Microsoft.Azure.ServiceBus.Management.
The connection string is correct and I can send and receive messages if I create the topic trough the Azure portal.
What am I doing wrong? Any help is appreciated.
var managementClient = new ManagementClient(ServiceBusConnectionString);
bool topicExists = await managementClient.TopicExistsAsync(TopicName).ConfigureAwait(false);
if (!topicExists) {
TopicDescription td = new TopicDescription(TopicName);
td.MaxSizeInMB = 1024;
td.DefaultMessageTimeToLive = new TimeSpan(2, 0, 0, 0);
await managementClient.CreateTopicAsync(td).ConfigureAwait(false);
}
Although the Service Bus ConnectionString might be correct, for your application to be able to create a Topic it needs to have the Manage right (claim).
Taken from Service Bus access control with Shared Access Signatures:
For each authorization policy rule, you decide on three pieces of information: name, scope, and rights. The name is just that; a unique name within that scope. The scope is easy enough: it's the URI of the resource in question. For a Service Bus namespace, the scope is the fully qualified domain name (FQDN), such as https://.servicebus.windows.net/.
The rights conferred by the policy rule can be a combination of:
'Send' - Confers the right to send messages to the entity
'Listen' - Confers the right to listen (relay) or receive (queue, subscriptions) and all related message handling
'Manage' - Confers the right to manage the topology of the namespace, including creating and deleting entities
The 'Manage' right includes the 'Send' and 'Receive' rights.
For now I have tried to filter the messages based on Message Attribute Name="Class". As you can see in the below code
//Specify attribute list
List<string> AttributesList = new List<string>();
AttributesList.Add("Class");
receiveMessageRequest.MessageAttributeNames = AttributesList;
receiveMessageRequest.QueueUrl = urlSQS;
receiveMessageRequest.MaxNumberOfMessages = 10;
ReceiveMessageResponse receiveMessageResponse = objClient.ReceiveMessage(receiveMessageRequest);
But the messages are not been filtered based on the provided MessageAttributeName = "class".
receiveMessageRequest.MessageAttributeNames = AttributesList;
This tells SQS which message attributes you want it to return with the message if the are present on the message. It is not a message filter. If the attributes aren't present, nothing happens.
But your confusion seems understandable -- it's not actually obvious why the API even has this functionality, though it may be a holdover from when SQS supported only smaller messages than it does today, or it may be so that you can avoid spending any time parsing information from the response that you will end up discarding. I almost always just ask for All.
Please note this regarding messaging services on AWS
SQS : No filtering support ( while picking up messages)
SNS : Supports attributes-based filtering: subscriber can set a subscription attribute (a subscription filter policy) which is applied on incoming messages and only relevant messages can be sent to the subscriber.
EventBridge: Amazon EventBridge supports declarative filtering using event patterns. With event pattern content filtering you can write complex rules that only trigger under very specific conditions. For instance, you might want a rule that will trigger only when a field of the event is within a specific numeric range, if the event comes from a specific IP address, or only if a specific field does not exist in the event JSON.
Please refer my article for a detailed difference between main messaging frameworks on AWS.
https://www.linkedin.com/pulse/mastering-art-decoupling-application-architecture-aws-amit-meena/
It depends on how the message in question gets onto the queue. If you are pushing the message via SNS then yes you can filtered messages; https://docs.aws.amazon.com/sns/latest/dg/message-filtering.html
Any other filtering mechanism doesn't exist right now.
Hope that helps!
As per the AWS SDK method, we can use following code to do the filter.
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest("QUEUE URL" );
receiveMessageRequest.setMaxNumberOfMessages(Integer.valueOf(1));
private static AmazonSQS sqs;
List<Message> messages = sqs.receiveMessage(receiveMessageRequest.withMessageAttributeNames("Attribute Name")).getMessages();
If you want all the message then use given code
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest("QUEUE URL" );
receiveMessageRequest.setMaxNumberOfMessages(Integer.valueOf(1));
private static AmazonSQS sqs;
List<Message> messages = sqs.receiveMessage(receiveMessageRequest.withMessageAttributeNames("All")).getMessages();
Using RabbitMQ with a cluster of nodes, I set up my factory and connection like this (using the .NET client):
var factory = new ConnectionFactory()
{
UserName = Properties.Settings.Default.RabbitMQUser,
Password = Properties.Settings.Default.RabbitMQPassword,
HostnameSelector = new RandomHostNameSelector(),
AutomaticRecoveryEnabled = true
};
connection = factory.CreateConnection(Properties.Settings.Default.RabbitMQServers.Split(';'));
Where Properties.Settings.Default.RabbitMQServers is use a semi-colon delimited list of servers:
clust01;clust02;clust03
After connecting, is there a way to see which host it actually picked and is there a way to detect when that might change (because the particular node crashed, was stopped, or rebooted)? factory.HostName just returns localhost.
Edit: It seems connection.Endpoint.HostName does give you the actual host name as #Evk and #wally have both stated (thank you). But is there a way to detect a change? It seems that IConnection has a ConnectionShutdown event which I guess might be called as part of switching hosts, but there isn't a corresponding start up or restart event (so I'm assuming it wouldn't know the new host yet). There are also ConnectionBlocked and ConnectionUnblocked events, but they are undocumented so I have no idea what they do...
So how to get host name you already found out, as for how to detect when it changes: when you set AutomaticRecoveryEnabled for your factory, CreateConnection will really return instance of AutorecoveringConnection. This class implements special IRecoverable interface, which has event you need. So to detect when automatic recovery happens, subscribe to it like this:
var connection = factory.CreateConnection();
((IRecoverable)connection).Recovery += (sender, args) =>
{
// recovery happened
};
I've not got an environment in which I can test this, so it might be completely wrong.
It looks like you can use the endpoint of the connection object (IConnection) returned:
(string) connection.Endpoint.HostName
Cite: https://www.rabbitmq.com/releases/rabbitmq-dotnet-client/v1.4.0/rabbitmq-dotnet-client-1.4.0-net-2.0-htmldoc/type-RabbitMQ.Client.IConnection.html