I wrote a code to read 1000 messages one shot from an Azure Service Bus queue. I read the messages with the line: await receiver.ReceiveMessagesAsync(1000); but only a subset of the messages are received.
I took the code from the sample: https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/servicebus/Azure.Messaging.ServiceBus/tests/Samples/Sample01_HelloWorld.cs, the SendAndReceiveMessageSafeBatch() method
This is my code:
public class Program
{
static void Main(string[] args)
{
SendAndReceiveMessage().GetAwaiter().GetResult();
}
public static async Task SendAndReceiveMessage()
{
var connectionString = "myconnectionstring";
var queueName = "myqueue";
// since ServiceBusClient implements IAsyncDisposable we create it with "await using"
await using var client = new ServiceBusClient(connectionString);
// create the sender
var sender = client.CreateSender(queueName);
IList<ServiceBusMessage> messages = new List<ServiceBusMessage>();
for (var i = 0; i < 1000; i++)
{
messages.Add(new ServiceBusMessage($"Message {i}"));
}
// send the messages
await sender.SendMessagesAsync(messages);
// create a receiver that we can use to receive the messages
var options = new ServiceBusReceiverOptions()
{
ReceiveMode = ServiceBusReceiveMode.ReceiveAndDelete
};
ServiceBusReceiver receiver = client.CreateReceiver(queueName, options);
// the received message is a different type as it contains some service set properties
IReadOnlyList<ServiceBusReceivedMessage> receivedMessages = await receiver.ReceiveMessagesAsync(1000);
Console.WriteLine($"Received {receivedMessages.Count} from the queue {queueName}");
foreach (ServiceBusReceivedMessage receivedMessage in receivedMessages)
{
var body = receivedMessage.Body.ToString();
Console.WriteLine(body);
}
Console.WriteLine("END");
Console.ReadLine();
}
}
Do you have any suggestion how to read all 1000 messages one shot?
This is expected behaviour with Azure Service Bus. The number of messages to receive, maxMessages is a maximum number that is not guaranteed.
I am using Confluent.Kafka .NET client version 1.3.0. I would like to start consuming messages from a given time onwards.
To do so, I could use OffsetsForTimes to get the desired offset and Commit that offset for that partition:
private void SetOffset()
{
const string Topic = "myTopic";
const string BootstrapServers = "server1, server2";
var adminClient = new AdminClientBuilder(
new Dictionary<string, string>
{
{ "bootstrap.servers", BootstrapServers },
{ "security.protocol", "sasl_plaintext" },
{ "sasl.mechanisms", "PLAIN" },
{ "sasl.username", this.kafkaUsername },
{ "sasl.password", this.kafkaPassword }
}).Build();
var consumer = new ConsumerBuilder<byte[], byte[]>(
new Dictionary<string, string>
{
{ "bootstrap.servers", BootstrapServers },
{ "group.id", this.groupId },
{ "enable.auto.commit", "false" },
{ "security.protocol", "sasl_plaintext" },
{ "sasl.mechanisms", "PLAIN" },
{ "sasl.username", this.kafkaUsername },
{ "sasl.password", this.kafkaPassword }
}).Build();
// Timestamp to which the offset should be set to
var timeStamp = new DateTime(2020, 3, 1, 0, 0, 0, DateTimeKind.Utc);
var newOffsets = new List<TopicPartitionOffset>();
var metadata = adminClient.GetMetadata(Topic, TimeSpan.FromSeconds(30));
foreach (var topicMetadata in metadata.Topics)
{
if (topicMetadata.Topic == Topic)
{
foreach (var partitionMetadata in topicMetadata.Partitions.OrderBy(p => p.PartitionId))
{
var topicPartition = new TopicPartition(topicMetadata.Topic, partitionMetadata.PartitionId);
IEnumerable<TopicPartitionOffset> found = consumer.OffsetsForTimes(
new[] { new TopicPartitionTimestamp(topicPartition, new Timestamp(timeStamp, TimestampType.CreateTime)) },
TimeSpan.FromSeconds(5));
newOffsets.Add(new TopicPartitionOffset(topicPartition, new Offset(found.First().Offset)));
}
}
}
consumer.Commit(newOffsets);
// Consume messages
consumer.Subscribe(Topic);
var consumerResult = consumer.Consume();
// process message
//consumer.Commit(consumerResult);
}
This works fine if I want to skip messages and jump to a given offset if the offset to which I would like to jump is after the last committed message.
However, the above approach won't work if the given timestamp is before the timestamp of the last committed message. In the above code, if the timeStamp is before the timestamp of the last committed message, then OffsetsForTimes will return the offset of that last committed message + 1. Even if I manually set the offset to a lower offset, then consumer.Commit(newOffsets) seems to have no effect and I am getting the first uncommitted message when consuming.
Is there a way to achive this from the code?
You can do it if you assign to each partition and state the offset from which to start reading.
this is how you get the list of topic partitions:
public static List<TopicPartition> GetTopicPartitions(string bootstrapServers, string topicValue) {
var tp = new List<TopicPartition>();
using (var adminClient = new AdminClientBuilder(new AdminClientConfig { BootstrapServers = bootstrapServers }).Build()) {
var meta = adminClient.GetMetadata(TimeSpan.FromSeconds(20));
meta.Topics.ForEach(topic => {
if (topic.Topic == topicValue) {
foreach (PartitionMetadata partition in topic.Partitions) {
tp.Add(new TopicPartition(topic.Topic, partition.PartitionId));
}
}
});
}
return tp;
}
This is how you find the offset of a particular time:
List<TopicPartition> topic_partitions = frmMain.GetTopicPartitions(mBootstrapServers, txtTopic.Text);
using (var consumer = new ConsumerBuilder<Ignore, string>(cfg).Build()) {
consumer.Assign(topic_partitions);
List<TopicPartitionTimestamp> new_times = new List<TopicPartitionTimestamp>();
foreach (TopicPartition tp in topic_partitions) {
new_times.Add(new TopicPartitionTimestamp(tp, new Timestamp(dtpNewTime.Value)));
}
List<TopicPartitionOffset> seeked_offsets = consumer.OffsetsForTimes(new_times, TimeSpan.FromSeconds(40));
string s = "";
foreach (TopicPartitionOffset tpo in seeked_offsets) {
s += $"{tpo.TopicPartition}: {tpo.Offset.Value}\n";
}
Console.WriteLine(s);
consumer.Close();
}
This is how you consume by assigning to all topic partition and specific offsets:
using (var consumer =
new ConsumerBuilder<string, string>(config)
.SetErrorHandler((_, e) => Log($"Error: {e.Reason}"))
.Build()) {
consumer.Assign(seeked_offsets);
try {
while (true) {
try {
var r = consumer.Consume(cancellationToken);
// do something with r
} catch (ConsumeException e) {
//Log($"Consume error: {e.Error.Reason}");
}
}
} catch (OperationCanceledException) {
//Log("Closing consumer.");
consumer.Close();
}
}
The other option, if you insist in applying this to the consumer group, would be to reset the consumer group and use your code, or to create a new consumer group.
I'm not an expert but i'm going to try to explain how you could do it.
In first place, we have to mention subscribe and assign methods.
When you use subscribe, you pass a one or more topics. With this, a list of partitions of each topics are assigned to the consumer depending of the number of consumers in its group. A topic partition is an object formed by the topic name and the partition number.
consumer.Subscribe(Topic);
You can use assign to pass the partitions of which the consumer will read. This method does not use the consumer's group management functionality (where no need of group.id)
If i'm not wrong, in assign method you can specify the initial offset.
consumer.Assign(topicName, 0, new Offset(lastConsumedOffset));
consumer.Assign(topicPartition, new Offset(lastConsumedOffset));
Another option is to use seek() method to set the offset
consumer.Seek(topicPartitionOffset);
If you are going to mix subscribe and assign remember that you have to use unsubscribe before.
Another option, if you want to re-consume all the messages is create a
consumer in a new different consumer group.
EXAMPLE (TO REVIEW)
I'm leaving you and example for now, i will check it later.
I have done the example in java because i'm more familiar with it.
In this example, i don't use subscribe, i use assign.
First topic partitions are retrieved, we set a start datetime to read messages from, we create a map specifing that that datetime for each partition.
With the created map we get the offset of each partition on the specified datetime with offsetsForTimes method. With the offset of each partition we use seek to move to that offset on each partitions and finally we consume the messages.
I don't have time now to check the code, but i will do it.
I hope it helps.
AdminClient client = AdminClient.create(getAdminClientProperties());
KafkaConsumer<String, GenericRecord> consumer = new KafkaConsumer<String, GenericRecord>(
getConsumerProperties());
String TOPIC = "topic";
// get info of all partitions of a topic
List<PartitionInfo> partitionsInfo = consumer.partitionsFor(TOPIC);
// create TopicPartition list
Set<TopicPartition> partitions = new HashSet<>();
for (PartitionInfo p : partitionsInfo) {
partitions.add(new TopicPartition(p.topic(), p.partition()));
}
// Consumer will read from all partitions
consumer.assign(partitions);
DateTime timeToStartReadMessagesFrom = new DateTime(2020, 3, 1, 0, 0, 0);
Map<TopicPartition, Long> timestamps = new HashMap<>();
for (TopicPartition tp : partitions) {
timestamps.put(tp, timeToStartReadMessagesFrom.getMillis());
}
// get the offset for that time in each partition
Map<TopicPartition, OffsetAndTimestamp> offsets = consumer.offsetsForTimes(timestamps);
for (TopicPartition tp : partitions) {
consumer.seek(tp, offsets.get(tp).offset());
}
while (true) {
final ConsumerRecords<String, GenericRecord> consumerRecords = consumer.poll(1000);
// do something
break;
}
consumer.close();
System.out.println("DONE");
I want to find duplicate messages in ServiceBusTopics.Below is my logic.If I find any duplicate message am adding to the list and sending to another service bus topic. But I wasn't able to read all the service bus messages that are present in the subscription before looping through the list.I want to read all the messages from the sevicebus and then loop through it.I'm not sure how should I stop the execution of loops until I read all the messages from suscription
private static string _serviceBusConn = "servicebusconnectionstring";
private static string _serviceBustopic = "topic1";
private static string _topic = "test_v1";
static void Main(string[] args)
{
IList<string> items = new List<string>();
int count;
IList<string> itemsRepeated = new List<string>();
var subClient = SubscriptionClient.CreateFromConnectionString(_serviceBusConn, _serviceBustopic, "DevTest");
subClient.OnMessage(m =>
{
Stream stream = m.GetBody<Stream>();
StreamReader reader = new StreamReader(stream);
string s = reader.ReadToEnd();
Console.WriteLine(s);
items.Add(s);
});
List<string> copy1= new List<string>(items);
List<string> copy2 = new List<string>(items);
foreach (var item in copy1)
{
count = 0;
foreach (var itemtopic in copy2)
{
count++;
Console.WriteLine("{0}{1}{2}", items, itemtopic, count);
if (item.Equals(itemtopic))
{
count++;
}
if (count > 1)
{
Console.WriteLine(count);
itemsRepeated.Add(itemtopic);
}
}
}
foreach (var repeateditem in itemsRepeated)
{
SendMessage(repeateditem);
}
}
static void SendMessage(string message)
{
var topicClient = TopicClient.CreateFromConnectionString(_serviceBusConn, _topic);
var msg = new BrokeredMessage(message);
topicClient.Send(msg);
}
You can use ReceiveBatch(Int32) method to receive messages from a topic subscription in a batch. You can get the number of messages in a topic subscription using Getsubscription() method of namespace manager.
var namespaceManager = NamespaceManager.CreateFromConnectionString(connnectionString);
var subscriptionDescription = namespaceManager.GetSubscription(topicName, subscriptionName);
var totalMessageCount= subscriptionDescription .MessageCount;
Now you can call the ReceiveBatch(minimumMessageCount) within a loop and terminate the loop when the received message count reaches the totalMessageCount.
int receivedMessageCount = 0;
List<BrokeredMessage> MessageList = new List<BrokeredMessage>();
do{
var messageList = subClient.ReceiveBatch(100);
receivedMessageCount += messageList.Count;
MessageList.AddRange(messageList);
}while(receivedMessageCount < totalMessageCount);
Now the MessageList will have all the messages in the topic subscription.You can use your custom logic on the message list to perform duplicate detection and forward that to another subscription.
Note: To receive all the messages from a ServiceBusTopic, you have to receive all the messages from all the topic subscriptions within that topic.
if (count > 1)
{
Console.WriteLine(count);
itemsRepeated.Add(itemtopic);
}
you can break the loop by using break like this
if (count > 1)
{
Console.WriteLine(count);
itemsRepeated.Add(itemtopic);
break;
}
This is my publisher. There are two consumers. MailConsumer and SmsConsumer.
using(var bus = RabbitHutch.CreateBus("host=localhost").Advanced) {
var queueName = "my.queue";
var queueName2 = "my.queue2";
var queue = bus.QueueDeclare(queueName);
var queue2 = bus.QueueDeclare(queueName2);
var channel = bus.ExchangeDeclare("MyFanout", ExchangeType.Fanout);
bus.Bind(channel, queue, "sms");
bus.Bind(channel, queue2, "mail");
var input = "";
Console.WriteLine("Enter a message. 'q' to quit.");
while((input = Console.ReadLine()) != "q") {
for(int i = 1; i < 2; i++) {
var message = new Message<TextMessage>(new TextMessage {
Text = input + i
});
bus.Publish(channel, "", false, false, message);
}
}
}
I can subscribe with this code:
using(var bus = RabbitHutch.CreateBus("host=localhost").Advanced) {
var queueName = "my.queue2";
var queue = bus.QueueDeclare(queueName);
bus.Consume(queue, x => x.Add<TextMessage>((message, info) => {
Thread.Sleep(1000);
Console.WriteLine("SMS: {0}", message.Body.Text);
}));
Console.WriteLine("Listening for messages. Hit <return> to quit.");
Console.ReadLine();
}
How can I achieve it via AutoSubscriber? There is no option for Queue Name in AutoSubscriber, there is "Subscription Id"
There's a property on the AutoSubscriber called 'GenerateSubscriptionId', which you can set to generate the subscription id for a consumer:
subscriber.GenerateSubscriptionId = subscriptionInfo =>
{
return "MyApplication:" + subscriptionInfo.ConcreteType.Name);
};
Then the subscription id will be used by the default conventions to generate a queue name.
Update:
I think you can achieve what you want just with normal pub-sub without declaring any queue.
In your publisher you can do this:
var bus = RabbitHutch.CreateBus("host=localhost")
while((input = Console.ReadLine()) != "q") {
for(int i = 1; i < 2; i++) {
var message = new Message<TextMessage>(new TextMessage {
Text = input + i
});
bus.Publish(message);
}
}
Then just create 2 consumers to subscribe to your message, and your message will be handled by both:
class SmsConsumer : IConsume<TextMessage>{
...
}
class LogConsumer : IConsume<TextMessage>{
...
}
and in your startup:
var bus = RabbitHutch.CreateBus("host=localhost")
var subscriber = new AutoSubscriber(bus,"my_applications_subscriptionId_prefix");
subscriber.Subscribe(Assembly.GetExecutingAssembly());
Console.ReadLine()
EasyNetQ will declare the exchange, queues and bindings for you. Just make sure your TextMessage class is in an assembly shared by both projects.
I am trying to implement a basic SSDP (UDP) broadcast/listener for a Windows Store application using C#.
I have found that Windows.Networking.Sockets contains the DatagramSocket class which is what I need to use for UDP networking.
However, my current attempts seem to execute just fine but have no results via Wireshark and do not get a response back from the devices on the network.
Here is the code I am currently using (and running through the RT Simulator):
public async static Task<IEnumerable<HueBridge>> DiscoverAsync(TimeSpan timeout)
{
if (timeout <= TimeSpan.Zero)
throw new ArgumentException("Timeout value must be greater than zero.", "timeout");
var discoveredBridges = new List<HueBridge>();
using (var socket = new DatagramSocket())
{
while (true)
{
var bridgeWasFound = false;
socket.MessageReceived += (sender, e) =>
{
var bpx = true; // breakpoint here for success
};
var multicastIP = new HostName("239.255.255.250");
await socket.BindServiceNameAsync("1900");
socket.JoinMulticastGroup(multicastIP);
using (var writer = new DataWriter(socket.OutputStream))
{
var request = new StringBuilder();
request.AppendLine("M-SEARCH * HTTP/1.1");
request.AppendLine("HOST: 239.255.255.250:1900");
request.AppendLine("MAN: ssdp:discover");
request.AppendLine("MX: 5");
request.AppendLine("ST: ssdp:all");
writer.WriteString(request.ToString());
await writer.FlushAsync();
}
if (timeout > TimeSpan.Zero)
await Task.Delay(timeout);
if (!bridgeWasFound)
break; // breakpoint here for failure check
}
}
return discoveredBridges;
}
Any ideas on what I may be doing incorrectly? I don't get an exception and I have the proper Capabilities set in the manifest. My breakpoint at the break always gets hit and I am using a timeout of 10 seconds.
Seems I have found the problem(s).
First, I should use socket.BindEndpointAsync(null, string.Empty) instead of socket.BindServiceNameAsync("1900"), which will properly listen for broadcast packets.
Secondly, writer.FlushAsync() does not write to the socket; however, writer.StoreAsync() does.
Here is the final result, which does work (almost) perfectly:
public async static Task<IEnumerable<HueBridge>> DiscoverAsync(TimeSpan timeout)
{
if (timeout <= TimeSpan.Zero)
throw new ArgumentException("Timeout value must be greater than zero.", "timeout");
var discoveredBridges = new List<HueBridge>();
var multicastIP = new HostName("239.255.255.250");
var bridgeWasFound = false;
using (var socket = new DatagramSocket())
{
socket.MessageReceived += (sender, e) =>
{
var reader = e.GetDataReader();
var bytesRemaining = reader.UnconsumedBufferLength;
var receivedString = reader.ReadString(bytesRemaining);
// TODO: Check for existing bridges, only add new ones to prevent infinite loop.
// TODO: Create new bridge and add to the list.
bridgeWasFound = true;
};
await socket.BindEndpointAsync(null, string.Empty);
socket.JoinMulticastGroup(multicastIP);
while (true)
{
bridgeWasFound = false;
using (var stream = await socket.GetOutputStreamAsync(multicastIP, "1900"))
using (var writer = new DataWriter(stream))
{
var request = new StringBuilder();
request.AppendLine("M-SEARCH * HTTP/1.1");
request.AppendLine("HOST: 239.255.255.250:1900");
request.AppendLine("MAN: ssdp:discover");
request.AppendLine("MX: 3");
request.AppendLine("ST: ssdp:all");
writer.WriteString(request.ToString());
await writer.StoreAsync();
if (timeout > TimeSpan.Zero)
await Task.Delay(timeout);
if (!bridgeWasFound)
break;
}
}
}
return discoveredBridges;
}
According Specifications :
MAN REQUIRED by HTTP Extension Framework. Unlike the NTS and ST field
values, the field value of the MAN header field is enclosed in double
quotes; it defines the scope (namespace) of the extension. MUST be
"ssdp:discover".
then your code
request.AppendLine("MAN: ssdp:discover");
must be
request.AppendLine("MAN: \"ssdp:discover\"");
Hope this help.