I want to find duplicate messages in ServiceBusTopics.Below is my logic.If I find any duplicate message am adding to the list and sending to another service bus topic. But I wasn't able to read all the service bus messages that are present in the subscription before looping through the list.I want to read all the messages from the sevicebus and then loop through it.I'm not sure how should I stop the execution of loops until I read all the messages from suscription
private static string _serviceBusConn = "servicebusconnectionstring";
private static string _serviceBustopic = "topic1";
private static string _topic = "test_v1";
static void Main(string[] args)
{
IList<string> items = new List<string>();
int count;
IList<string> itemsRepeated = new List<string>();
var subClient = SubscriptionClient.CreateFromConnectionString(_serviceBusConn, _serviceBustopic, "DevTest");
subClient.OnMessage(m =>
{
Stream stream = m.GetBody<Stream>();
StreamReader reader = new StreamReader(stream);
string s = reader.ReadToEnd();
Console.WriteLine(s);
items.Add(s);
});
List<string> copy1= new List<string>(items);
List<string> copy2 = new List<string>(items);
foreach (var item in copy1)
{
count = 0;
foreach (var itemtopic in copy2)
{
count++;
Console.WriteLine("{0}{1}{2}", items, itemtopic, count);
if (item.Equals(itemtopic))
{
count++;
}
if (count > 1)
{
Console.WriteLine(count);
itemsRepeated.Add(itemtopic);
}
}
}
foreach (var repeateditem in itemsRepeated)
{
SendMessage(repeateditem);
}
}
static void SendMessage(string message)
{
var topicClient = TopicClient.CreateFromConnectionString(_serviceBusConn, _topic);
var msg = new BrokeredMessage(message);
topicClient.Send(msg);
}
You can use ReceiveBatch(Int32) method to receive messages from a topic subscription in a batch. You can get the number of messages in a topic subscription using Getsubscription() method of namespace manager.
var namespaceManager = NamespaceManager.CreateFromConnectionString(connnectionString);
var subscriptionDescription = namespaceManager.GetSubscription(topicName, subscriptionName);
var totalMessageCount= subscriptionDescription .MessageCount;
Now you can call the ReceiveBatch(minimumMessageCount) within a loop and terminate the loop when the received message count reaches the totalMessageCount.
int receivedMessageCount = 0;
List<BrokeredMessage> MessageList = new List<BrokeredMessage>();
do{
var messageList = subClient.ReceiveBatch(100);
receivedMessageCount += messageList.Count;
MessageList.AddRange(messageList);
}while(receivedMessageCount < totalMessageCount);
Now the MessageList will have all the messages in the topic subscription.You can use your custom logic on the message list to perform duplicate detection and forward that to another subscription.
Note: To receive all the messages from a ServiceBusTopic, you have to receive all the messages from all the topic subscriptions within that topic.
if (count > 1)
{
Console.WriteLine(count);
itemsRepeated.Add(itemtopic);
}
you can break the loop by using break like this
if (count > 1)
{
Console.WriteLine(count);
itemsRepeated.Add(itemtopic);
break;
}
Related
I'd like to peek all messages from several Azure Service Bus queues. After that I want to filter them after queueName, insertDate and give the opportunity to make a full text search on the body.
Currently, I'm using the Microsoft.Azure.ServiceBus package to create a ManagementClient for gathering queue information and then use a MessageReceiver to peek the messages.
var managementClient = new ManagementClient(connectionString);
var queue = await managementClient.GetQueueRuntimeInfoAsync(queueName);
var count = queue.MessageCount;
var receiver = new MessageReceiver(connectionString, queueName);
var messagesOfQueue = new List<Message>();
for (var i = 1; i <= count; i++)
{
messagesOfQueue.Add(await receiver.PeekAsync());
}
Is there a better way to get all messages? Or is there even a way to only peek messages that apply to a filter?
I've also tried to use the QueueClient.PeekBatch Method from the WindowsAzure.ServiceBus package. But that method didn't return all messages although I've set the correct messageCount parameter.
And then there is also the package Azure.Messaging.ServiceBus... What's up with all these packages?
So which of the packages should I use and what is the best way for peeking messages of queues based on some filters?
The solution I'm currently using and which works as expected looks like this:
var receiver = serviceBusClient.CreateReceiver(queueName);
var messagesOfQueue = new List<ServiceBusReceivedMessage>();
var previousSequenceNumber = -1L;
var sequenceNumber = 0L;
do
{
var messageBatch = await receiver.PeekMessagesAsync(int.MaxValue, sequenceNumber);
if (messageBatch.Count > 0)
{
sequenceNumber = messageBatch[^1].SequenceNumber;
if (sequenceNumber == previousSequenceNumber)
break;
messagesOfQueue.AddRange(messageBatch);
previousSequenceNumber = sequenceNumber;
}
else
{
break;
}
} while (true);
It uses the nuget package Azure.Messaging.ServiceBus.
Currently you're receiving a single message from the receiver. A better option would be to receive messages in batch using PeekBatchAsync(Int64, Int32) method of MessageReceiver.
Here's the sample code to do so (untested though):
var messagesOfQueue = new List<Message>();
var sequenceNumber = 0;
var batchSize = 100;//number of messages to receive in a single call
do
{
var messages = await receiver.PeekBatchAsync(sequenceNumber, batchSize);
messagesOfQueue.AddRange(messages);
if (messages.Count > 0)
{
sequenceNumber = messages[messages.Count-1].SequenceNumber;
}
else
{
break;
}
} while (true);
The solution avoids getting the message with the same SequenceNumber twice.
Sequence numbers monotonically increase. And I've tested most cases except rolling over sequenceNumber to 0 when it reaches the maximum value (Long.MaxValue).
using Azure.Messaging.ServiceBus;
private static async Task<List<ServiceBusReceivedMessage>> PeekAllMessages(string serviceBusConnectionString, string queueName)
{
var client = new ServiceBusClient(serviceBusConnectionString);
var receiver = client.CreateReceiver(queueName);
var messages = new List<ServiceBusReceivedMessage>();
var batchSize = 20;
var sequenceNumber = 0L;
do
{
var messageBatch = await receiver.PeekMessagesAsync(batchSize, sequenceNumber);
if (messageBatch.Count <= 0)
{
break;
}
// Increasing the SequenceNumber by 1 to avoid getting the message with the same SequenceNumber twice
sequenceNumber = messageBatch[^1].SequenceNumber + 1;
messages.AddRange(messageBatch);
} while (true);
return messages;
}
Well, I'm trying to run a task 100 times on each run (with paralellism) but I can't manage this to work.
I'm trying to bruteforce an API, for this the API allows me to concatenate as many IDS as possible (without exceeding the timeout)
// consts:
// idsNum = 1000
// maxTasks = 100
// We prepare the ids that we will process (in that case 100,000)
var ids = PrepareIds(i * idsNum * maxTasks, idsNum, maxTasks);
// This was my old approach (didn't work)
//var result = await Task.WhenAll(ids.AsParallel().Select(async x => (await client.GetItems(x)).ToArray()));
// This is my new approach (also this didn't worked...)
var items = new List<item[]>();
ids.AsParallel().Select(x => client.GetItems(x).GetAwaiter().GetResult()).ForAll(item =>
{
//Console.WriteLine("processed!");
items.Add(item.ToArray());
});
var result = items.ToArray();
As you can see I put Console.WriteLine("processed!"); statment, in order to check if anything worked... But I can't manage this to work.
Those are my other methods:
private static IEnumerable<ulong[]> PrepareIds(int startingId, int idsNum = 1000, int maxTasks = 100)
{
for (int i = 0; i < maxTasks; i++)
yield return Range((ulong)startingId, (ulong)(startingId + idsNum)).ToArray();
}
And...
public static async Task<IEnumerable<item>> GetItems(this HttpClient client, ulong[] ids, Action notFoundCallback = null)
{
var keys = PrepareDataInKeys(type, ids); // This prepares the data to be sent to the API server
var content = new FormUrlEncodedContent(keys);
content.Headers.ContentType =
new MediaTypeHeaderValue("application/x-www-form-urlencoded") { CharSet = "UTF-8" };
client.DefaultRequestHeaders.ExpectContinue = false;
// We create a post request
var response = await client.PostAsync(new Uri(FILE_URL), content);
string contents = null;
JObject jObject;
try
{
contents = await response.Content.ReadAsStringAsync();
jObject = JsonConvert.DeserializeObject<JObject>(contents);
}
catch (Exception ex)
{
Console.WriteLine(ex);
Console.WriteLine(contents);
return null;
}
// Then we read the items from the parsed JObject
JArray items;
try
{
items = jObject
.GetValue("...")
.ToObject<JObject>()
.GetValue("...").ToObject<JArray>();
}
catch (Exception ex)
{
Console.WriteLine(ex);
return null;
}
int notFoundItems = 0;
int nonxxxItems = 0;
int xxxItems = 0;
var all = items.BuildEnumerable(notFoundCallback, item =>
{
if (item.Result != 1)
++notFoundItems;
else if (item.id != 5)
++nonxxxItems;
else
++xxxItems;
});
CrawledItems += ids.Length;
NotFoundItems += notFoundItems;
NonXXXItems += nonxxxItems;
XXXItems += xxxItems;
return all;
}
private static IEnumerable<item> BuildEnumerable(this JArray items, Action notFoundCallback, Action<item> callback = null)
{
foreach (var item in items)
{
item _item;
try
{
_item = new item(item.ToObject<JObject>());
callback?.Invoke(_item);
}
catch (Exception ex)
{
if (notFoundCallback == null)
Console.WriteLine(ex, Color.Red);
else
notFoundCallback();
continue;
}
yield return _item;
}
}
So as you can see I create 100 parallel post requests using an HttpClient. But I can't manage it to work.
So the thing that I want to achieve is to retrieve as many items as possible because I need to crawl +2,000,000,000 items.
But any breakpoint is triggered, neither any caption is updated on Console (I'm using Konsole project in order to print values at a fixed position on console), so any advice can be given there?
So I'm pulling in a list of items and for each item I'm creating an instance of an object to run a task on that item. All the objects are the same, they updated based off of a received message every three seconds. This update does not all occur at once though, sometimes it takes 3.1 seconds, etc. This is data I need to serialize in XML once it all exists so I'm looking for a way to see when its all done.
I've explored tasks in .net 4.6 but that initiates a task and it reports complete and then to run again the task class would initiate it again but in my case that won't work because each instance stays alive and initiates itself when a new message comes in.
What is the best way to have it report it reached the last line of code and then look at a list of these instances and say when all of them show as complete then run task to serialize?
I've included code below of the instance that is running.
private void OnMessageReceived(object sender, MessageReceivedEventArgs e)
{
var eventArgs = new CallDataReceivedEventArgs();
this.OnCallDataReceived(eventArgs);
try
{
List<Tuple<String, TimeSpan>> availInItems = new List<Tuple<string, TimeSpan>>();
List<Tuple<string, int, TimeSpan, string, string, string>> agentlist = new List<Tuple<string, int, TimeSpan, string, string, string>>();
if (e == null)
{
return;
}
List<TimeSpan> listOfTimeSpans = new List<TimeSpan>();
if (e.CmsData != null)
{
#region Gathering Agent Information
// Create a list of all timespans for all _agents in a queue using the property AgentTimeInState
foreach (var item in e.CmsData.Agents)
{
//AgentData = new ScoreBoardAgentDataModel(AgentName, AgentExtension, AgentTimeInState, AgentAuxReason, AgentId, AgentAdcState);
_agentData.AgentName = item.AgName;
_agentData.AgentExtension = item.Extension;
_agentData.AgentAuxReason = item.AuxReasonDescription;
_agentData.AgentId = item.LoginId;
_agentData.AgentAcdState = item.WorkModeDirectionDescription;
_agentData.AgentTimeInState = DateTime.Now - item.DateTimeUpdated;
_agentData.TimeSubmitted = DateTime.Now;
agentlist.Add(Tuple.Create(_agentData.AgentName, _agentData.AgentExtension, _agentData.AgentTimeInState, _agentData.AgentId, _agentData.AgentAcdState, _agentData.AgentAuxReason));
if (_agentData.AgentAcdState == "AVAIL")
{
listOfTimeSpans.Add(_agentData.AgentTimeInState);
availInItems.Add(Tuple.Create(_agentData.AgentName, _agentData.AgentTimeInState));
}
availInItems.Sort((t1, t2) => t1.Item2.CompareTo(t2.Item2));
}
var availInAgents =
agentlist
.Where(ag => ag.Item5 == "AVAIL")
.ToList();
availInAgents.Sort((t1, t2) =>
t1.Item3.CompareTo(t2.Item3));
var max3 = availInAgents.Skip(availInAgents.Count - 3);
max3.Reverse();
_agents.AgentsOnBreak = 0;
foreach (var agent in agentlist)
{
if (!string.IsNullOrEmpty(agent.Item6) && agent.Item6.StartsWith("Break"))
{
_agents.AgentsOnBreak++;
}
}
_agents.AgentsOnLunch = 0;
foreach (var agent in agentlist)
{
//If the current agent's aux reason is Lunch
if (!string.IsNullOrEmpty(agent.Item6) && agent.Item6.StartsWith("Lunch"))
{
//add one to agentsonlunch
_agents.AgentsOnLunch++;
}
}
_agents.NextInLine = string.Empty;
foreach (var agent in max3.Reverse())
{
//assign agent to NextInLine and start a new line
_agents.NextInLine += agent.Item1 + Environment.NewLine;
//reverse NextInLine
_agents.NextInLine.Reverse();
}
_agents.TimeSubmitted = DateTime.Now;
#endregion
#region Gathering Skill Information
_skillData.OldestCall = e.CmsData.Skill.OldestCall;
_skillData.AgentsStaffed = e.CmsData.Skill.AgentsStaffed;
_skillData.AgentsAuxed = e.CmsData.Skill.AgentsInAux;
_skillData.AgentsAvailable = e.CmsData.Skill.AgentsAvailable;
_skillData.AgentsOnCalls = e.CmsData.Skill.AgentsOnAcdCall;
_skillData.CallsWaitingInQueue = e.CmsData.Skill.InQueueInRing;
_skillData.Asa = e.CmsData.Skill.AnswerTimePerAcdCall;
_skillData.TimeSubmitted = DateTime.Now;
_skillData.EstimatedHoldTimeLow = e.CmsData.Skill.ExpectedWaitTimeLow;
_skillData.EstimatedHoldTimeMedium = e.CmsData.Skill.ExpectedWaitTimeMedium;
_skillData.EstimatedHoldTimeHigh = e.CmsData.Skill.ExpectedWaitTimeHigh;
#endregion
}
}
catch (Exception ex)
{
_logger.Info(ex.Message, ex);
}
}
With tasks you can start many at the same time and wait for them all to finish like this:
var taskList = new List<Task>();
foreach (var thingToDo in work)
{
taskList.Add(thingToDo.StartTask());
}
Task.WaitAll(taskList.ToArray());
This way you can run everything in parallel and wont get after the last line until everything is done.
Edit following your comment
You can embed your work in a task with this:
public async Task DoWork()
{
var taskList = new List<Task>();
foreach (var thingToDo in work)
{
taskList.Add(thingToDo.StartTask());
}
await Task.WhenAll(taskList.ToArray());
}
This is my publisher. There are two consumers. MailConsumer and SmsConsumer.
using(var bus = RabbitHutch.CreateBus("host=localhost").Advanced) {
var queueName = "my.queue";
var queueName2 = "my.queue2";
var queue = bus.QueueDeclare(queueName);
var queue2 = bus.QueueDeclare(queueName2);
var channel = bus.ExchangeDeclare("MyFanout", ExchangeType.Fanout);
bus.Bind(channel, queue, "sms");
bus.Bind(channel, queue2, "mail");
var input = "";
Console.WriteLine("Enter a message. 'q' to quit.");
while((input = Console.ReadLine()) != "q") {
for(int i = 1; i < 2; i++) {
var message = new Message<TextMessage>(new TextMessage {
Text = input + i
});
bus.Publish(channel, "", false, false, message);
}
}
}
I can subscribe with this code:
using(var bus = RabbitHutch.CreateBus("host=localhost").Advanced) {
var queueName = "my.queue2";
var queue = bus.QueueDeclare(queueName);
bus.Consume(queue, x => x.Add<TextMessage>((message, info) => {
Thread.Sleep(1000);
Console.WriteLine("SMS: {0}", message.Body.Text);
}));
Console.WriteLine("Listening for messages. Hit <return> to quit.");
Console.ReadLine();
}
How can I achieve it via AutoSubscriber? There is no option for Queue Name in AutoSubscriber, there is "Subscription Id"
There's a property on the AutoSubscriber called 'GenerateSubscriptionId', which you can set to generate the subscription id for a consumer:
subscriber.GenerateSubscriptionId = subscriptionInfo =>
{
return "MyApplication:" + subscriptionInfo.ConcreteType.Name);
};
Then the subscription id will be used by the default conventions to generate a queue name.
Update:
I think you can achieve what you want just with normal pub-sub without declaring any queue.
In your publisher you can do this:
var bus = RabbitHutch.CreateBus("host=localhost")
while((input = Console.ReadLine()) != "q") {
for(int i = 1; i < 2; i++) {
var message = new Message<TextMessage>(new TextMessage {
Text = input + i
});
bus.Publish(message);
}
}
Then just create 2 consumers to subscribe to your message, and your message will be handled by both:
class SmsConsumer : IConsume<TextMessage>{
...
}
class LogConsumer : IConsume<TextMessage>{
...
}
and in your startup:
var bus = RabbitHutch.CreateBus("host=localhost")
var subscriber = new AutoSubscriber(bus,"my_applications_subscriptionId_prefix");
subscriber.Subscribe(Assembly.GetExecutingAssembly());
Console.ReadLine()
EasyNetQ will declare the exchange, queues and bindings for you. Just make sure your TextMessage class is in an assembly shared by both projects.
I have a telephony application, in which I want to invoke simultaneous calls,. Each call will occupy a channel or port. So I added all channels to a BlockingCollection. The application is a windows service.
Let's see the code.
public static BlockingCollection<Tuple<ChannelResource, string>> bc = new BlockingCollection<Tuple<ChannelResource, string>>();
public static List<string> list = new List<string>();// then add 100 test items to it.
The main application has the code:
while (true)
{
ThreadEvent.WaitOne(waitingTime, false);
lock (SyncVar)
{
Console.WriteLine("Block begin");
for (int i = 0; i < ports; i++)
{
var firstItem = list.FirstOrDefault();
if (bc.Count >= ports)
bc.CompleteAdding();
else
{
ChannelResource cr = OvrTelephonyServer.GetChannel();
bc.TryAdd(Tuple.Create(cr, firstItem));
list.Remove(firstItem);
}
}
pc.SimultaneousCall();
Console.WriteLine("Blocking end");
if (ThreadState != State.Running) break;
}
Now for the simultaneous call code:
public void SimultaneousCall()
{
Console.WriteLine("There are {0} channels to be processed.", bc.Count);
var workItemBlock = new ActionBlock<Tuple<ChannelResource, string>>(
workItem =>
{
ProcessEachChannel(workItem);
});
foreach (var workItem in bc.GetConsumingEnumerable())
{
bool result = workItemBlock.SendAsync(workItem).Result;
}
workItemBlock.Complete();
}
private void ProcessEachChannel(Tuple<ChannelResource, string> workItem)
{
ChannelResource cr = workItem.Item1;
string sipuri = workItem.Item2;
VoiceResource vr = workItem.Item1.VoiceResource;
workItem.Item1.Disconnected += new Disconnected(workItemItem1_Disconnected);
bool success = false;
try
{
Console.WriteLine("Working on {0}", sipuri);
DialResult dr = new DialResult();
// blah blah for calling....
}
catch (Exception ex)
{
Console.WriteLine("Exception: {0}", ex.Message);
}
finally
{
if (cr != null && cr.VoiceResource != null)
{
cr.Disconnect();
cr.Dispose();
cr = null;
Console.WriteLine("Release channel for item {0}.", sipuri);
}
}
}
The question was when I tested the application with 4 ports, I thought the code should reach at
Console.WriteLine("Blocking end");
However it was not. Please see the snapshot.
The application is just hanging on after releasing the last channel. I guess that I may use the blockingcollection incorrectly. Thanks for help.
UPDATE:
Even I changed the code by using POST action as below, the situation is still unchanged.
private bool ProcessEachChannel(Tuple<ChannelResource, string> workItem)
{
// blah blah to return true or false respectively.
public void SimultaneousCall()
{
Console.WriteLine("There are {0} channels to be processed.", bc.Count);
var workItemBlock = new ActionBlock<Tuple<ChannelResource, string>>(
workItem =>
{
bool success = ProcessEachChannel(workItem);
});
foreach (var workItem in bc.GetConsumingEnumerable())
{
workItemBlock.Post(workItem);
}
workItemBlock.Complete();
}
I believe the problem is that you never call bc.CompleteAdding(): the if means it would be called in ports + 1-th iteration of the loop, but the loop iterates only ports-times. Because of this, GetConsumingEnumerable() returns a sequence that never ends, which means the foreach inside SimultaneousCall() blocks forever.
I think the right solution is to call bc.CompleteAdding() after the for loop, not in an impossible condition inside it.