I have a BlockingCollection with one producer and one consumer in my application. The consumer writes the received data to a file. I need to make a real-time graph drawing from this data.
Is it possible to run a second consumer that would draw graph with the same data from the collection in parallel?
however, is it so that when one consumer takes out the data, the other consumer can no longer access it? In the sense that only one copy of the data can be taken out of the collection? (it seems to me that this is how this 'buffer' works).
in that case, is it a good idea that in this consumer that I have to write data to a file to do at the same time adding this data to the other defined BlockingCollection: collection2.Add(dataBuffer);? And then create a consumer2 that will draw the graph? I still just don't know how in this case consumer2 would get to the graph, which would be part of the GUI thread.
// -----------------------------------------------------------------
// A method that fetches data from a queue and writes it to a file.
// It works in a dedicated Task.
// -----------------------------------------------------------------
static void Consumer()
{
while (true)
{
foreach (DataBuffer dataBuffer in collection.GetConsumingEnumerable())
{
_FileStream.Write(dataBuffer.Buffer, 0, dataBuffer.Length);
_FileStream.Flush();
}
// adding dataBuffer to the second consumer here?
}
}
To be more precise. My question is how to organize such an application in an optimal way, because I have no experience (until a month ago, I had no idea about the existence of BlockingCollection).
At the moment:
I have Form window of the application,
I have an object in which a Task is started, which receives data from an external device,
after receiving the data, an Event is generated, with which I pass the received data,
I have another object in which a Handler for this Event runs, and in this Handler the data is taken from the argument and is added to the BlockingCollection,
I have the consumer Task running, which saves the received to a file.
Now I also need to add drawing an overview chart from the same data as it goes to the file and I am wondering how to do it.
Related
If the processing of an Azure Service Bus message depends on another resource, e.g. an API or a database service, and this resource is not available, not calling CompleteMessageAsync() is not an option, because the message will be immediately received again until the Max Delivery Count is reached, and then put into the DLQ. If an API is down for maintenance, we want to wait a bit before retrying.
One of the answers to this question has the general steps for deferring and receiving deferred messages. This is a little better than Microsoft's documentation, but not enough for me to understand the intent of the API, and how it is to be implemented in a hosted service that basically sits in ServiceBusProcessor.StartProcessingAsync all day long.
This is the basic structure of my service:
public class ServiceBusWatcher : IHostedService, IDisposable
{
public Task StartAsync(CancellationToken stoppingToken)
{
ReceiveMessagesAsync();
return Task.CompletedTask;
}
private async void ReceiveMessagesAsync()
{
ServiceBusClient client = new ServiceBusClient(connectionString);
processor = client.CreateProcessor(queueName, new ServiceBusProcessorOptions());
processor.ProcessMessageAsync += MessageHandler;
await processor.StartProcessingAsync();
}
async Task MessageHandler(ProcessMessageEventArgs args)
{
// a dependency is not available that allows me to process a message. so:
await args.DeferMessageAsync(args.Message);
Once the message is deferred, it is my understanding that the processor will not get to it anymore (or will it?). Instead, I have to use ReceiveDeferredMessageAsync() to receive it, along with the sequence number of the originally received message.
In my case, it will make sense to wait minutes or hours before trying again.
This could be done with a separate service that uses a timer and an explicit call to ReceiveDeferredMessageAsync(), as opposed to using a ServiceBusProcessor. I also suppose that the deferred message sequence numbers will have to be persisted in non-volatile storage so that they don't get lost.
Does this sound like a viable approach? I don't like having to remember its sequence numbers so that I can get to a message later. It goes against everything that using a message queue brings to the table in the first place.
Or, instead of deferring, I could just post a new "internal" message with the sequence number and use the ScheduledEnqueueTimeUtc property to delay receiving it. Once I receive this message, I could call ReceiveDeferredMessageAsync() with that sequence number to get to the original message. This seems elegant at the surface, but messages could quickly multiply if there is a longer outage of a dependency.
Another idea that could work without another service: I could complete and repost the payload of the message and set ScheduledEnqueueTimeUtc to a time in the future, as described in another answer to the question I mentioned earlier. Assuming that this works (Microsoft's documentation does not mention what this property is for), it seems simple and clean, and I like simple.
How have you solved this? Is there a better/preferred way that balances low complexity with high robustness without requiring a large amount of code?
Deferring a message works when you know what message you want to retrieve later and your receiver will have the message sequence number saved to retrieve the deferred message. If the receiver has no ability to save message sequence number, the delaying the message is a better option. Delaying a message will mean to copy the original message data into a newly scheduled one and completing the original message. That way the consumer doesn't have to neither hold on to the message sequence number nor initiate the retrieval of a specific message.
I'm working on a P2P application in C#.
It's a file transfer with a file splitting, and text chat.
On a client there are 2 threads, 1 for listening, 1 for sending.
When i send a file, it's first split into let's say 10 pieces, these 10 pieces are added to a send queue in the client, it then starts sending file chunk 1.
But now i want to send a message through the same pipe.
My idea is then to insert that message into the send list before file chunk 2.
What kind of threading do i need for 2 threads to work on the same list?
I have accounted for the objects being received this way.
My initial idea for the send function was something along these lines:
public void Send()
{
while (IsConnected())
{
if (unSentObjects.Count > 1)
{
Task sendTask = new Task(() => SendObj(unSentObjects[0]));
sendTask.Start();
}
}
}
You could use a Synchronization Object such as a mutex to prevent race conditions or simultaneous write/read to same file. Basically only 1 thread will be able to access the object.
If the data is global to the threads and they are all once process, you can use the synchronization object simply to signal when to use global shared data and when to not use it. Other than that using the shared global data is exactly the same, you are just trafficking the use of it.
I originally had a race condition when sending data, the issue was that I was allowing multiple SocketAsyncEventArgs to be used to send data, but the first packet didn't send fully before the 2nd packet, this is because I have it so if the data doesn't fit in the buffer it loops until all the data is sent, and the first packet was larger than the second packet which is tiny, so the second packet was being sent and reached to the client before the first packet.
I have solved this by assigning 1 SocketAyncEventArgs to an open connection to be used for sending data and used a Semaphore to limit the access to it, and make the SocketAsyncEventArgs call back once it completed.
Now this works fine because all data is sent, calls back when its complete ready for the next send. The issue with this is, its causing blocking when I want to send data randomly to the open connection, and when there is a lot of data sending its going to block my threads.
I am looking for a work around to this, I thought of having a Queue which when data is requested to be sent, it simply adds the packet to the Queue and then 1 SocketAsyncEventArgs simply loops to send that data.
But how can I do this efficiently whilst still being scalable? I want to avoid blocking as much as I can whilst sending my packets in the order they are requested to be sent in.
Appreciate any help!
If the data needs to be kept in order, and you don't want to block, then you need to add a queue. The way I do this is by tracking, on my state object, whether we already have an active send async-loop in process for that connection. After enqueue (which obviously must be synchronized), just check what is in-progress:
public void PromptToSend(NetContext context)
{
if(Interlocked.CompareExchange(ref writerCount, 1, 0) == 0)
{ // then **we** are the writer
context.Handler.StartSending(this);
}
}
Here writerCount is the count of write-loops (which should be exactly 1 or 0) on the connection; if there aren't any, we start one.
My StartSending tries to read from that connection's queue; if it can do so, it does the usual SendAsync etc:
if (!connection.Socket.SendAsync(args)) SendCompleted(args);
(note that SendCompleted here is for the "sync" case; it would have got to SendCompleted via the event-model for the "async" case). SendCompleted repeats this "dequeue, try send async" step, obviously.
The only thing left is to make sure that when we try to dequeue, we note the lack of action if we find nothing more to do:
if (bufferedLength == 0)
{ // nothing to do; report this worker as inactive
Interlocked.Exchange(ref writerCount, 0);
return 0;
}
Make sense?
I'm writing a message layer for my distributed system. I'm using IOCP, ie the Socket.XXXAsync methods.
Here's something pretty close to what I'm doing (in fact, my receive function is based on his):
http://vadmyst.blogspot.com/2008/05/sample-code-for-tcp-server-using.html
What I've found now is that at the start of the program (two test servers talking to each other) I each time get a number of SAEA objects where the .Buffer is entirely filled with zeroes, yet the .BytesTransferred is the size of the buffer (1024 in my case).
What does this mean? Is there a special condition I need to check for? My system interprets this as an incomplete message and moves on, but I'm wondering if I'm actually missing some data. I was under the impression that if nothing was being received, you'd not get a callback. In any case, I can see in WireShark that there aren't any zero-length packets coming in.
I've found the following when I Googled it, but I'm not sure my problem is the same:
http://social.msdn.microsoft.com/Forums/en-US/ncl/thread/40fe397c-b1da-428e-a355-ee5a6b0b4d2c
http://go4answers.webhost4life.com/Example/socketasynceventargs-buffer-not-ready-121918.aspx
I am sure not what is going on in the linked example. It appears to be using asynchronous sockets in a synchronous way. I cannot see any callbacks or similar in the code. You may need to rethink whether you need synchronous or asynchronous sockets :).
To the problem at hand stems from the possibility that your functions are trying to read/write to the buffer before the network transmit/receive has been completed. Try using the callback functionality included in the async Socket. E.g.
// This goes into your accept function, to begin receiving the data
socketName.BeginReceive(yourbuffer, 0, yourbuffer.Length,
SocketFlags.None, new AsyncCallback(OnRecieveData), socketName);
// In your callback function you know that the socket has finished receiving data
// This callback will fire when the receive is complete.
private void OnRecieveData(IAsyncResult input) {
Socket inSocket = (Socket)input.AsyncState; // This is just a typecast
inSocket.EndReceive(input);
// Pull the data out of the socket as you already have before.
// state.Data.Write ......
}
My application connects to a device and sends multiple commands across a single socket connection. It then reads the response to these the basic structure is
command 1
stream.write
stream.read
command 2
stream.write
stream.read
.
.
.
i am wondering if there is a better way of doing this. I am not worried about blocking because this is running on a different thread than the rest of the program. the problem i am encountering is that sometimes the data for command 1 lands in the read for command 2. The other thing is the 1st byte that i receive is unique to the command.
any help would be appreciated
Assuming TCP - there is no way to ensure that each command is read as it was sent. At the destination end, each command can be fragmented or joined to other commands, so you need to manually decide where the boundaries are between them.
A common technique is to prefix the commands with their length, which you can read first, and know precisely how many bytes to read before the next one. At the destination end, you usually have some kind of queue which you push all received data onto, and you read off the queue one command at a time, only when there is one or more completely received commands.
I wouldn't recommend using blocking sockets under any circumstances really, even if you're using a separate thread. If you need to both send and receive on the same socket, you could encounter issues where you attempt to call Read when no data is waiting, and you will not be able to send any data until some is received.
Rather than using the blocking calls, use BeginRead,EndRead for asynchronous receiving, then you'll be able to send and receive in the same thread without having those worries.
Because you are in multithreading, use lock around sending commands, Like:
public void SendCommand(Command command)
{
lock (_commandLocker)
{
stream write
stream read
}
}
So only one command at time will send and receive data.
However, if you are receiving data from the device at any time "maybe it sends a notifications.." then consider do something like:
private Queue<Notification> _notificationsBuffer = new Queue<Notification>();//Or use BlockingCollection if your are using 4.0
At SendCommand
...
while (stream read)
{
if (this is a notification)
{
then add it to the notification buffer and continue read
continue;
}
else (this is our command)
{ ... read the response..}
}