Develop Reference

DropQueue mechanism for RX.net

I came across a back pressure issue with RX.net I can't find a solution for. I have an observable real-time stream of log messages.
var logObservable = /* Observable stream of log messages */
Which I want to expose via a TCP interface which serializes the real-time log messages from the logObservable before they are sent over the wire. So I do the following:
foreach (var message in logObservable.ToEnumerable())
{
// 1. Serialize message
// 2. Send it over the wire.
}
The problem arises with the .ToEnumerable() if a back pressure scenario happens e.g. if the client on the other end pauses the stream. The problem is that .ToEnumerable() caches the items which result in a lot of memory usage. I'm looking for a mechanism something like a DropQueue which only buffers, let say, the last 10 messages e.g.
var observableStream = logObservable.DropQueue(10).ToEnumerable();
Is this the right way to way to solve this issue? And do you know to implement such a mechanism to avoid possible back pressure issue?

My DropQueue implementation:
public static IEnumerable<TSource> ToDropQueue<TSource>(
this IObservable<TSource> source,
int queueSize,
Action backPressureNotification = null,
CancellationToken token = default(CancellationToken))
{
var queue = new BlockingCollection<TSource>(new ConcurrentQueue<TSource>(), queueSize);
var isBackPressureNotified = false;
var subscription = source.Subscribe(
item =>
{
var isBackPressure = queue.Count == queue.BoundedCapacity;
if (isBackPressure)
{
queue.Take(); // Dequeue an item to make space for the next one
// Fire back-pressure notification if defined
if (!isBackPressureNotified && backPressureNotification != null)
{
backPressureNotification();
isBackPressureNotified = true;
}
}
else
{
isBackPressureNotified = false;
}
queue.Add(item);
},
exception => queue.CompleteAdding(),
() => queue.CompleteAdding());
token.Register(() => { subscription.Dispose(); });
using (new CompositeDisposable(subscription, queue))
{
foreach (var item in queue.GetConsumingEnumerable())
{
yield return item;
}
}
}

Client hangs up then getting a non empty list from server

I am using a combination of Entity Framework and WCF to provide Data for a client. But for some reason the client hang up (only when result is a non empty list) then calling a simple server function:
public List<CardSet> GetCollactions(string
{
try
{
if (db.Users.Any(x => x.username == username))
{
User requestingUser = db.Users.FirstOrDefault(x => x.username == username);
List<CardSet> result = requestingUser.Collections;
if (result != null)
return result;
}
}
catch (Exception e)
{
db.Errors.Add(new Error() { exception = e.Message, innerException = e.InnerException.Message, source = "GetCollection for user '" + username + "'", time = DateTime.Now });
db.SaveChanges();
}
return new List<CardSet>();
}
On the server the function is perfectly executed, the result for the database is correct. I call the function on the client like this:
List<CardSet> collections = client.GetCollactions(username).ToList();
foreach (CardSet collection in collections)
{
CollectionList.Items.Add(collection.name);
}
Edit: Client is a WPF form.

You must request from service in async manner to avoid client hangs up, First request async data from list in a thread and when data received continued operation in UI thread:
Task.Factory.StartNew(() => client.GetCollactions(username).ToList())
.ContinueWith(result =>
{
foreach (CardSet collection in result.Result)
{
CollectionList.Items.Add(collection.name);
}
},TaskScheduler.FromCurrentSynchronizationContext());

BlockingCollection Consumer is repeating output

TL;DR I have an application that is reading messages from a USB device in the background, and displaying the messages on the screen. I am using a BlockingCollection, as I need to read messages quickly so the device does not get a BufferOverflow.
I am reading messages like this (my producer):
private void ReadMessages(BlockingCollection<object> logMessages)
{
uint numMsgs;
Status status;
Message[] msgs = new Message[10];
while(!logMessages.IsAddingCompleted)
{
numMsgs = (uint) msgs.Length;
status = readMessages(channel, msgs, ref numMsgs, 1000);
if(status == Status.ERR_BUFFER_OVERFLOW)
{
logMessages.Add("BUFFER OVERFLOW - MESSAGES LOST!");
logMessages.Add(CopyMessages(msgs, numMsgs));
}
else if(status == Status.STATUS_NOERROR)
{
logMessages.Add(CopyMessages(msgs, numMsgs));
}
else
{
throw new Exception("Error");
}
}
The readMessages() method will fill the msgs array with the Message objects read, and the numMsgs reference holds the number of messages that were read (up to 10). I use a function called CopyMessages() so I only pass a Message[] that is the right size. i.e. if 5 messages are read, I send a Message[5] instead of Message[10].
I read the messages (my consumer) like this:
private void DisplayMessages(BlockingCollection<object> messages)
{
string[] msgs;
try
{
foreach (var item in messages.GetConsumingEnumerable(_cancellationTokenSource.Token))
{
if (item is string)
{
msgs = new string[] { item.ToString() };
}
else if (item is PassThruMsg[])
{
msgs = FormatMessages((PassThruMsg[])item);
}
else
{
msgs = new string[0];
}
Task.Factory.StartNew(new Action(() => outputTextBox.AppendText(String.Join(Environment.NewLine, msgs) + Environment.NewLine)), _cancellationTokenSource.Token, TaskCreationOptions.None, uiContext);
}
}
catch (OperationCanceledException)
{
//TODO:
}
}
I start the tasks inside a button click, like this:
var results = new BlockingCollection<object>();
var display = Task.Factory.StartNew(() => DisplayMessages(results));
var readMessages = Task.Factory.StartNew(() => ReadMessages(results));
Task[] tasks = new Task[] { display, readMessages };
try
{
await Task.Factory.ContinueWhenAll(tasks, result => { results.CompleteAdding(); }, _cancellationTokenSource.Token, TaskContinuationOptions.None, uiContext);
}
catch (TaskCanceledException)
{
//TODO:
}
This works fine, and when running idly it prints the messages from the device without a problem. However, after the device starts doing work under a really heavy load (the consumer is called so quickly it locks the UI temporarily) that I notice the output textbox is repeating values. It is my understanding that GetConsumingEnumerable() also removes items from the blocking collection, but I don't know why else I would see the messages printed multiple times. Each message has a timestamp, and when I readMessages from the device it clears the buffer so I know that I am not reading that message multiple times.
Am I missing something here? Is there a better way to handle this producer/consumer scenario to ensure accurate data? I have looked to see if there are references somewhere that may be overlapping, but I don't see it.

Queue to ConcurrentQueue

I have a regular Queue object in C# (4.0) and I'm using BackgroundWorkers that access this Queue.
The code I was using is as follows:
do
{
while (dataQueue.Peek() == null // nothing waiting yet
&& isBeingLoaded == true // and worker 1 still actively adding stuff
)
System.Threading.Thread.Sleep(100);
// otherwise ready to do something:
if (dataQueue.Peek() != null) // because maybe the queue is complete and also empty
{
string companyId = dataQueue.Dequeue();
processLists(companyId);
// use up the stuff here //
} // otherwise nothing was there yet, it will resolve on the next loop.
} while (isBeingLoaded == true // still have stuff coming at us
|| dataQueue.Peek() != null); // still have stuff we haven’t done
However, I guess when dealing with threads I should be using a ConcurrentQueue.
I was wondering if there were examples of how to use a ConcurrentQueue in a Do While Loop like above?
Everything I tried with the TryPeek wasn't working..
Any ideas?

You can use a BlockingCollection<T> as a producer-consumer queue.
My answer makes some assumptions about your architecture, but you can probably mold it as you see fit:
public void Producer(BlockingCollection<string> ids)
{
// assuming this.CompanyRepository exists
foreach (var id in this.CompanyRepository.GetIds())
{
ids.Add(id);
}
ids.CompleteAdding(); // nothing left for our workers
}
public void Consumer(BlockingCollection<string> ids)
{
while (true)
{
string id = null;
try
{
id = ids.Take();
} catch (InvalidOperationException) {
}
if (id == null) break;
processLists(id);
}
}
You could spin up as many consumers as you need:
var companyIds = new BlockingCollection<string>();
Producer(companyIds);
Action process = () => Consumer(companyIds);
// 2 workers
Parallel.Invoke(process, process);

How can I call a function that requires a return value that calls WebClient?

Here is the function I have now that obviously doesn't work. The reason it doesn't work is because WebClient is asynchronous and data is empty before it gets filled by WebClient and crashes on the XML reader. How can I call WebClient within this function and still allow it to return ServerResult as required with or without needing an external event handler?
static public ServerResult isBarcodeCorrectOnServer(string barcode)
{
Dictionary<string, IPropertyListItem> dict = configDictionary();
string urlString = (string.Format("http://www.myurl.com/app/getbarcodetype.php?realbarcode={0}&type={1}", barcode, dict["type"]));
WebClient wc = new WebClient();
string data = "";
wc.DownloadStringCompleted += (sender, e) =>
{
if (e.Error == null)
{
//Process the result...
data = e.Result;
}
};
wc.DownloadStringAsync(new Uri(urlString));
StringReader stream = new StringReader(data);
var reader = XmlReader.Create(stream);
var document = XDocument.Load(reader);
var username = document.Descendants("item");
var theDict = username.Elements().ToDictionary(ev => ev.Name.LocalName, ev => ev.Value);
if (theDict.ContainsKey("type") == true && theDict["type"].ToString() == dict["type"].ToString())
{
return ServerResult.kOnServer;
}
else if (theDict.ContainsKey("type") == true)
{
return ServerResult.kWrongType;
}
else
{
return ServerResult.kNotOnServer;
}
}

You can't without "hacks" and you shouldn't - embrace asynchrony and pass in a delegate that you want to be executed once the download is completed:
static public void isBarcodeCorrectOnServer(string barcode, Action<string> completed)
{
//..
wc.DownloadStringCompleted += (sender, e) =>
{
if (e.Error == null)
{
//Process the result...
data = e.Result;
completed(data);
}
};
//..
}
You can move all your processing code now into a separate method which you call with the download result.

You can't, basically. Or at the very least you shouldn't. You've got a method which is designed to be synchronous, on a platform which is designed for asynchronous IO.
Fundamentally, you should design your code to work with the platform. Accept that it will be asynchronous, and make the calling code deal with that.
Note that when C# 5 comes out and when it's supported by Windows Phone, all of this will be a lot simpler using async. You'd return a Task<ServerResult> from the method, and await the result of the WebClient. If you're only developing for fun (so don't mind working with a CTP which has some bugs, and may not be valid to ship for marketplace apps) you can do that today.