TL;DR I have an application that is reading messages from a USB device in the background, and displaying the messages on the screen. I am using a BlockingCollection, as I need to read messages quickly so the device does not get a BufferOverflow.
I am reading messages like this (my producer):
private void ReadMessages(BlockingCollection<object> logMessages)
{
uint numMsgs;
Status status;
Message[] msgs = new Message[10];
while(!logMessages.IsAddingCompleted)
{
numMsgs = (uint) msgs.Length;
status = readMessages(channel, msgs, ref numMsgs, 1000);
if(status == Status.ERR_BUFFER_OVERFLOW)
{
logMessages.Add("BUFFER OVERFLOW - MESSAGES LOST!");
logMessages.Add(CopyMessages(msgs, numMsgs));
}
else if(status == Status.STATUS_NOERROR)
{
logMessages.Add(CopyMessages(msgs, numMsgs));
}
else
{
throw new Exception("Error");
}
}
The readMessages() method will fill the msgs array with the Message objects read, and the numMsgs reference holds the number of messages that were read (up to 10). I use a function called CopyMessages() so I only pass a Message[] that is the right size. i.e. if 5 messages are read, I send a Message[5] instead of Message[10].
I read the messages (my consumer) like this:
private void DisplayMessages(BlockingCollection<object> messages)
{
string[] msgs;
try
{
foreach (var item in messages.GetConsumingEnumerable(_cancellationTokenSource.Token))
{
if (item is string)
{
msgs = new string[] { item.ToString() };
}
else if (item is PassThruMsg[])
{
msgs = FormatMessages((PassThruMsg[])item);
}
else
{
msgs = new string[0];
}
Task.Factory.StartNew(new Action(() => outputTextBox.AppendText(String.Join(Environment.NewLine, msgs) + Environment.NewLine)), _cancellationTokenSource.Token, TaskCreationOptions.None, uiContext);
}
}
catch (OperationCanceledException)
{
//TODO:
}
}
I start the tasks inside a button click, like this:
var results = new BlockingCollection<object>();
var display = Task.Factory.StartNew(() => DisplayMessages(results));
var readMessages = Task.Factory.StartNew(() => ReadMessages(results));
Task[] tasks = new Task[] { display, readMessages };
try
{
await Task.Factory.ContinueWhenAll(tasks, result => { results.CompleteAdding(); }, _cancellationTokenSource.Token, TaskContinuationOptions.None, uiContext);
}
catch (TaskCanceledException)
{
//TODO:
}
This works fine, and when running idly it prints the messages from the device without a problem. However, after the device starts doing work under a really heavy load (the consumer is called so quickly it locks the UI temporarily) that I notice the output textbox is repeating values. It is my understanding that GetConsumingEnumerable() also removes items from the blocking collection, but I don't know why else I would see the messages printed multiple times. Each message has a timestamp, and when I readMessages from the device it clears the buffer so I know that I am not reading that message multiple times.
Am I missing something here? Is there a better way to handle this producer/consumer scenario to ensure accurate data? I have looked to see if there are references somewhere that may be overlapping, but I don't see it.
Related
I have looked all over for a solution to an issue. I have noticed that in my android app, every time I fire an event from <button Clicked="GPSToggle_Clicked">, for some reason it increments the number of times my methods get called. So after I compile and load this on my phone, I hit my "GPSToggle_Clicked" button, and then to stop hit that button again. On the first "stop", I'll get a single instance of the below output:
---------------------------------------------------------------Attempting string parsing
---------------------------------------------------------------Sending string to SubmitGeneratedGPX
---------------------------------------------------------------path: /storage/emulated/0/Download/GPX/2022-10-27-02-44-06.gpx
---------------------------------------------------------------GPX File creation success
---------------------------------------------------------------:RawBufferToJsonString: [{"Timestamp":"2022-10-27T18:43:52.849+00:00","Latitude":41.5263818,"Longitude":-81.6507923,"Altitude":153.29998779296875,"Accuracy":20.0,"VerticalAccuracy":1.7990270853042603,"Speed":null,"Course":null,"IsFromMockProvider":false,"AltitudeReferenceSystem":2},{"Timestamp":"2022-10-27T18:43:53.696+00:00","Latitude":41.5263819,"Longitude":-81.6507921,"Altitude":153.29998779296875,"Accuracy":20.0,"VerticalAccuracy":1.7697961330413818,"Speed":null,"Course":null,"IsFromMockProvider":false,"AltitudeReferenceSystem":2},{"Timestamp":"2022-10-27T18:43:54.526+00:00","Latitude":41.5263819,"Longitude":-81.6507921,"Altitude":153.29998779296875,"Accuracy":20.0,"VerticalAccuracy":1.7697961330413818,"Speed":null,"Course":null,"IsFromMockProvider":false,"AltitudeReferenceSystem":2},{"Timestamp":"2022-10-27T18:43:55.374+00:00","Latitude":41.5263819,"Longitude":-81.6507921,"Altitude":153.29998779296875,"Accuracy":20.0,"VerticalAccuracy":1.7697961330413818,"Speed":null,"Course":null,"IsFromMockProvider":false,"AltitudeReferenceSystem":2},{"Timestamp":"2022-10-27T18:43:56.21+00:00","Latitude":41.5263811,"Longitude":-81.650792,"Altitude":153.29998779296875,"Accuracy":20.0,"VerticalAccuracy":1.7160584926605225,"Speed":null,"Course":null,"IsFromMockProvider":false,"AltitudeReferenceSystem":2}]
Every subsequent time I hit start/stop on the app, I get the real-time data in the output multiplied by the number of times I've started/stopped since the last compiling.
the main app page button event thats fired:
private async void GPSToggle_Clicked(object sender, EventArgs e)
{
var LocationPermissionStatus = await Xamarin.Essentials.Permissions.RequestAsync<Xamarin.Essentials.Permissions.LocationAlways>();
var FileReadPermissionStatus = await Xamarin.Essentials.Permissions.RequestAsync<Xamarin.Essentials.Permissions.StorageRead>();
var FileWritePermissionStatus = await Xamarin.Essentials.Permissions.RequestAsync<Xamarin.Essentials.Permissions.StorageWrite>();
if(LocationPermissionStatus == Xamarin.Essentials.PermissionStatus.Denied)
{
// TODO
return;
}
// run if device is android
if(Device.RuntimePlatform == Device.Android)
{
if (!CrossGeolocator.Current.IsGeolocationAvailable || !CrossGeolocator.Current.IsGeolocationEnabled)
{
// gps is not enabled, throw alert
Console.WriteLine("---------------------------------------------------------------GPS is DISABLED");
await DisplayAlert("Error", "GPS is not enabled. You must enable GPS to use this feature", "Ok");
}
else
{
// set our IsTracking = true flag
if (!IsTracking)
{
// start background listening for GPS
await StartListening();
Console.WriteLine("---------------------------------------------------------------Listening: " + CrossGeolocator.Current.IsListening);
StartService();
Console.WriteLine("---------------------------------------------------------------Service initiated");
IsTracking = true;
Console.WriteLine("---------------------------------------------------------------Tracking initiated");
GPSToggle.Text = "Stop Tracking";
}
else
{
//
// verify that the submittal wasn't done in error, before stopping services and submitting data
bool DoneInError = await DisplayAlert("Finish?", "Are you sure you want to stop services and submit?", "No", "Yes");
if (!DoneInError)
{
await StopListening();
Console.WriteLine("---------------------------------------------------------------listening:" + CrossGeolocator.Current.IsListening);
IsTracking = false;
Console.WriteLine("---------------------------------------------------------------Tracking ceased");
// stop the gps service
StopService();
Console.WriteLine("---------------------------------------------------------------Service ceased");
// stop the background listening for gps
Console.WriteLine("---------------------------------------------------------------Attempt GPX parse from buffer obj");
GPSToggle.Text = "Start Tracking";
}
}
}
}
}
Specifically the line:
StartService();
Fires this method off within the same class, specifically the MessagingCenter.Send<>, which initiates my foreground service to handle logging the gps data into a buffer:
private void StartService()
{
var startServiceMessage = new StartServiceMessage();
MessagingCenter.Send(startServiceMessage, "ServiceStarted");
Preferences.Set("LocationServiceRunning", true);
StatusLabel.Text = "Location service has been started";
Console.WriteLine("---------------------------------------------------------------location service has been started. preferences saved");
}
and
StopService();
Fires this method off to stop the services and retrieve the gps buffer data from the foreground to the main thread:
private void StopService()
{
var stopServiceMessage = new StopServiceMessage();
MessagingCenter.Unsubscribe<App, List<Location>>(this, "GPXBufferData");
MessagingCenter.Subscribe<App, List<Location>>(this, "GPXBufferData", (sender, args) =>
{
RawGPXData = args;
Generate_CreateGPX_File(RawGPXData);
RawBufferToJsonString = GPXParse.GenerateJSON_GPXPoints(RawGPXData);
Console.WriteLine("---------------------------------------------------------------:RawBufferToJsonString: " + RawBufferToJsonString);
PromptForSubmission_GPXPoints_API();
});
Console.WriteLine("--------------------------------------------------------------------------");
MessagingCenter.Send(stopServiceMessage, "ServiceStopped");
Preferences.Set("LocationServiceRunning", false);
Console.WriteLine("---------------------------------------------------------------Location service stopped. preferences saved");
}
In the above snippet, this line is subscribed to in the GPSLocationService.cs file:
MessagingCenter.Send(stopServiceMessage, "ServiceStopped");
This is a portion of my GPSLocationService.cs file that is relevant to this:
public async Task Run(CancellationToken token)
{
int ObjCount = 0;
await Task.Run(async () => {
// if the task was stopped
// check the buffer for data, if data, send to GPXGenerator
MessagingCenter.Subscribe<StopServiceMessage>(this, "ServiceStopped",
message =>
{
if (GPSBufferObj != null)
{
Device.BeginInvokeOnMainThread(() =>
{
MessagingCenter.Unsubscribe<App, List<Location>>((App)Xamarin.Forms.Application.Current, "GPXBufferData");
MessagingCenter.Send<App, List<Location>>((App)Xamarin.Forms.Application.Current, "GPXBufferData", GPSBufferObj);
});
}
});
return;
}, token);
}
I believe I have tracked down where the issue is starting. In my StopService() method, I have the following line (just to keep track of where Im at in the buffer) and it is only sent to output once.
Console.WriteLine("--------------------------------------------------------------------------");
BUT if I place that same line within the pasted portion of my GPSLocationService.cs file, I will get the incremented output. I'm leaning towards the nested task being the issue, I wrote this based losely off of this example repro:
https://github.com/jfversluis/XFBackgroundLocationSample
You don't have MessagingCenter.Unsubscribe<StopServiceMessage> anywhere in your code. StopServiceMessage is what you are accumulating subscriptions to.
You need to make sure Unsubscribe is unsubscribing the instance that was previously subscribed to. It sounds to me like there are multiple instances of GPSLocationService. [In which case, this is no longer referring to the original instance. Unsubscribe won't do anything, unless you have the this that was originally Subscribed.]
If so, instead create an instance of GPSLocationService ONCE, and store it in a static variable. Re-use it. start/stop it, but don't discard it.
Alternatively, if you only want a message ONE TIME from each Subscribe, then Unsubscribe as soon as you receive each message:
MessagingCenter.Subscribe<StopServiceMessage>(this, "ServiceStopped",
message =>
{
MessagingCenter.Unsubscribe<StopServiceMessage>(this, "ServiceStopped");
... your code ...
});
Use this same pattern EVERYWHERE you have a Subscribe (unless you Subscribe ONLY ONE TIME at app start, as Jason suggested.)
I have been working on a webscraping project.
I am having two issues, one being presenting the number of urls processed as percentage but a far larger issue is that I can not figure out how I know when all the threads i am creating are totaly finished.
NOTE: I am aware of that the a parallel foreach once done moves on BUT this is within a recursive method.
My code below:
public async Task Scrape(string url)
{
var page = string.Empty;
try
{
page = await _service.Get(url);
if (page != string.Empty)
{
if (regex.IsMatch(page))
{
Parallel.For(0, regex.Matches(page).Count,
index =>
{
try
{
if (regex.Matches(page)[index].Groups[1].Value.StartsWith("/"))
{
var match = regex.Matches(page)[index].Groups[1].Value.ToLower();
if (!links.Contains(BaseUrl + match) && !Visitedlinks.Contains(BaseUrl + match))
{
Uri ValidUri = WebPageValidator.GetUrl(match);
if (ValidUri != null && HostUrls.Contains(ValidUri.Host))
links.Enqueue(match.Replace(".html", ""));
else
links.Enqueue(BaseUrl + match.Replace(".html", ""));
}
}
}
catch (Exception e)
{
log.Error("Error occured: " + e.Message);
Console.WriteLine("Error occured, check log for further details."); ;
}
});
WebPageInternalHandler.SavePage(page, url);
var context = CustomSynchronizationContext.GetSynchronizationContext();
Parallel.ForEach(links, new ParallelOptions { MaxDegreeOfParallelism = 25 },
webpage =>
{
try
{
if (WebPageValidator.ValidUrl(webpage))
{
string linkToProcess = webpage;
if (links.TryDequeue(out linkToProcess) && !Visitedlinks.Contains(linkToProcess))
{
ShowPercentProgress();
Thread.Sleep(15);
Visitedlinks.Enqueue(linkToProcess);
Task d = Scrape(linkToProcess);
Console.Clear();
}
}
}
catch (Exception e)
{
log.Error("Error occured: " + e.Message);
Console.WriteLine("Error occured, check log for further details.");
}
});
Console.WriteLine("parallel finished");
}
}
catch (Exception e)
{
log.Error("Error occured: " + e.Message);
Console.WriteLine("Error occured, check log for further details.");
}
}
NOTE that Scrape gets called multiple times(recursive)
call the method like this:
public Task ExecuteScrape()
{
var context = CustomSynchronizationContext.GetSynchronizationContext();
Scrape(BaseUrl).ContinueWith(x => {
Visitedlinks.Enqueue(BaseUrl);
}, context).Wait();
return null;
}
which in turn gets called like so:
static void Main(string[] args)
{
RunScrapper();
Console.ReadLine();
}
public static void RunScrapper()
{
try
{
_scrapper.ExecuteScrape();
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
my result:
How do I solve this?
(Is it ethical for me to answer a question about web page scraping?)
Don't call Scrape recursively. Place the list of urls you want to scrape in a ConcurrentQueue and begin processing that queue. As the process of scraping a page returns more urls, just add them into the same queue.
I wouldn't use just a string, either. I recommend creating a class like
public class UrlToScrape //because naming things is hard
{
public string Url { get; set; }
public int Depth { get; set; }
}
Regardless of how you execute this it's recursive, so you have to somehow keep track of how many levels deep you are. A website could deliberately generate URLs that send you into infinite recursion. (If they did this then they don't want you scraping their site. Does anybody want people scraping their site?)
When your queue is empty that doesn't mean you're done. The queue could be empty, but the process of scraping the last url dequeued could still add more items back into that queue, so you need a way to account for that.
You could use a thread safe counter (int using Interlocked.Increment/Decrement) that you increment when you start processing a url and decrement when you finish. You're done when the queue is empty and the count of in-process urls is zero.
This is a very rough model to illustrate the concept, not what I'd call a refined solution. For example, you still need to account for exception handling, and I have no idea where the results go, etc.
public class UrlScraper
{
private readonly ConcurrentQueue<UrlToScrape> _queue = new ConcurrentQueue<UrlToScrape>();
private int _inProcessUrlCounter;
private readonly List<string> _processedUrls = new List<string>();
public UrlScraper(IEnumerable<string> urls)
{
foreach (var url in urls)
{
_queue.Enqueue(new UrlToScrape {Url = url, Depth = 1});
}
}
public void ScrapeUrls()
{
while (_queue.TryDequeue(out var dequeuedUrl) || _inProcessUrlCounter > 0)
{
if (dequeuedUrl != null)
{
// Make sure you don't go more levels deep than you want to.
if (dequeuedUrl.Depth > 5) continue;
if (_processedUrls.Contains(dequeuedUrl.Url)) continue;
_processedUrls.Add(dequeuedUrl.Url);
Interlocked.Increment(ref _inProcessUrlCounter);
var url = dequeuedUrl;
Task.Run(() => ProcessUrl(url));
}
}
}
private void ProcessUrl(UrlToScrape url)
{
try
{
// As the process discovers more urls to scrape,
// pretend that this is one of those new urls.
var someNewUrl = "http://discovered";
_queue.Enqueue(new UrlToScrape { Url = someNewUrl, Depth = url.Depth + 1 });
}
catch (Exception ex)
{
// whatever you want to do with this
}
finally
{
Interlocked.Decrement(ref _inProcessUrlCounter);
}
}
}
If I was doing this for real the ProcessUrl method would be its own class, and it would take HTML, not a URL. In this form it's difficult to unit test. If it were in a separate class then you could pass in HTML, verify that it outputs results somewhere, and that it calls a method to enqueue new URLs it finds.
It's also not a bad idea to maintain the queue as a database table instead. Otherwise if you're processing a bunch of urls and you have to stop, you'd have start all over again.
Can't you add all tasks Task d to some type of concurrent collection you thread through all recursive calls (via method argument) and then simply call Task.WhenAll(tasks).Wait()?
You'd need an intermediate method (makes it cleaner) that launches the base Scrape call and passes in the empty task collection. When the base call returns you have in hand all tasks and you simply wait them out.
public async Task Scrape (
string url) {
var tasks = new ConcurrentQueue<Task>();
//call your implementation but
//change it so that you add
//all launched tasks d to tasks
Scrape(url, tasks);
//1st option: Wait().
//This will block caller
//until all tasks finish
Task.WhenAll(tasks).Wait();
//or 2nd option: await
//this won't block and will return to caller.
//Once all tasks are finished method
//will resume in WriteLine
await Task.WhenAll(tasks);
Console.WriteLine("Finished!"); }
Simple rule: if you want to know when something finishes, the first step is to keep track of it. In your current implementation you are essentially firing and forgetting all launched tasks...
I am bulding a web-scraping project.
I have two lists:
private ConcurrentQueue<string> links = new ConcurrentQueue<string>();
private ConcurrentQueue<string> Visitedlinks = new ConcurrentQueue<string>();
On for all the links that I find on a page and one which will hold all the links I have scrapped.
Method that handels the business:
public async Task GetUrlContent(string url)
{
var page = string.Empty;
try
{
page = await service.Get(url);
if (page != string.Empty)
{
Regex regex = new Regex(#"<a[^>]*?href\s*=\s*[""']?([^'"" >]+?)[ '""][^>]*?>",
RegexOptions.Singleline | RegexOptions.CultureInvariant);
if (regex.IsMatch(page))
{
Console.WriteLine("Downloading url: " + url);
for (int i = 0; i < regex.Matches(page).Count; i++)
{
if (regex.Matches(page)[i].Groups[1].Value.StartsWith("/"))
{
if (!links.Contains(BaseUrl + regex.Matches(page)[i].Groups[1].Value.ToLower().Replace(".html", "")) &&
!Visitedlinks.Contains(BaseUrl + regex.Matches(page)[i].Groups[1].Value.ToLower()))
{
Uri ValidUri = GetUrl(regex.Matches(page)[i].Groups[1].Value);
if (ValidUri != null && HostUrls.Contains(ValidUri.Host))
links.Enqueue(regex.Matches(page)[i].Groups[1].Value.ToLower().Replace(".html", ""));
else
links.Enqueue(BaseUrl + regex.Matches(page)[i].Groups[1].Value.ToLower().Replace(".html", ""));
}
}
}
}
var results = links.Where(m => !Visitedlinks.Contains(m)); // problkem here, get multiple values
if (!results.Any())
{
// do nothing
}
else
{
Parallel.ForEach(results, new ParallelOptions { MaxDegreeOfParallelism = 4 },
webpage =>
{
if (ValidUrl(webpage))
{
if (!Visitedlinks.Contains(webpage))
{
Visitedlinks.Enqueue(webpage);
GetUrlContent(webpage).Wait();
}
}
});
}
}
}
catch (Exception e)
{
throw;
}
}
Problem is here:
var results = links.Where(m => !Visitedlinks.Contains(m));
The first iteration I might get:
Link1, link2, link3, link4,
Second iteration:
Link2 link3 link4, link5, link6 ,link 7
Third:
Link 3, link4, link 5, link 6, etc
This means that I will get the same links multiple times since this is a parallel foreach which does several operations at once. I can't figure out how to make sure that I dont get multiple values.
Anyone that can lend a helping hand?
If I understand correctly, the first queue contains the links you want to scrape, and the second queue contains the ones you have scraped.
The problem is that you're trying to iterate over the contents of your ConcurrentQueue:
var results = links.Where(m => !Visitedlinks.Contains(m));
This won't work predictably if you're accessing these queues from multiple threads.
What you should do is take items out of the queue and process them. What stands out is that TryDequeue doesn't appear anywhere in your code. Items are going into the queue but never coming out. The whole purpose of a queue is that we put things in and take them out. ConcurrentQueue makes it safe for multiple threads to put items in and take them out without stepping all over each other.
If you dequeue a link that you want to process:
string linkToProcess = null;
if(links.TryDequeue(out linkToProcess)) // if this returns false, the queue was empty
{
// process it
}
Then as soon as you've taken an item out of the queue to process it, it won't be in the queue anymore. Other threads don't have to check to see if an item has been processed. They just take the next item out of the queue, if there is one. Two threads won't ever take the same item out of the queue. Only one thread can take a given item out of the queue, because as soon as it does, the item isn't in the queue anymore.
Thanks to #Scott Hannen
The final solution is as follows:
Parallel.ForEach(links, new ParallelOptions { MaxDegreeOfParallelism = 25 },
webpage =>
{
try
{
if (WebPageValidator.ValidUrl(webpage))
{
string linkToProcess = webpage;
if (links.TryDequeue(out linkToProcess) && !Visitedlinks.Contains(linkToProcess))
{
Task obj = Scrape(linkToProcess);
Visitedlinks.Enqueue(linkToProcess);
}
}
}
catch (Exception e)
{
log.Error("Error occured: " + e.Message);
Console.WriteLine("Error occured, check log for further details.");
}
I came across a back pressure issue with RX.net I can't find a solution for. I have an observable real-time stream of log messages.
var logObservable = /* Observable stream of log messages */
Which I want to expose via a TCP interface which serializes the real-time log messages from the logObservable before they are sent over the wire. So I do the following:
foreach (var message in logObservable.ToEnumerable())
{
// 1. Serialize message
// 2. Send it over the wire.
}
The problem arises with the .ToEnumerable() if a back pressure scenario happens e.g. if the client on the other end pauses the stream. The problem is that .ToEnumerable() caches the items which result in a lot of memory usage. I'm looking for a mechanism something like a DropQueue which only buffers, let say, the last 10 messages e.g.
var observableStream = logObservable.DropQueue(10).ToEnumerable();
Is this the right way to way to solve this issue? And do you know to implement such a mechanism to avoid possible back pressure issue?
My DropQueue implementation:
public static IEnumerable<TSource> ToDropQueue<TSource>(
this IObservable<TSource> source,
int queueSize,
Action backPressureNotification = null,
CancellationToken token = default(CancellationToken))
{
var queue = new BlockingCollection<TSource>(new ConcurrentQueue<TSource>(), queueSize);
var isBackPressureNotified = false;
var subscription = source.Subscribe(
item =>
{
var isBackPressure = queue.Count == queue.BoundedCapacity;
if (isBackPressure)
{
queue.Take(); // Dequeue an item to make space for the next one
// Fire back-pressure notification if defined
if (!isBackPressureNotified && backPressureNotification != null)
{
backPressureNotification();
isBackPressureNotified = true;
}
}
else
{
isBackPressureNotified = false;
}
queue.Add(item);
},
exception => queue.CompleteAdding(),
() => queue.CompleteAdding());
token.Register(() => { subscription.Dispose(); });
using (new CompositeDisposable(subscription, queue))
{
foreach (var item in queue.GetConsumingEnumerable())
{
yield return item;
}
}
}
I have a regular Queue object in C# (4.0) and I'm using BackgroundWorkers that access this Queue.
The code I was using is as follows:
do
{
while (dataQueue.Peek() == null // nothing waiting yet
&& isBeingLoaded == true // and worker 1 still actively adding stuff
)
System.Threading.Thread.Sleep(100);
// otherwise ready to do something:
if (dataQueue.Peek() != null) // because maybe the queue is complete and also empty
{
string companyId = dataQueue.Dequeue();
processLists(companyId);
// use up the stuff here //
} // otherwise nothing was there yet, it will resolve on the next loop.
} while (isBeingLoaded == true // still have stuff coming at us
|| dataQueue.Peek() != null); // still have stuff we haven’t done
However, I guess when dealing with threads I should be using a ConcurrentQueue.
I was wondering if there were examples of how to use a ConcurrentQueue in a Do While Loop like above?
Everything I tried with the TryPeek wasn't working..
Any ideas?
You can use a BlockingCollection<T> as a producer-consumer queue.
My answer makes some assumptions about your architecture, but you can probably mold it as you see fit:
public void Producer(BlockingCollection<string> ids)
{
// assuming this.CompanyRepository exists
foreach (var id in this.CompanyRepository.GetIds())
{
ids.Add(id);
}
ids.CompleteAdding(); // nothing left for our workers
}
public void Consumer(BlockingCollection<string> ids)
{
while (true)
{
string id = null;
try
{
id = ids.Take();
} catch (InvalidOperationException) {
}
if (id == null) break;
processLists(id);
}
}
You could spin up as many consumers as you need:
var companyIds = new BlockingCollection<string>();
Producer(companyIds);
Action process = () => Consumer(companyIds);
// 2 workers
Parallel.Invoke(process, process);