Develop Reference

Define Next Start Point When Number of Items Unknown

I have a web service I need to query and it takes a value that supports pagination for its data. Due to the amount of data I need to fetch and how that service is implemented I intended to do a series of concurrent http web requests to accumulate this data.
Say I have number of threads and page size how could I assign each thread to pick its starting point that doesn't overlap with the other thread? Its been a long time since I took parallel programming and I'm floundering a bit. I know I could find my start point with something like start = N/numThreads * threadNum however I don't know N. Right now I just spin up X threads and each loop until they get no more data. Problem is they tend to overlap and I end up with duplicate data. I need unique data and not to waste requests.
Right now I have code that looks something like this. This is one of many attempts and I see why this is wrong but its better to show something. The goal is to in parallel collect pages of data from a webservice:
int limit = pageSize;
data = new List<RequestStuff>();
List<Task> tasks = new List<Task>();
for (int i = 0; i < numThreads; i++)
{
tasks.Add(Task.Factory.StartNew(() =>
{
try
{
List<RequestStuff> someData;
do
{
int start;
lock(myLock)
{
start = data.Count;
}
someKeys = GetDataFromService(start, limit);
lock (myLock)
{
if (someData != null && someData.Count > 0)
{
data.AddRange(someData);
}
}
} while (hasData);
}
catch (AggregateException ex)
{
//Exception things
}
}));
}
Task.WaitAll(tasks.ToArray());
Any inspiration to solve this without race conditions? I need to stick to .NET 4 if that matters.

I'm not sure there's a way to do this without wasting some requests unless you know the actual limit. The code below might help eliminate the duplicate data as you will only query on each index once:
private int _index = -1; // -1 so first request starts at 0
private bool _shouldContinue = true;
public IEnumerable<RequestStuff> GetAllData()
{
var tasks = new List<Task<RequestStuff>>();
while (_shouldContinue)
{
tasks.Add(new Task<RequestStuff>(() => GetDataFromService(GetNextIndex())));
}
Task.WaitAll(tasks.ToArray());
return tasks.Select(t => t.Result).ToList();
}
private RequestStuff GetDataFromService(int id)
{
// Get the data
// If there's no data returned set _shouldContinue to false
// return the RequestStuff;
}
private int GetNextIndex()
{
return Interlocked.Increment(ref _index);
}
It could also be improved by adding cancellation tokens to cancel any indexes you know to be wasteful, i.e, if index 4 returns nothing you can cancel all queries on indexes above 4 that are still active.
Or if you could make a reasonable guess at the max index you might be able to implement an algorithm to pinpoint the exact limit before retrieving any data. This would probably only be more efficient if your guess was fairly accurate though.

Are you attempting to force parallelism on the part of the remote service by issuing multiple concurrent requests? Paging is generally used to limit the amount of data returned to only that which is needed, but if you need all of the data, then attempting to first page and then reconstruct it later seems like a poor design. Your code becomes needlessly complex, difficult to maintain, you'll likely just move the bottleneck from code you control to somewhere else, and now you've introduced data integrity issues (what happens if all of these threads access different versions of the data you are trying to query?). By increasing the complexity and number of calls, you are also increasing the likelihood of problems occurring (eg. one of the connections gets dropped).
Can you state the problem you are attempting to solve so perhaps instead we can help architect a better solution?

System.Core Pipe is broken

I currently just inherited some complex code that unfortunately I do not fully understand. It handles a large number of inventory records inputting/outputting to a database. The solution is extremely large/advanced where I am still on the newer side of c#. The issue I am encountering is that periodically the program will throw an IO Exception. It doesn't actually throw a failure code, but it messes up our output data.
the try/catch block is as follows:
private static void ReadRecords(OleDbRecordReader recordReader, long maxRows, int executionTimeout, BlockingCollection<List<ProcessRecord>> processingBuffer, CancellationTokenSource cts, Job theStack, string threadName) {
ProcessRecord rec = null;
try {
Thread.CurrentThread.Name = threadName;
if(null == cts)
throw new InvalidOperationException("Passed CancellationToken was null.");
if(cts.IsCancellationRequested)
throw new InvalidOperationException("Passed CancellationToken is already been cancelled.");
long reportingFrequency = (maxRows <250000)?10000:100000;
theStack.FireStatusEvent("Opening "+ threadName);
recordReader.Open(maxRows, executionTimeout);
theStack.FireStatusEvent(threadName + " Opened");
theStack.FireInitializationComplete();
List<ProcessRecord> inRecs = new List<PIRecord>(500);
ProcessRecord priorRec = rec = recordReader.Read();
while(null != priorRec) { //-- note that this is priorRec, not Rec. We process one row in arrears.
if(cts.IsCancellationRequested)
theStack.FireStatusEvent(threadName + " cancelling due to request or error.");
cts.Token.ThrowIfCancellationRequested();
if(rec != null) //-- We only want to count the loop when there actually is a record.
theStack.RecordCountRead++;
if(theStack.RecordCountRead % reportingFrequency == 0)
theStack.FireProgressEvent();
if((rec != null) && (priorRec.SKU == rec.SKU) && (priorRec.Store == rec.Store) && (priorRec.BatchId == rec.BatchId))
inRecs.Add(rec); //-- just store it and keep going
else { //-- otherwise, we need to process it
processingBuffer.Add(inRecs.ToList(),cts.Token); //-- note that we don't enqueue the original LIST! That could be very bad.
inRecs.Clear();
if(rec != null) //-- Again, we need this check here to ensure that we don't try to enqueue the EOF null record.
inRecs.Add(rec); //-- Now, enqueue the record that fired this condition and start the loop again
}
priorRec = rec;
rec = recordReader.Read();
} //-- end While
}
catch(OperationCanceledException) {
theStack.FireStatusEvent(threadName +" Canceled.");
}
catch(Exception ex) {
theStack.FireExceptionEvent(ex);
theStack.FireStatusEvent("Error in RecordReader. Requesting cancellation of other threads.");
cts.Cancel(); // If an exception occurs, notify all other pipeline stages, then rethrow
// throw; //-- This will also propagate Cancellation, but that's OK
}
In the log of our job we see the output loader stopping and the exception is
System.Core: Pipe is broken.
Does any one have any ideas as to what may cause this? More importantly, the individual who made this large-scale application is no longer here. When I debug all of my applications, I am able to add break points in the solution and do the standard VS stepping through everything to find the issue. However, this application is huge and has a GUI that pops up when I debug the application. I believe the GUI was made for testing purposes, but it hinders me from actually being able to step through everything. However when the .exe is run from our actual job stream, there is no GUI it just executes the way it's supposed to.
The help I am asking for is 2 things:
just suggestions as to what may cause this. Could an OleDB driver be the cause? Reason I ask is because I have this running on 2 different servers. One test and one not. The one with a new OleDB driver version does not fail (7.0 i believe whereas the other where it fails is 6.0).
Is there any code that I could add that may give me a better indication as to what may be causing the broken pipe? The error only happens periodically. If I run the job again right after, it may not happen. I'd say it's 30-40% of the time it throws the exception.
If you have any additional questions about the structure just let me know.

Nice and clean way to retry something several times

I constantly have some situations in which I have to retry some operations if they fail, giving up after certain number of times, and taking a short breaks between attempts.
Is there a way to create a 'retry method' that will enable me not to copy the code around every time I do that?

Got tired of copy/paste the same code over and over again, so I created a method that accepts the delegate of the task that has to be done. Here it is:
// logger declaration (I use NLog)
private static readonly Logger Log = LogManager.GetCurrentClassLogger();
delegate void WhatTodo();
static void TrySeveralTimes(WhatTodo Task, int Retries, int RetryDelay)
{
int retries = 0;
while (true)
{
try
{
Task();
break;
}
catch (Exception ex)
{
retries++;
Log.Info<string, int>("Problem doing it {0}, try {1}", ex.Message, retries);
if (retries > Retries)
{
Log.Info("Giving up...");
throw;
}
Thread.Sleep(RetryDelay);
}
}
}
To use it, I would simply write:
TrySeveralTimes(() =>
{
string destinationVpr = Path.Combine(outdir, "durations.vpr");
File.AppendAllText(destinationVpr, file + ", " + lengthInMiliseconds.ToString() + "\r\n");
}, 10, 100);
In this example I am appending a file that gets locked with some external process, only way to write it is to retry it several times until the process is done with it...
I would sure love to see better ways of handling this particular pattern (retrying).
EDIT: I looked at Gallio in another answer, and it is really great. Look at this example:
Retry.Repeat(10) // Retries maximum 10 times the evaluation of the condition.
.WithPolling(TimeSpan.FromSeconds(1)) // Waits approximatively for 1 second between each evaluation of the condition.
.WithTimeout(TimeSpan.FromSeconds(30)) // Sets a timeout of 30 seconds.
.DoBetween(() => { /* DoSomethingBetweenEachCall */ })
.Until(() => { return EvaluateSomeCondition(); });
It does everything. It will even watch your kids while you code :) But, I strive for simplicity, and still am using .NET 2.0. So I guess that my example will still be of some use to you.

I've created such helpers based on specific domain requirements, but as a generic starting point have a look at Gallio's implementation.
http://www.gallio.org/api/html/T_MbUnit_Framework_Retry.htm
https://code.google.com/p/mb-unit/source/browse/trunk/v3/src/MbUnit/MbUnit/Framework/Retry.cs
http://interfacingreality.blogspot.co.uk/2009/05/retryuntil-in-mbunit-v3.html

Why does parallel this code work sometimes?

I wanted to parallelize a piece of code, but the code actually got slower probably because of overhead of Barrier and BlockCollection. There would be 2 threads, where the first would find pieces of work wich the second one would operate on. Both operations are not much work so the overhead of switching safely would quickly outweigh the two threads.
So I thought I would try to write some code myself to be as lean as possible, without using Barrier etc. It does not behave consistent however. Sometimes it works, sometimes it does not and I can't figure out why.
This code is just the mechanism I use to try to synchronize the two threads. It doesn't do anything useful, just the minimum amount of code you need to reproduce the bug.
So here's the code:
// node in linkedlist of work elements
class WorkItem {
public int Value;
public WorkItem Next;
}
static void Test() {
WorkItem fst = null; // first element
Action create = () => {
WorkItem cur=null;
for (int i = 0; i < 1000; i++) {
WorkItem tmp = new WorkItem { Value = i }; // create new comm class
if (fst == null) fst = tmp; // if it's the first add it there
else cur.Next = tmp; // else add to back of list
cur = tmp; // this is the current one
}
cur.Next = new WorkItem { Value = -1 }; // -1 means stop element
#if VERBOSE
Console.WriteLine("Create is done");
#endif
};
Action consume = () => {
//Thread.Sleep(1); // this also seems to cure it
#if VERBOSE
Console.WriteLine("Consume starts"); // especially this one seems to matter
#endif
WorkItem cur = null;
int tot = 0;
while (fst == null) { } // busy wait for first one
cur = fst;
#if VERBOSE
Console.WriteLine("Consume found first");
#endif
while (true) {
if (cur.Value == -1) break; // if stop element break;
tot += cur.Value;
while (cur.Next == null) { } // busy wait for next to be set
cur = cur.Next; // move to next
}
Console.WriteLine(tot);
};
try { Parallel.Invoke(create, consume); }
catch (AggregateException e) {
Console.WriteLine(e.Message);
foreach (var ie in e.InnerExceptions) Console.WriteLine(ie.Message);
}
Console.WriteLine("Consume done..");
Console.ReadKey();
}
The idea is to have a Linkedlist of workitems. One thread adds items to the back of that list, and another thread reads them, does something, and polls the Next field to see if it is set. As soon as it is set it will move to the new one and process it. It polls the Next field in a tight busy loop because it should be set very quickly. Going to sleep, context switching etc would kill the benefit of parallizing the code.
The time it takes to create a workitem would be quite comparable to executing it, so the cycles wasted should be quite small.
When I run the code in release mode, sometimes it works, sometimes it does nothing. The problem seems to be in the 'Consumer' thread, the 'Create' thread always seems to finish. (You can check by fiddling with the Console.WriteLines).
It has always worked in debug mode. In release it about 50% hit and miss. Adding a few Console.Writelines helps the succes ratio, but even then it's not 100%. (the #define VERBOSE stuff).
When I add the Thread.Sleep(1) in the 'Consumer' thread it also seems to fix it. But not being able to reproduce a bug is not the same thing as knowing for sure it's fixed.
Does anyone here have a clue as to what goes wrong here? Is it some optimization that creates a local copy or something that does not get updated? Something like that?
There's no such thing as a partial update right? like a datarace, but then that one thread is half doen writing and the other thread reads the partially written memory? Just checking..
Looking at it I think it should just work.. I guess once every few times the threads arrive in different order and that makes it fail, but I don't get how. And how I could fix this without adding slowing it down?
Thanks in advance for any tips,
Gert-Jan

I do my damn best to avoid the utter minefield of closure/stack interaction at all costs.
This is PROBABLY a (language-level) race condition, but without reflecting Parallel.Invoke i can't be sure. Basically, sometimes fst is being changed by create() and sometimes not. Ideally, it should NEVER be changed (if c# had good closure behaviour). It could be due to which thread Parallel.Invoke chooses to run create() and consume() on. If create() runs on the main thread, it might change fst before consume() takes a copy of it. Or create() might be running on a separate thread and taking a copy of fst. Basically, as much as i love c#, it is an utter pain in this regard, so just work around it and treat all variables involved in a closure as immutable.
To get it working:
//Replace
WorkItem fst = null
//with
WorkItem fst = WorkItem.GetSpecialBlankFirstItem();
//And
if (fst == null) fst = tmp;
//with
if (fst.Next == null) fst.Next = tmp;

A thread is allowed by the spec to cache a value indefinitely.
see Can a C# thread really cache a value and ignore changes to that value on other threads? and also http://www.yoda.arachsys.com/csharp/threads/volatility.shtml

Example of "using exceptions to control flow" [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
What would a piece of code which "uses exceptions to control flow" look like? I've tried to find a direct C# example, but cannot. Why is it bad?
Thanks

By definition, an exception is an occurrence which happens outside the normal flow of your software. A quick example off the top of my head is using a FileNotFoundException to see if a file exists or not.
try
{
File.Open(#"c:\some nonexistent file.not here");
}
catch(FileNotFoundException)
{
// do whatever logic is needed to create the file.
...
}
// proceed with the rest of your program.
In this case, you haven't used the File.Exists() method which achieves the same result but without the overhead of the exception.
Aside from the bad usage, there is overhead associated with an exception, populating the properties, creating the stack trace, etc.

It's roughly equivalent to a goto, except worse in terms of the word Exception, and with more overhead. You're telling the code to jump to the catch block:
bool worked;
try
{
foreach (Item someItem in SomeItems)
{
if (someItem.SomeTestFailed()) throw new TestFailedException();
}
worked = true;
}
catch(TestFailedException testFailedEx)
{
worked = false;
}
if (worked) // ... logic continues
As you can see, it's running some (made-up) tests; if they fail, an exception is thrown, and worked will be set to false.
Much easier to just update the bool worked directly, of course!
Hope that helps!

Bad
The below code catches an exception that could easily be avoided altogether. This makes the code more difficult to follow and typically incurs a performance cost as well.
int input1 = GetInput1();
int input2 = GetInput2();
try
{
int result = input1 / input2;
Output("{0} / {1} = {2}", input1, input2, result);
}
catch (OverflowException)
{
Output("There was an overflow exception. Make sure input2 is not zero.");
}
Better
This code checks for a condition that would throw an exception, and corrects the situation before the error occurs. This way there is no exception at all. The code is more readable, and the performance is very likely to be better.
int input1 = GetInput1();
int input2 = GetInput2();
while (input2 == 0)
{
Output("input2 must not be zero. Enter a new value.");
input2 = GetInput2();
}
int result = input1 / input2;
Output("{0} / {1} = {2}", input1, input2, result);

Here's a common one:
public bool TryParseEnum<T>(string value, out T result)
{
result = default(T);
try
{
result = (T)Enum.Parse(typeof(T), value, true);
return true;
}
catch
{
return false;
}
}

Probably the grossest violation I've ever seen:
// I haz an array...
public int ArrayCount(object[] array)
{
int count = 0;
try
{
while (true)
{
var temp = array[count];
count++;
}
}
catch (IndexOutOfRangeException)
{
return count;
}
}

I'm currently working with a 3rd party program that does this. They have a "cursor" interface (basically an IEnumerable alternative), where the only way to tell the program you're finished is to raise an exception. The code basically looks like:
// Just showing the relevant section
bool finished = false;
public bool IsFinished()
{
return finished;
}
// Using something like:
// int index = 0;
// int count = 42;
public void NextRecord()
{
if (finished)
return;
if (index >= count)
throw new APIProgramSpecificException("End of cursor", WEIRD_CONSTANT);
else
++index;
}
// Other methods to retrieve the current value
Needless to say, I hate the API - but its a good example of exceptions for flow control (and an insane way of working).

I'm not fond of C# but you can see some similarities between try-catch-finally statements and normal control flow statements if-then-else.
Think about that whenever you throw an exception you force your control to be passed to the catch clause. So if you have
if (doSomething() == BAD)
{
//recover or whatever
}
You can easily think of it in terms of try-catch:
try
{
doSomething();
}
catch (Exception e)
{
//recover or do whatever
}
The powerful thing about exception is that you don't have to be in the same body to alter the flow of the program, you can throw an exception whenever you want with the guarantee that control flow will suddently diverge and reach the catch clause. This is powerful but dangerous at the same time since you could have done actions that need some backup at the end, that's why the finally statement exists.
In addition you can model also a while statement without effectively using the condition of it:
while (!finished)
{
//do whatever
}
can become
try
{
while (true)
{
doSomethingThatEventuallyWillThrowAnException();
}
}
catch (Exception e)
{
//loop finished
}

A module developed by a partner caused our application to take a very long time to load. On closer examination, the module was looking for a config file at app startup. This by itself was not too objectionable, but the way in which it was doing it was outrageously bad:
For every file in the app directory, it opened the file and tried to parse it as XML. If a file threw an exception (because it wasn't XML), it caught the exception, squelched it, and tried the next file!
When the partner tested this module, they only had 3 files in the app directory. The bonehead config file search didn't have a noticeable effect on the test app startup. When we added it to our application, there were 100's of files in the app directory, and the app froze for nearly a minute at startup.
To add salt to the wound, the name of the config file the module was searching for was predetermined and constant. There was no need for a file search of any kind.
Genius has its limits. Stupidity is unbounded.

One example would be using exceptions to return a result from a recursive method:
public void Search(Node node, object data)
{
if(node.Data.Equals(data))
{
throw new ResultException(node);
}
else
{
Search(node.LeftChild, data);
Search(node.RightChild, data);
}
}
Doing something like this is a problem for several reasons.
It's completely counter-intuitive. Exceptions are designed for exceptional cases. Something working as intended should (we hope) never be an exceptional scenario.
You can't always rely on an exception being thrown and propagated to you. For example, if the exception-throwing code runs in a separate thread, you'll need some extra code to capture it.
It is a potential performance problem. There is an overhead associated with exceptions and if you throw a lot of them, you might see a performance drop in your application.
There are a few more examples and some interesting discussion on this subject here.
Disclaimer: The code above is adapted from the first sample on that wiki page to turn it into C#.