Develop Reference

Does the need to make the code simpler justify the use of wrong abstractions?

Suppose we have a CommandRunner class that runs Commands, when a Command is created it's kept in the processingQueue for proccessing, if the execution of the Command finishes with errors the Command is moved to the faultedQueue for later processing but when everything is OK the Command is moved to the archiveQueue, the archiveQueue is not going to be processed in any way
the CommandRunner is something like this
class CommandRunner
{
public CommandRunner(IQueue<Command> processingQueue,
IQueue<Command> faultedQueue,
IQueue<Command> archiveQueue)
{
this.processingQueue = processingQueue;
this.faultedQueue= faultedQueue;
this.archiveQueue= archiveQueue;
}
public void RunCommands()
{
while(processingQueue.HasItems)
{
var current = processingQueue.Dequeue();
var result = current.Run();
if(result.HasError)
curent.MoveTo(faultedQueue);
else
curent.MoveTo(archiveQueue);
...
}
}
}
The CommandeRunner recives the three dependecies as a PersistentQueue the PersistentQueue is responsible for the long term storage of the Commands and so we free the CommandRunner from handling this
And the only purpose of the archiveQueue is to keep the design homogenous, to keep the CommandRunner persistence ignorant and with few dependencies
for example we can imagine a Property like this
IEnumerable<Command> AllCommands
{
get
{
return Enumerate(archiveQueue).Union(processingQueue).Union(faultedQueue);
}
}
many portions of the class need to do so(handle the Archive as a Queue to make the code simpler as shown above)
Does it make sense to use a Queue even if it's not the best abstraction, or do I have to use another abstraction for the archive concept.
what are other alternatives to meet these requirement?

Keep in mind that code, especially running code usually gets tangled and messy as time pass. To combat this, good names, good design, and meaningful comments come into play.
If you don't going to process the archiveQueue, and it's just a storage for messages that has been successfully processed, you can always store it as a different type (list, collection, set, whatever suits your needs), and then choose one of the following two:
Keep the name archiveQueue and change the underlying type. I would leave a comment where it's defined (or injected) saying : Notice that this might not be an actual queue. Name is for consistency reasons only.
Change the name to archiveRepository or something similar, while keeping the queue type. Obviously, since it's still a queue, you'll leave a comment saying: Notice, this is actually a queue.
Another thing to keep in mind, is that if you have n people working on your code base, you'll probably get n+1 different perferences about which way it shoud be done :)

Queue is a useful structure when you need to take care about the order of items inside it. If you need in your command post process, take care about the orders commands ran, then the queue can be a good choice.
If you don't need info about the order or commands, maybe you can use a List (on System.Collections namespace).
I think your choice are good, in the same case, I'll use a queues, we have a good example with OS design principles, inside OS (on Kernel) the process are queued for execution, clearly the OS queues are more complicated because they have other variables in mind like priority, and CPU utilization, but we can learn about the use of queues like data structures in process management.

List<> items lost during ToArray()?

A while ago, I coded a system for collecting public transport disruptions. Information about any incident is collected in an MSSQL database. Consumers access these data by calling an .asmx web service. Data are collected from the DB using ADO.NET, each data row is then populating a Deviation object and added to a List. In the service layer, the list is applied a ToArray() call and returned to the consumer.
So far, so good. But the problem is that in some cases (5% or so), we have been aware that the array somehow is curtailed. Instead of the usual number of 15-20 items, only half of them, or even fewer, are returned. The remaining items are always at the end of the original list. And, even fewer times, a couple of items are repeated/shuffled at the beginning of the array.
After doing some testing on the different layers, it seems as the curtailing occurs at the end of the process, i.e. during the casting to an array or the SOAP serialization. But the code seems so innocent, huh??:
[WebMethod]
public Deviation[] GetDeviationsByTimeInterval(DateTime from, DateTime to)
{
return DeviRoutines.GetDeviationsByTimeInterval(from, to).ToArray();
}
I am not 100% sure the error doesn't occur in the SQL or data access layer, but they have proved to do their job during the testing. Any help on the subject would be of great help! :)

I'd do something like:
public Deviation[] GetDeviationsByTimeInterval(DateTime from, DateTime to)
{
var v1 = DeviRoutines.GetDeviationsByTimeInterval(from, to);
LogMe( "v1: " + v1.Count );
var v2 = v1.ToArray();
LogMe( "v2: " + v2.Length );
return v2;
}
Proofing what you expect usually pays out :-)

http://msdn.microsoft.com/en-us/library/x303t819.aspx
Type: T[] An array containing copies
of the elements of the List.
You didn't find a bug in .NET, it's most likely something in your GetDeviationsByTimeInterval

I'd be willing to bet ToArray is doing exactly what it's told, but either your from or to values are occasionally junk (validation error?) or GetDeviationsByTimeInterval is misinterpreting them for some reason.
Stick some logging into both Deviation[] and GetDeviationsByTimeInterval to see what values get passed to it, and the next time it goes pear-shaped you'll be able to diagnose where the problem is.

Duplicate returned by Guid.NewGuid()?

We have an application that generates simulated data for one of our services for testing purposes. Each data item has a unique Guid. However, when we ran a test after some minor code changes to the simulator all of the objects generated by it had the same Guid.
There was a single data object created, then a for loop where the properties of the object were modified, including a new unique Guid, and it was sent to the service via remoting (serializable, not marshal-by-ref, if that's what you're thinking), loop and do it again, etc.
If we put a small Thread.Sleep( ...) inside of the loop, it generated unique id's. I think that is a red-herring though. I created a test app that just created one guid after another and didn't get a single duplicate.
My theory is that the IL was optimized in a way that caused this behavior. But enough about my theories. What do YOU think? I'm open to suggestions and ways to test it.
UPDATE: There seems to be a lot of confusion about my question, so let me clarify. I DON'T think that NewGuid() is broken. Clearly it works. Its FINE! There is a bug somewhere though, that causes NewGuid() to either:
1) be called only once in my loop
2) be called everytime in my loop but assigned only once
3) something else I haven't thought of
This bug can be in my code (MOST likely) or in optimization somewhere.
So to reiterate my question, how should I debug this scenario?
(and thank you for the great discussion, this is really helping me clarify the problem in my mind)
UPDATE # 2: I'd love to post an example that shows the problem, but that's part of my problem. I can't duplicate it outside of the whole suite of applications (client and servers).
Here's a relevant snippet though:
OrderTicket ticket = new OrderTicket(... );
for( int i = 0; i < _numOrders; i++ )
{
ticket.CacheId = Guid.NewGuid();
Submit( ticket ); // note that this simply makes a remoting call
}

Does Submit do an async call, or does the ticket object go into another thread at any stage.
In the code example you are reusing the same object. What if Submit sends the ticket in a background thread after a short delay (and does not take a copy). When you change the CacheId you are actually updating all the pending submits. This also explains why a Thread.Sleep fixes the problem. Try this:
for( int i = 0; i < _numOrders; i++ )
{
OrderTicket ticket = new OrderTicket(... );
ticket.CacheId = Guid.NewGuid();
Submit( ticket ); // note that this simply makes a remoting call
}
If for some reason this is not possible, try this and see if they are still the same:
ticket.CacheId = new Guid("00000000-0000-0000-0000-" +
string.Format("{0:000000000000}", i));

Thousands of developers use Guids in .NET. If Guid.NewGuid() had any tendency at all to get "stuck" on one value, the problem would have been encountered long ago.
The minor code changes are the sure culprit here. The fact that Thread.Sleep (which is less a red herring than a fish rotting in the sun) "fixes" your problem suggests that your properties are being set in some weird way that can't take effect until the loop stops blocking (either by ending or by Thread.Sleep). I'd even be willing to bet that the "minor change" was to reset all the properties from a separate thread.
If you posted some sample code, that would help.

It's a bug in your code. If you've managed to generate multiple guid's it is the most likely explanation. The clue is here in your question: "when we ran a test after some minor code changes to the simulator all of the objects generated by it had the same Guid"

See this article about how a Guid is created.
This artcile came from This answer.
Bottom line if you are creating the GUIDs too quickly and the clock hasn't moved forward that is why you are getting some as the same. However when you put a sleep in it works because the clock has moved.

The code in Submit and OrderTicket would be helpful as well...
You're reusing OrderTicket. I'd suspect that either you (or remoting itself) is batching calls out - probably in respect to # of connections/host limits - and picking up the last value of CacheId when it finally sends them along.
If you debug or Thread.Sleep the app, you're changing the timing so that the remoting call finishes before you assign a new CacheId.
Are you asyncing the remoting call? I'd think a sync call would block - but I'd check with a packet sniffer like Wireshark to be sure. Regardless, just changing to creating a new OrderTicket in each iteration would probably do the trick.
Edit: The question is not about NewGuid being broken...so my previous answer has been removed.

I dont know the details of how GUIDs are generated.. yet. However currently my org. is breeding GUIDs at a rate that would put rabbits to shame. So I can vouch for the fact that GUIDs aren't broken.. yet.
Post the source code if possible.. or a clone repro app. Many times I find the act of creating that clone app to repro the problem shows me the issue.
The other approach would be to comment out "those minor changes". If that fixes the problem, you can then triangularize to find the offending line of code. Eye-ball the minor changes hard... I mean real Hard.
Do let us know how it goes... this sounds interesting.

My gut is telling me something along these lines is going on...
class OrderTicket
{
Guid CacheId {set {_guid = new Guid("00000000-0000-0000-0000-");}
}
Log the value of CacheId into a log file every time its called with a stack trace ... Maybe someone else is setting it.

Does This ASP.NET Consultant Know What He's Doing?

The IT department of a subsidiary of ours had a consulting company write them an ASP.NET application. Now it's having intermittent problems with mixing up who the current user is and has been known to show Joe some of Bob's data by mistake.
The consultants were brought back to troubleshoot and we were invited to listen in on their explanation. Two things stuck out.
First, the consultant lead provided this pseudo-code:
void MyFunction()
{
Session["UserID"] = SomeProprietarySessionManagementLookup();
Response.Redirect("SomeOtherPage.aspx");
}
He went on to say that the assignment of the session variable is asynchronous, which seemed untrue. Granted the call into the lookup function could do something asynchronously, but this seems unwise.
Given that alleged asynchronousness, his theory was that the session variable was not being assigned before the redirect's inevitable ThreadAbort exception was raised. This faulure then prevented SomeOtherPage from displaying the correct user's data.
Second, he gave an example of a coding best practice he recommends. Rather than writing:
int MyFunction(int x, int x)
{
try
{
return x / y;
}
catch(Exception ex)
{
// log it
throw;
}
}
the technique he recommended was:
int MyFunction(int x, int y, out bool isSuccessful)
{
isSuccessful = false;
if (y == 0)
return 0;
isSuccessful = true;
return x / y;
}
This will certainly work and could be better from a performance perspective in some situations.
However, from these and other discussion points it just seemed to us that this team was not well-versed technically.
Opinions?

Rule of thumb: If you need to ask if a consultant knows what he's doing, he probably doesn't ;)
And I tend to agree here. Obviously you haven't provided much, but they don't seem terribly competent.

I would agree. These guys seem quite incompetent.
(BTW, I'd check to see if in "SomeProprietarySessionManagementLookup," they're using static data. Saw this -- with behavior exactly as you describe on a project I inherited several months ago. It was a total head-slap moment when we finally saw it ... And wished we could get face to face with the guys who wrote it ... )

If the consultant has written an application that's supposed to be able to keep track of users and only show the correct data to the correct users and it doesn't do that, then clearly something's wrong. A good consultant would find the problem and fix it. A bad consultant would tell you that it was asynchronicity.

On the asynchronous part, the only way that could be true is if the assignment going on there is actually an indexer setter on Session that is hiding an asynchronous call with no callback indicating success/failure. This would seem to be a HORRIBLE design choice, and it looks like a core class in your framework, so I find it highly unlikely.
Usually asynchronous calls have a way to specify a callback so you can determine what the result is, or if the operation was successful. The documentation for Session should be pretty clear though on if it is actually hiding an asynchronous call, but yeah... doesn't look like the consultant knows what he is talking about...
The method call that is being assigned to the Session indexer cannot be asynch, because to get a value asynchronously, you HAVE to use a callback... no way around that, so if there is no explicit callback, it's definitely not asynch (well, internally there could be an asynchronous call, but the caller of the method would perceive it as synchronous, so it is irrelevant if the method internally for example invokes a web service asynchronously).
For the second point, I think this would be much better, and keep the same functionality essentially:
int MyFunction(int x, int y)
{
if (y == 0)
{
// log it
throw new DivideByZeroException("Divide by zero attempted!");
}
return x / y;
}

For the first point, that does indeed seem bizarre.
On the second one, it's reasonable to try to avoid division by 0 - it's entirely avoidable and that avoidance is simple. However, using an out parameter to indicate success is only reasonable in certain cases, such as int.TryParse and DateTime.TryParseExact - where the caller can't easily determine whether or not their arguments are reasonable. Even then, the return value is usually the success/failure and the out parameter is the result of the method.

Asp.net sessions, if you're using the built-in providers, won't accidentally give you someone else's session. SomeProprietarySessionManagementLookup() is the likely culprit and is returning bad values or just not working.
Session["UserID"] = SomeProprietarySessionManagementLookup();
First of all assigning the return value from an asynchronously SomeProprietarySessionManagementLookup() just wont work. The consultants code probably looks like:
public void SomeProprietarySessionManagementLookup()
{
// do some async lookup
Action<object> d = delegate(object val)
{
LookupSession(); // long running thing that looks up the user.
Session["UserID"] = 1234; // Setting session manually
};
d.BeginInvoke(null,null,null);
}
The consultant isn't totally full of BS, but they have written some buggy code. Response.Redirect() does throw a ThreadAbort, and if the proprietary method is asynchronous, asp.net doesn't know to wait for the asynchronous method to write back to the session before asp.net itself saves the session. This is probably why it sometimes works and sometimes doesn't.
Their code might work if the asp.net session is in-process, but a state server or db server wouldn't. It's timing dependent.
I tested the following. We use state server in development. This code works because the session is written to before the main thread finishes.
Action<object> d = delegate(object val)
{
System.Threading.Thread.Sleep(1000); // waits a little
Session["rubbish"] = DateTime.Now;
};
d.BeginInvoke(null, null, null);
System.Threading.Thread.Sleep(5000); // waits a lot
object stuff = Session["rubbish"];
if( stuff == null ) stuff = "not there";
divStuff.InnerHtml = Convert.ToString(stuff);
This next snippet of code doesn't work because the session was already saved back to state server by the time the asynchronous method gets around to setting a session value.
Action<object> d = delegate(object val)
{
System.Threading.Thread.Sleep(5000); // waits a lot
Session["rubbish"] = DateTime.Now;
};
d.BeginInvoke(null, null, null);
// wait removed - ends immediately.
object stuff = Session["rubbish"];
if( stuff == null ) stuff = "not there";
divStuff.InnerHtml = Convert.ToString(stuff);
The first step is for the consultant to make their code synchronous because their performance trick didn't work at all. If that fixes it, have the consultant properly implement using the Asynchronous Programming Design Pattern

I agree with him in part -- it's definitely better to check y for zero rather than catching the (expensive) exception. The out bool isSuccessful seems really dated to me, but whatever.
re: the asynchronous sessionid buffoonery -- may or may not be true, but it sounds like the consultant is blowing smoke for cover.

Cody's rule of thumb is dead right. If you have to ask, he probably doesn't.
It seems like point two its patently incorrect. .NET's standards explain that if a method fails it should throw an exception, which seems closer to the original; not the consulstant's suggestion. Assuming the exception is accurately & specifically describing the failure.

The consultants created the code in the first place right? And it doesn't work. I think you have quite a bit of dirt on them already.
The asynchronous answer sounds like BS, but there may be something in it. Presumably they have offered a suitable solution as well as pseudo-code describing the problem they themselves created. I would be more tempted to judge them on their solution rather than their expression of the problem. If their understanding is flawed their new solution won't work either. Then you'll know they are idiots. (In fact look round to see if you have a similar proof in any other areas of their code already)
The other one is a code style issue. There are a lot of different ways to cope with that. I personally don't like that style, but there will be circumstances under which it is suitable.

They're wrong on the async.
The assignment happens and then the page redirects. The function can start something asynchronously and return (and could even conceivably alter the Session in its own way), but whatever it does return has to be assigned in the code you gave before the redirect.
They're wrong on that defensive coding style in any low-level code and even in a higher-level function unless it's a specific business case that the 0 or NULL or empty string or whatever should be handled that way - in which case, it's always successful (that successful flag is a nasty code smell) and not an exception. Exceptions are for exceptions. You don't want to mask behaviors like this by coddling the callers of the functions. Catch things early and throw exceptions. I think Maguire covered this in Writing Solid Code or McConnell in Code Complete. Either way, it smells.

This guy does not know what he is doing. The obvious culprit is right here:
Session["UserID"] = SomeProprietarySessionManagementLookup();

I have to agree with John Rudy. My gut tells me the problem is in SomeProprietarySessionManagementLookup().
.. and your consultants do not sound to sure of themselves.

Storing in Session in not async. So that isn't true unless that function is async. But even so, since it isn't calling a BeginCall and have something to call on completion, the next line of code wouldn't execute until the Session line is complete.
For the second statement, while that could be used, it isn't exactly a best practice and you have a few things to note with it. You save the cost of throwing an exception, but wouldn't you want to know that you are trying to divide by zero instead of just moving past it?
I don't think that is a solid suggestion at all.

Quite strange. On the second item it may or may not be faster. It certainly isn't the same functionality though.

Typical "consultant" bollocks:
The problem is with whatever SomeProprietarySessionManagementLookup is doing
Exceptions are only expensive if they're thrown. Don't be afraid of try..catch, but throws should only occur in exceptional circumstances. If variable y shouldn't be zero then an ArgumentOutOfRangeException would be appropriate.

I'm guessing your consultant is suggesting use a status variable instead of exception for error handling is a better practice? I don't agree. How often does people forgot or too lazy to do error checking for return values? Also, pass/fail variable is not informative. There are more things can go wrong other than divide by zero like integer x/y is too big or x is NaN. When things go wrong, status variable cannot tell you what went wrong, but exception can. Exception is for exceptional case, and divide by zero or NaN are definitely exceptional cases.

The session thing is possible. It's a bug, beyond doubt, but it could be that the write arrives at whatever custom session state provider you're using after the next read. The session state provider API accommodates locking to prevent this sort of thing, but if the implementor has just ignored all that, your consultant could be telling the truth.
The second issue is also kinda valid. It's not quite idiomatic - it's a slightly reversed version of things like int.TryParse, which are there to avoid performance issues caused by throwing lots of exceptions. But unless you're calling that code an awful lot, it's unlikely it'll make a noticeable difference (compared to say, one less database query per page etc). It's certainly not something you should do by default.

If SomeProprietarySessionManagementLookup(); is doing an asynchronous assignment it would more likely look like this:
SomeProprietarySessionManagementLookup(Session["UserID"]);
The very fact that the code is assigning the result to Session["UserID"] would suggest that it is not supposed to be asynchronous and the result should be obtained before Response.Redirect is called. If SomeProprietarySessionManagementLookup is returning before its result is calculated they have a design flaw anyway.
The throw an exception or use an out parameter is a matter of opinion and circumstance and in actual practice won't amount to a hill of beans which ever way you do it. For the performance hit of exceptions to become an issue you would need to be calling the function a huge number of times which would probably be a problem in itself.

If the consultants deployed their ASP.NET application on your server(s), then they may have deployed it in uncompiled form, which means there would be a bunch of *.cs files floating around that you could look at.
If all you can find is compiled .NET assemblies (DLLs and EXEs) of theirs, then you should still be able to decompile them into somewhat readable source code. I'll bet if you look through the code you'll find them using static variables in their proprietary lookup code. You'd then have something very concrete to show your bosses.

This entire answer stream is full of typical programmer attitudes. It reminds me of Joel's 'Things you should never do' article (rewrite from scratch.) We don't really know anything about the system, other than there's a bug, and some guy posted some code online. There are so many unknowns that it is ridiculous to say "This guy does not know what he is doing."

Rather than pile on the Consultant, you could just as easily pile on the person who procured their services. No consultant is perfect, nor is a hiring manager ... but at the end of the day the real direction you should be taking is very clear: instead of trying to find fault you should expend energy into working collaboratively to find solutions. No matter how skilled someone is at their roles and responsibilities they will certainly have deficiencies. If you determine there is a pattern of incompentencies then you may choose to transition to another resource going forward, but assigning blame has never solved a single problem in history.

On the second point, I would not use exceptions here. Exceptions are reserved for exceptional cases.
However, division of anything by zero certainly does not equal zero (in math, at least), so this would be case specific.

Threadsafe foreach enumeration of lists

I need to enumerate though generic IList<> of objects. The contents of the list may change, as in being added or removed by other threads, and this will kill my enumeration with a "Collection was modified; enumeration operation may not execute."
What is a good way of doing threadsafe foreach on a IList<>? prefferably without cloning the entire list. It is not possible to clone the actual objects referenced by the list.

Cloning the list is the easiest and best way, because it ensures your list won't change out from under you. If the list is simply too large to clone, consider putting a lock around it that must be taken before reading/writing to it.

There is no such operation. The best you can do is
lock(collection){
foreach (object o in collection){
...
}
}

Your problem is that an enumeration does not allow the IList to change. This means you have to avoid this while going through the list.
A few possibilities come to mind:
Clone the list. Now each enumerator has its own copy to work on.
Serialize the access to the list. Use a lock to make sure no other thread can modify it while it is being enumerated.
Alternatively, you could write your own implementation of IList and IEnumerator that allows the kind of parallel access you need. However, I'm afraid this won't be simple.

ICollection MyCollection;
// Instantiate and populate the collection
lock(MyCollection.SyncRoot) {
// Some operation on the collection, which is now thread safe.
}
From MSDN

You'll find that's a very interesting topic.
The best approach relies on the ReadWriteResourceLock which use to have big performance issues due to the so called Convoy Problem.
The best article I've found treating the subject is this one by Jeffrey Richter which exposes its own method for a high performance solution.

So the requirements are: you need to enumerate through an IList<> without making a copy while simultaniously adding and removing elements.
Could you clarify a few things? Are insertions and deletions happening only at the beginning or end of the list?
If modifications can occur at any point in the list, how should the enumeration behave when elements are removed or added near or on the location of the enumeration's current element?
This is certainly doable by creating a custom IEnumerable object with perhaps an integer index, but only if you can control all access to your IList<> object (for locking and maintaining the state of your enumeration). But multithreaded programming is a tricky business under the best of circumstances, and this is a complex probablem.

Forech depends on the fact that the collection will not change. If you want to iterate over a collection that can change, use the normal for construct and be prepared to nondeterministic behavior. Locking might be a better idea, depending on what you're doing.

Default behavior for a simple indexed data structure like a linked list, b-tree, or hash table is to enumerate in order from the first to the last. It would not cause a problem to insert an element in the data structure after the iterator had already past that point or to insert one that the iterator would enumerate once it had arrived, and such an event could be detected by the application and handled if the application required it. To detect a change in the collection and throw an error during enumeration I could only imagine was someone's (bad) idea of doing what they thought the programmer would want. Indeed, Microsoft has fixed their collections to work correctly. They have called their shiny new unbroken collections ConcurrentCollections (System.Collections.Concurrent) in .NET 4.0.

I recently spend some time multip-threading a large application and had a lot of issues with the foreach operating on list of objects shared across threads.
In many cases you can use the good old for-loop and immediately assign the object to a copy to use inside the loop. Just keep in mind that all threads writing to the objects of your list should write to different data of the objects. Otherwise, use a lock or a copy as the other contributors suggest.
Example:
foreach(var p in Points)
{
// work with p...
}
Can be replaced by:
for(int i = 0; i < Points.Count; i ++)
{
Point p = Points[i];
// work with p...
}

Wrap the list in a locking object for reading and writing. You can even iterate with multiple readers at once if you have a suitable lock, that allows multiple concurrent readers but also a single writer (when there are no readers).

This is something that I've recently had to deal with and to me it really depends on what you're doing with the list.
If you need to use the list at a point in time (given the number of elements currently in it) AND another thread can only ADD to the end of the list, then maybe you just switch out to a FOR loop with a counter. At the point you grab the counter, you're only seeing X numbers of elements in the list. You can walk through the list (while others are adding to the end of it) . . . should not cause a problem.
Now, if the list needs to have items taken OUT of it by other threads, or CLEARED by other threads, then you'll need to implement one of the locking mechanisms mentioned above. Also, you may want to look at some of the newer "concurrent" collection classes (though I don't believe they implement IList - so you may need refactor for a dictionary).