Develop Reference

C# custom file parsing with 2 delimiters and different record types

I have a (not quite valid) CSV file that contains rows of multiple types. Any record could be one of about 6 different types and each type has a different number of properties. The first part of any row contains the timestamp and the type of record, followed by a standard CSV of the data.
Example
1456057920 PERSON, Ted Danson, 123 Fake Street, 555-123-3214, blah
1476195120 PLACE, Detroit, Michigan, 12345
1440581532 THING, Bucket, Has holes, Not a good bucket
And to make matters more complex, I need to be able to do different things with the records depending on certain criteria. So a PERSON type can be automatically inserted into a DB without user input, but a THING type would be displayed on screen for the user to review and approve before adding to DB and continuing the parse, etc.
Normally, I would use a library like CsvHelper to map the records to a type, but in this case since the types could be different, and the first part uses a space instead of comma, I dont know how to do that with a standard CSV library. So currently how I am doing it each loop is:
String split based off comma.
Split the first array item by the space.
Use a switch statement to determine the type and create the object.
Put that object into a List of type object.
Get confused as to where to go now because i now have a list of various types and will have to use yet another switch or if to determine the next parts.
I don't really know for sure if I will actually need that List but I have a feeling the user will want the ability to manually flip through records in the file.
By this point, this is starting to make for very long, confusing code, and my gut feeling tells me there has to be a cleaner way to do this. I thought maybe using Type.GetType(string) would help simplify the code some, but this seems like it might be terribly inefficient in a loop with 10k+ records and might make things even more confusing. I then thought maybe making some interfaces might help, but I'm not the greatest at using interfaces in this context and I seem to end up in about this same situation.
So what would be a more manageable way to parse this file? Are there any C# parsing libraries out there that would be able to handle something like this?

You can implement an IRecord interface that has a Timestamp property and a Process method (perhaps others as well).
Then, implement concrete types for each type of record.
Use a switch statement to determine the type and create and populate the correct concrete type.
Place each object in a List
After that you can do whatever you need. Some examples:
Loop through each item and call Process() to handle it.
Use linq .OfType<{concrete type}> to segment the list. (Warning with 10k
records, this would be slow since it would traverse the entire list for each concrete type.)
Use an overridden ToString method to give a single text representation of the IRecord
If using WPF, you can define a datatype template for each concrete type, bind an ItemsControl derivative to a collection of IRecords and your "detail" display (e.g. ListItem or separate ContentControl) will automagically display the item using the correct DataTemplate

Continuing in my comment - well that depends. What u described is actually pretty good for starters, u can of course expand it to a series of factories one for each object type - so that you move from explicit switch into searching for first factory that can parse a line. Might prove useful if u are looking to adding more object types in the future - you just add then another factory for new kind of object. Up to you if these objects should share a common interface. Interface is used generally to define a a behavior, so it doesn't seem so. Maybe you should rather just a Dictionary? You need to ask urself if you actually need strongly typed objects here? Maybe what you need is a simple class with ObjectType property and Dictionary of properties with some helper methods for easy typed properties access like GetBool, GetInt or generic Get?

Provide Dapper with a class instance to map to instead of always instantiating a new one?

I've looked through the source and I'm not finding anything (although I'm not great at IL), but I would like see if there is a way to provide Dapper a class instance instead of it always instantiating a new one. The reason for this is we may sometimes make two separate calls to two different stored procedures - one returns some columns of an 'entity', the other returns other columns. However, instead of the second query using the entity we received in the first call, we instead get two instances of essentially the same entity. It would be much more preferable for Dapper to use the existing entity class and map the query results to that existing class.
Is there any way to intercept Dapper's class instantiation so as to provide it with an existing instance if needed?

Excellent question. At the moment, it allows you to indicate a particular constructor, but it always news:
il.Emit(OpCodes.Newobj, specializedConstructor);
What we could do is make it possible to specify either a constructor or a static factory method; I suspect this would be just a three-line change to the core materializer code, plus a few other places. Not impossible, but then it gets into questions like calling-context: how does dapper provide caller-specified context to the factory. Again: all possible (protobuf-net does pretty much the same thing).
But none of that exists today. It wouldn't be impossible.

Real meaning of these linq querying styles

These linq queries can be written in both the ways. But choosing which way seems really to be confusing task. Please explain the difference in performance (if any) of these commands.
from table1Details in objDataContext.Table1s where table1Details.SomeId == 15
select new {....};
from table1Details in objDataContect.GetTable<Table1>() where table1Details.SomeId==15
select new {...};

That's not a difference related to LINQ at all. The data context is providing a property Table1s, which, internally, is just going to call GetTable<Table1>(). It's a convenience method with virtually no performance cost and absolutely no functional difference.

DataContext.GetTable Method
This method is the main entry point for querying. When a strongly
typed DataContext is created, new generated properties encapsulate
calls to this method. For example, a Customers property is
generated that returns GetTable<Customer>.
So for your case, there is no difference. Your DataContext has a property Table1, when you directly access it using objDataContext.Table1s it calls objDataContext.GetTable<Table1>()

But choosing which way seems really to be confusing task.
Why? What is confusing about it? Don't both ways work? Did you read the manual?
This method is the main entry point for querying. When a strongly typed DataContext is created, new generated properties encapsulate calls to this method. For example, a Customers property is generated that returns GetTable<Customer>.
GetTable<T>() just provides a generic way to access datasets, with runtime lookup (GetTable<UnknownEntityType>() will throw), as opposed to the generated, compile-time checked properties.

Linq to DataTable without enumerating fields

i´m trying to query a DataTable object without specifying the fields, like this :
var linqdata = from ItemA in ItemData.AsEnumerable()
select ItemA
but the returning type is
System.Data.EnumerableRowCollection<System.Data.DataRow>
and I need the following returning type
System.Data.EnumerableRowCollection<<object,object>>
(like the standard anonymous type)
Any idea?
Thanks

If I understand you correctly, you'd like to get a collection of objects that you don't need to define in your code but that are usable in a strongly typed fashion. Sadly, no you can't.
An anonymous type seems like some kind of variant or dynamic object, but it is in fact a strongly typed class that is defined at compile time. .NET defines the type for you automatically behind the scenes. In order for .net to be able to do this, it has to have some clue from the code with which to infer the type definition. It has to have something like:
from ItemA in ItemData.AsEnumerable()
select ItemA.Item("Name"), ItemA.Item("Email")
so it knows what members to define. There's no way to get around it, the information has to logically be there for the anonymous type to be defined.
Depending on why exactly your are trying to do this, there are some options.
If you want intellisense while still encapsulating your data access, you can return xml instead of a datatable from your encapsulated data access class. (You can convert data tables to xml very easily. You'll want to use the new System.Xml.Linq classes like the XElement. They're great!) Then you can use VS2008's ability to create an xsd schema from xml. Then use/import that schema at the top of your code page, and you have intellisense.
If you have to have an object an with properties for your data, but don't want to define a class/structure for them, you'll love the new dynamic objects coming in C#4.0/VB10. You have object properties based on what the sql returns, but you won't have intellisense. There is also a performance cost to this, but (a) that might not matter for your situation and (b) it actually is not so bad in some situations.
If you're just trying to avoid making a lot of classes, consider defining structs/structures on the same code file, beneath your class definition. When you add more columns to your result set, it's easy to adjust a struct with more public fields.
In short you can have any two of the following three: (a) dynamic, (b) strontly-typed objects, (3) intellisense. But not all three.

There is one way to accomplish what you want, but it required knowledge of dynamic linq. You would build the query during run-time and then use it. I am no expert and have never really played around with it, but here is a link to Scott Guthrie's blog about it - Dynamic Linq. Hope that helps.
Wade

Common problem for me in C#, is my solution good, stupid, reasonable? (Advanced Beginner)

Ok, understand that I come from Cold Fusion so I tend to think of things in a CF sort of way, and C# and CF are as different as can be in general approach.
So the problem is: I want to pull a "table" (thats how I think of it) of data from a SQL database via LINQ and then I want to do some computations on it in memory. This "table" contains 6 or 7 values of a couple different types.
Right now, my solution is that I do the LINQ query using a Generic List of a custom Type. So my example is the RelevanceTable. I pull some data out that I want to do some evaluation of the data, which first start with .Contains. It appears that .Contains wants to act on the whole list or nothing. So I can use it if I have List<string>, but if I have List<ReferenceTableEntry> where ReferenceTableEntry is my custom type, I would need to override the IEquatable and tell the compiler what exactly "Equals" means.
While this doesn't seem unreasonable, it does seem like a long way to go for a simple problem so I have this sneaking suspicion that my approach is flawed from the get go.
If I want to use LINQ and .Contains, is overriding the Interface the only way? It seems like if there way just a way to say which field to operate on. Is there another collection type besides LIST that maybe has this ability. I have started using List a lot for this and while I have looked and looked, a see some other but not necessarily superior approaches.
I'm not looking for some fine point of performance or compactness or readability, just wondering if I am using a Phillips head screwdriver in a Hex screw. If my approach is a "decent" one, but not the best of course I'd like to know a better, but just knowing that its in the ballpark would give me little "Yeah! I'm not stupid!" and I would finish at least what I am doing completely before switch to another method.
Hope I explained that well enough. Thanks for you help.

What exactly is it you want to do with the table? It isn't clear. However, the standard LINQ (-to-Objects) methods will be available on any typed collection (including List<T>), allowing any range of Where, First, Any, All, etc.
So: what is you are trying to do? If you had the table, what value(s) do you want?
As a guess (based on the Contains stuff) - do you just want:
bool x= table.Any(x=>x.Foo == foo); // or someObj.Foo
?

There are overloads for some of the methods in the List class that takes a delegate (optionally in the form of a lambda expression), that you can use to specify what field to look for.
For example, to look for the item where the Id property is 42:
ReferenceTableEntry found = theList.Find(r => r.Id == 42);
The found variable will have a reference to the first item that matches, or null if no item matched.
There are also some LINQ extensions that takes a delegate or an expression. This will do the same as the Find method:
ReferenceTableEntry found = theList.FirstOrDefault(r => r.Id == 42);

Ok, so if I'm reading this correctly you want to use the contains method. When using this with collections of objects (such as ReferenceTableEntry) you need to be careful because what you're saying is you're checking to see if the collection contains an object that IS the same as the object you're comparing against.
If you use the .Find() or .FindAll() method you can specify the criteria that you want to match on using an anonymous method.
So for example if you want to find all ReferenceTableEntry records in your list that have an Id greater than 1 you could do something like this
List<ReferenceTableEntry> listToSearch = //populate list here
var matches = listToSearch.FindAll(x => x.Id > 1);
matches will be a list of ReferenceTableEntry records that have an ID greater than 1.
having said all that, it's not completely clear that this is what you're trying to do.

Here is the LINQ query involved that creates the object I am talking about, and the problem line is:
.Where (searchWord => queryTerms.Contains(searchWord.Word))
List<queryTerm> queryTerms = MakeQueryTermList();
public static List<RelevanceTableEntry> CreateRelevanceTable(List<queryTerm> queryTerms)
{
SearchDataContext myContext = new SearchDataContext();
var productRelevance = (from pwords in myContext.SearchWordOccuranceProducts
where (myContext.SearchUniqueWords
.Where (searchWord => queryTerms.Contains(searchWord.Word))
.Select (searchWord => searchWord.Id)).Contains(pwords.WordId)
orderby pwords.WordId
select new {pwords.WordId, pwords.Weight, pwords.Position, pwords.ProductId});
}
This query returns a list of WordId's that match the submitted search string (when it was List and it was just the word, that works fine, because as an answerer mentioned before, they were the same type of objects). My custom type here is queryTerms, a List that contains WordId, ProductId, Position, and Weight. From there I go about calculating the relevance by doing various operations on the created object. Sum "Weight" by product, use position matches to bump up Weights, etc. My point for keeping this separate was that the rules for doing those operations will change, but the basic factors involved will not. I would have even rather it be MORE separate (I'm still learning, I don't want to get fancy) but the rules for local and interpreted LINQ queries seems to trip me up when I do.
Since CF has supported queries of queries forever, that's how I tend to lean. Pull the data you need from the db, then do your operations (which includes queries with Aggregate functions) on the in-memory table.
I hope that makes it more clear.