Develop Reference

C# custom file parsing with 2 delimiters and different record types

I have a (not quite valid) CSV file that contains rows of multiple types. Any record could be one of about 6 different types and each type has a different number of properties. The first part of any row contains the timestamp and the type of record, followed by a standard CSV of the data.
Example
1456057920 PERSON, Ted Danson, 123 Fake Street, 555-123-3214, blah
1476195120 PLACE, Detroit, Michigan, 12345
1440581532 THING, Bucket, Has holes, Not a good bucket
And to make matters more complex, I need to be able to do different things with the records depending on certain criteria. So a PERSON type can be automatically inserted into a DB without user input, but a THING type would be displayed on screen for the user to review and approve before adding to DB and continuing the parse, etc.
Normally, I would use a library like CsvHelper to map the records to a type, but in this case since the types could be different, and the first part uses a space instead of comma, I dont know how to do that with a standard CSV library. So currently how I am doing it each loop is:
String split based off comma.
Split the first array item by the space.
Use a switch statement to determine the type and create the object.
Put that object into a List of type object.
Get confused as to where to go now because i now have a list of various types and will have to use yet another switch or if to determine the next parts.
I don't really know for sure if I will actually need that List but I have a feeling the user will want the ability to manually flip through records in the file.
By this point, this is starting to make for very long, confusing code, and my gut feeling tells me there has to be a cleaner way to do this. I thought maybe using Type.GetType(string) would help simplify the code some, but this seems like it might be terribly inefficient in a loop with 10k+ records and might make things even more confusing. I then thought maybe making some interfaces might help, but I'm not the greatest at using interfaces in this context and I seem to end up in about this same situation.
So what would be a more manageable way to parse this file? Are there any C# parsing libraries out there that would be able to handle something like this?

You can implement an IRecord interface that has a Timestamp property and a Process method (perhaps others as well).
Then, implement concrete types for each type of record.
Use a switch statement to determine the type and create and populate the correct concrete type.
Place each object in a List
After that you can do whatever you need. Some examples:
Loop through each item and call Process() to handle it.
Use linq .OfType<{concrete type}> to segment the list. (Warning with 10k
records, this would be slow since it would traverse the entire list for each concrete type.)
Use an overridden ToString method to give a single text representation of the IRecord
If using WPF, you can define a datatype template for each concrete type, bind an ItemsControl derivative to a collection of IRecords and your "detail" display (e.g. ListItem or separate ContentControl) will automagically display the item using the correct DataTemplate

Continuing in my comment - well that depends. What u described is actually pretty good for starters, u can of course expand it to a series of factories one for each object type - so that you move from explicit switch into searching for first factory that can parse a line. Might prove useful if u are looking to adding more object types in the future - you just add then another factory for new kind of object. Up to you if these objects should share a common interface. Interface is used generally to define a a behavior, so it doesn't seem so. Maybe you should rather just a Dictionary? You need to ask urself if you actually need strongly typed objects here? Maybe what you need is a simple class with ObjectType property and Dictionary of properties with some helper methods for easy typed properties access like GetBool, GetInt or generic Get?

Fill object from SQL result

I have a regular C# class called "vehicle" with properties like Name, NumberPlate, MaxSpeed, etc.
All the data for the class is stored in a SQLite Database where I have a Table "Car" and "Boat". The tables colums have the same names as the class properties (however, there are more columns than class properties - vehicle is a more generic abstraction). At the moment, I have to assign the result of the query individually one by one like this:
while (await statement.StepAsync())
{
myVehicle.Name = statement.Columns["Name"];
//[...]
myVehicle.MaxSpeed = decimal.TryParse(statement.Columns["MaxSpeed"]);
}
Additionally, I have to check if some columns exist ("Car" and "Boat" have a different set of columns) which is more code than I'd like it to be.
I read about EntityFramework to map my db table to my class - but that seems overkill. My requirement is to map properties and columns that have the same name and ignore everything else.
Is there a "easy" (dev time, lines of code) way to map my table columns to my class?
Thanks for reading!

The restrictions in phone 8 mean that a lot of the standard answers to this ("just use {some ORM / micro-ORM}") won't apply, since they don't work on phone 8. You can probably use reflection for a lot of this, but: reflection can be (relatively) slow, so it depends on how much data you will be processing. If it is occasional and light: fine, reflect away.
Runtime meta-programming (the tricks used by libraries like "dapper" in full .NET to make these things really fast) is not available on restricted runtimes, so if you want to avoid lots of boiler-plate that leaves build-time meta-programming. At the simplest, I wonder if you could use something like T4 to automate creating these methods for you as C#. There are also ways to use the reflection-emit API to construct assemblies (at build-time) for phone 8, but that is a pretty hard-core route.
My thoughts:
if the amount of types here isn't huge, just write the code
if you have a lot of types, or you just feel like it, consider a build-time code-generation meta-programming step; you might even think "hmm, is this something I could make available to the community?"
of course, the first thing to do is to check that such a thing doesn't already exist

There is a little helper which might fit your case. Basically, it will take a dictionary and try it's best to populate a objects properties using reflection. I didn't try it by myself though.
You'd simply do something like:
while (await statement.StepAsync())
{
myVehicle = DictionaryToObject<Car>(statement.Columns);
}
It might need some further work to get it running but maybe a good start.

Best way to represent text in memory

I have 2 classes that inherit from a common base class.
Each of these specialized classes load some data from a data base, process it and then save that information in text files.
The first class represents the information as a XML document.
The second class will store its information as a text file with delimiters separating fields.
What I want to do is to write a single Save method in the base class that can be used by both classes. As all classes will write to text files I was thinking in use a common representation to store their data in memory - for instance the first class will transform the XmlDocument to that common representation.
What is the best way to store this in memory, string, Stream?
Thanks

If the derived classes represent the data very differently, don't implement a common Save method for them, Those classes knows best how to Save their data.
Make Save() abstract and have each of the subclass implement the saving.
There might be something in common for doing a Save() (e.g. opening the actual file, error handling). So have your base class provide a Save() method that's responsible for that which in turn calls a virtual Save(System.IO.TextWriter writer); method that each of your subclasses implement.

Given that XML is the richer of the two formats you mention, and relatively easy to manipulate, why not have a single representation and 2 save methods?

If the input data is uniformly structured, you can likely store it cleanly in a DataSet, and maybe load directly from your XML using DataSet.ReadXml on a TextReader input.
If you only have one type of record to output to the delimited textfile, DataTable could be used directly - a DataSet encapsulates multiple DataTables.
Another alternative might be to convert XML directly to CSV (comma = delimiter here, you could use whatever you wanted though) using XSLT as shown here by #Welbog.

DataTable vs. Collection in .Net

I am writing a program that needs to read a set of records that describe the register map of a device I need to communicate with. Each record will have a handfull of fields that describe the properties of each register.
I don't really need to edit or modify the data in my VB or C# program, though I would like to be able to display the data on a grid. I would like to store the data in a CSV file, or perhaps an XML file. I need to enable users to edit the data off-line, preferably in excel.
I am considering using a DataTable or a Collection of "Register" objects (which I would define).
I prototyped a DataTable, and found I can read/write XML easily using the built in methods and I can easily bind to a DataGridView. I was not able to find a way to retreive info on a single register without using a query that returns a collection of rows, even though I defined a unique primaty key column. The syntax to get a value from a column is also complex, though I could be missing something on both counts.
I'm tempted to use a collection of "Register" objects that I can access via a unique key. It would be a little more coding up front, but seems like a cleaner solution overall. I should still be able to use LINQ to dataset to query subsets of registers when I need them, but would also be able to grab a single field using a the key value, something like this: Registers(keyValue).fieldName).
Which would be a cleaner approach to the problem?
Is there a way to read/write XML into a Collection without needing custom code?
Could this be accomplished using String for a key?
UPDATE: Sounds like the consensus is towards the Collection of register Objects. Makes sense to me. I was leaning that way, and since nobody pointed out any DataTable features that would simplify acessing a single row, it looks like the Collection is clearly the way to go. Thanks to those who weighed in.

I would be inclined not to use data sets. It would be better to work with objects and collections. Your code will be more maintainable/readable, composable, testable & reusable.
Given that you can do queries on the data set to return particular row, you might find that a LINQ query to turn the rows into objects may be all the custom code that you need.
Using a Dictionary<string, Register> for look ups is a good idea if you have a large number of items (say greater than 1000). Otherwise a simple LINQ query should be fine.

It depends on how you define 'clean'.
A generic collection is potentially MUCH more lightweight than a DataTable. But on the other hand that doesn't seem to be too much of an issue for you. And unless you go into heavy reflection you'll have to write some code to read/write xml.
If you use a key I'd also recommend (in the case of the collection) to use a Dictionary. That way you have a Collection of the raw data and still can identify each entry through the key in the Dictionary.

I usually use datatables if its something quick and unlikely to be used in any other way. If it's something I can see evolving into an object that has its own use within the app (like your Register Object you mentioned).
It might be a little extra code up front - but it saves converting from a datatable to the collection in the future if you come up with something you would like to do based on an individual row, or if you want/need to add some sort of extra functionality to that element down the road.

I would go with the collection of objects so you can swap out the data access later if you need to.

You can serialize classes with an xml serializer and defining a Serialize attribute or something like that (it has been a while since I done that, sorry for the vagueness). A DataSet or DataTable works great with XML.
Both DS and DT have ReadXml and WriteXml methods. XML must be predefined format, but it works seamlessly.
Otherwise, I personally like collections or dictionaries; DS/DT are OK, but I like custom objects, and LINQ adds in some power.
HTH.

What is the most DRY way to get data out of my database?

I have to write an ASP.NET application that connects to our legacy IBM Universe Database and we are using a product called mv.net which allows us to connect, read, write, select, run server side programs, etc.
I want as little code repetition as possible but I also want as little data transfer as possible as well.
In order to open a connection we must first get a reference to the account using code such as:
mvAccount myAccount = new mvAccount(serverName, login);
Then we can read an item:
mvFile myInvoiceFile = myAccount.FileOpen("INVOICE");
mvItem myInvoiceRecord = myInvoiceFile.Read(invoiceID);
Then we we're done:
myAccount.Logout();
I have a Class for each module, so I may have INVOICE, PURCHASE_ORDER, RMA, REQ, SHIPMENT, and so on. Within INVOICE, I may need to access multiple tables such as CUSTOMER, INVOICE, TERMS, SHIPVIA, etc.
What I planned to do was create a class called TechDB which is the name of our database and put the code in there so in my INVOICE class I can just say:
TechDB connection = new TechDB();
mvItem myInvoiceRecord = connection.Read("INVOICE", invoiceID)
When I do this my TechDB class would open the connection, read the record, and then logout all in one step.
I think I'm heading down the right path but please let me know if not. Here are my problems with this:
How do I return errors my INVOICE class? For example, errors could occur if we are unable to connect to the database, unable to open the file, unable to read the record.
What if I then need to take some data from my INVOICE and then read the TERMS table. I'd hate to have to open a new connection to the database when I just opened one.
Should I call the Dispose method on all classes that have this? For example, the mvAccount has a Dispose method. None of the documentation says to call it, but should I after the Logout()?
Could I create a Dispose method on the TechDB class that does the myAccount.Logout()? That way the connection would remain open and I could close it from my INVOICE class when I was completely done with it?
Give me some opinions on the best way to handle this? My goal is a robust application that is easy to modify and as little code repitition as possible.

I would use closures, I think for C# you have Delegates. So something like:
MyAccount.loginAndDo(servername, login, delegate(account){
invoice = account.read("INVOICE");
.
.
.
});
In loginAndDo, you would login, call the delegate, and then close the account.

Custom exception classes
One idea would be to construct batch requests, (a list of delegates).
as per 3/4. In my case, all of my data access objects inherit from a class that holds a static reference to a connection. I'm hesitant to implement disconnect logic in Dispose because of the possibility that there is a power out or system crash or something and that connection isn't released.