Develop Reference

How to detect if a row has extra columns (more than the header)

While reading a CSV file, how can I configure CsvHelper to enforce that each row has no extra columns that are not found in the header? I cannot find any obvious property under CsvConfiguration nor under CsvHelper.Configuration.Attributes.
Context: In our CSV file format, the last column is a string description, which our users (using plain-text editors) sometimes forget to quote when the description contains commas. Such "raw" commas cause that row to have extra columns, and the intended description read into the software omits the description after the first raw comma. I want to detect this and throw an exception that suggests to the user they may have forgotten to quote the description cell.
It looks like CsvConfiguration.DetectColumnCountChanges might be related, but presently the 29.0.0 library lacks any Intellisense description of CsvConfiguration properties, so I have no idea how to use this.
Similar information for other CSV libraries:
With LINQtoCSV this was done by setting IgnoreUnknownColumns = false in CsvFileDescription.
Can Lumenworks CSV parser error when there are too many columns in a row?

You were on the right track with CsvConfiguration.DetectColumnCountChanges.
void Main()
{
var config = new CsvConfiguration(CultureInfo.InvariantCulture)
{
DetectColumnCountChanges = true
};
using (var reader = new StringReader("Id,Name\n1,MyName\n2,YourName,ExtraColumn"))
using (var csv = new CsvReader(reader, config))
{
try
{
var records = csv.GetRecords<Foo>().ToList();
}
catch (BadDataException ex)
{
if (ex.Message.StartsWith("An inconsistent number of columns has been detected."))
{
Console.WriteLine("There is an issue with an inconsistent number of columns on row {0}", ex.Context.Parser.RawRow);
Console.WriteLine("Row data: \"{0}\"", ex.Context.Parser.RawRecord);
Console.WriteLine("Please check for commas in a field that were not properly quoted.");
}
}
}
}
public class Foo
{
public int Id { get; set; }
public string Name { get; set; }
}

Send the contents of a datatable to a csv file with headers

I am trying to send the contents of a datatable to a csv file, with headers. There is a duplicate question but the the accepted answers only seem to work half way. At this point I have mixed and matched the upvoted answers with no luck and need a point in the right direction.
I can write the columns to the file just fine, and I can write data just fine but not together. Also the data never comes out quoted, only comma delimited without quotes.
//This is how the FileHelpers class is built
public class ScannedFileInfo
{
//this prefix will handle the double quotes issue
//delimiters must reside between double quotes
//Must specify FieldOrder too
[FieldQuoted('"', QuoteMode.AlwaysQuoted)]
[FieldOrder(1)]
public string DA_Id;
[FieldQuoted('"', QuoteMode.AlwaysQuoted)]
[FieldOrder(2)]
public string Name;
[FieldQuoted('"', QuoteMode.AlwaysQuoted)]
[FieldOrder(3)]
public string Extension;
[FieldQuoted('"', QuoteMode.AlwaysQuoted)]
[FieldOrder(4)]
public string Fullname;
[FieldQuoted('"', QuoteMode.AlwaysQuoted)]
[FieldOrder(5)]
public string PathLength;
[FieldQuoted('"', QuoteMode.OptionalForBoth)]
[FieldOrder(6)]
public string Directory;
}
//this is how I send it to the file
public static void ImportDirectory(string result, SqlConnection myconn, SqlConnection destConn ,ListBox lbInfo, DataGridView dgView)
{
//create data table code here - works fine...
MessageBox.Show("Scan complete. Starting Import...");
//build file
var engine = new FileHelperEngine<ScannedFileInfo>();
var orders = new List<ScannedFileInfo>();
engine.HeaderText = engine.GetFileHeader();
engine.WriteFile(#"C:\DirectoryScan.csv", orders);
MessageBox.Show("You now have proper labeled columns in your file.");
//now the data import is successful but it overwrites the column info
CommonEngine.DataTableToCsv(dt, #"C:\DirectoryScan.csv", ',');
MessageBox.Show("You now have data in your file, but no columns");
}

The CommonEngine.DataTableToCsv() call does not support all of the features of FileHelpersEngine. In particular, the field quoting and the column headers are missing. The CommonEngine is not as full featured as the FileHelpersEngine<T>.
Instead, you should create a list of ScannedFileInfo instead of a datatable and then use FileHelpersEngine<ScannedFileInfo>.WriteFile().
//create list of ScannedFileInfo here
var records = new List<ScannedFileInfo>();
records.Add(new ScannedFileInfo() { Name = ..., etc... });
// ( or if you prefer, loop through your datatable to build the list)
MessageBox.Show("Scan complete. Starting Import...");
// build file
var engine = new FileHelperEngine<ScannedFileInfo>();
engine.HeaderText = engine.GetFileHeader();
engine.WriteFile(#"C:\DirectoryScan.csv", records);
MessageBox.Show("You now have data and properly labelled columns in your file.");

I cannot recommend that you look into CSVHelper enough. This nuget package has saved me from pulling my hair out on so many occasions, I can't count them all.
If you're into using it, you can use the configuration settings to quickly set a header record, delimiter, ignore quotes or not, etc.

LinqToExcel returns null

I have an excel sheet of xls format, I am using LinqToExcel to read it and then import it to my DB.
This sheet consist of about 3K row and only 6 cols. I am using .addmapping to map my class properties to the column names
The problem i have is: the cells of column "web-code" are SOMETIMES coming back as null although there are data in the cells.
Here is a sample data that is coming as null!
My Code watch
And here is a sample data where the data coming correct:
My Code Watch
I have tried applying ExcelColumn attribute for mapping, but no luck!
code:
var factory = new ExcelQueryFactory(_excelFilePath);
factory.AddMapping<ExcelPriceEntity>(x => x.WebCode, "WEB-CODE");
factory.AddMapping<ExcelPriceEntity>(x => x.Type, "TYPE");
factory.AddMapping<ExcelPriceEntity>(x => x.Style, "STYLE");
factory.AddMapping<ExcelPriceEntity>(x => x.Qty, "QTY");
factory.AddMapping<ExcelPriceEntity>(x => x.UnitPrice, "Unit Price");
factory.AddMapping<ExcelPriceEntity>(x => x.Bucket, "WEBCODE W/BUCKET");
factory.StrictMapping = StrictMappingType.ClassStrict;
factory.TrimSpaces = TrimSpacesType.Both;
factory.ReadOnly = true;
var prices = factory.Worksheet<ExcelPriceEntity>(_allPricesSheetName).ToList();
var priccerNP = prices.Where(p => p.Type.Contains("900 ARROW TAPE")).ToList();
My PriceEntity Class:
public class ExcelPriceEntity
{
//[ExcelColumn("TYPE")]
public string Type { get; set; }
public string WebCode { get; set; }
//[ExcelColumn("STYLE")]
public string Style { get; set; }
//[ExcelColumn("QTY")]
public string Qty { get; set; }
//[ExcelColumn("Unit Price")]
public string UnitPrice { get; set; }
//[ExcelColumn("WEBCODE W/BUCKET")]
public string Bucket { get; set; }
}

Alternate Solution:
I ended up saving the excel sheet as csv file, then import to SQL table.Then i used linq-to-sql to read the data.
Root Cause:
After researching i found out the problem was that the first cell of this column(web-code) was interger number, and excel trys to figure out the datatype of the column by looking at the first rows!
So next rows of (web-code) column was some text data. So excel couldn't parse it as integer, and assign null value to it!
What I could've done is, assing text value to the first cell so excel would guess the data type as string. But I didn't test that. For anyone reading this answer, try having text value in you first row if you came across the same problem

here, the Contains is not like string contains. It compares a list of cell values to the exact value u give inside the contains method. just try with the full text "900 AMMONIA STOCK OR CUST...)

Another alternative solution to #alsafoo is to convert the column from "general" to "text".
These are the steps:
1. Right click on any cell in the column.
2. In Number tab, select text.
3. Select Format cell
4. Press Ok.
After then, the library will read all values as string.

Column headers in CSV using fileHelpers library?

Is there a built-in field attribute in the FileHelper library which will add a header row in the final generated CSV?
I have Googled and didn't find much info on it. Currently I have this:
DelimitedFileEngine _engine = new DelimitedFileEngine(T);
_engine.WriteStream
(HttpContext.Current.Response.Output, dataSource, int.MaxValue);
It works, but without a header.
I'm thinking of having an attribute like FieldTitleAttribute and using this as a column header.
So, my question is at which point do I check the attribute and insert header columns? Has anyone done something similar before?
I would like to get the headers inserted and use custom text different from the actual field name just by having an attribute on each member of the object:
[FieldTitleAttribute("Custom Title")]
private string Name
and maybe an option to tell the engine to insert the header when it's generated.
So when WriteStream or WriteString is called, the header row will be inserted with custom titles.
I have found a couple of Events for DelimitedFileEngine, but not what's the best way to detect if the current record is the first row and how to insert a row before this.

I know this is an old question, but here is an answer that works for v2.9.9
FileHelperEngine<Person> engine = new FileHelperEngine<Person>();
engine.HeaderText = engine.GetFileHeader();

Here's some code that'll do it: https://gist.github.com/1391429
To use it, you must decorate your fields with [FieldOrder] (a good FileHelpers practice anyway). Usage:
[DelimitedRecord(","), IgnoreFirst(1)]
public class Person
{
// Must specify FieldOrder too
[FieldOrder(1), FieldTitle("Name")]
string name;
[FieldOrder(2), FieldTitle("Age")]
int age;
}
...
var engine = new FileHelperEngine<Person>
{
HeaderText = typeof(Person).GetCsvHeader()
};
...
engine.WriteFile(#"C:\people.csv", people);
But support for this really needs to be added within FileHelpers itself. I can think of a few design questions off the top of my head that would need answering before it could be implemented:
What happens when reading a file? Afaik FileHelpers is currently all based on ordinal column position and ignores column names... but if we now have [FieldHeader] attributes everywhere then should we also try matching properties with column names in the file? Should you throw an exception if they don't match? What happens if the ordinal position doesn't agree with the column name?
When reading as a data table, should you use A) the field name (current design), or B) the source file column name, or C) the FieldTitle attribute?

I don't know if you still need this, but here is the way FileHelper is working :
To include headers of columns, you need to define a string with headers delimited the same way as your file.
For example with '|' as delimiter :
public const string HeaderLine = #"COLUMN1|COLUMN2|COLUMN3|...";
Then, when calling your engine :
DelimitedFileEngine _engine = new DelimitedFileEngine<T> { HeaderText = HeaderLine };
If you don't want to write the headers, just don't set the HeaderText attribute on the engine.

List<MyClass> myList = new List<MyClass>();
FileHelperEngine engine = new FileHelperEngine(typeof(MyClass));
String[] fieldNames = Array.ConvertAll<FieldInfo, String>(typeof(MyClass).GetFields(), delegate(FieldInfo fo) { return fo.Name; });
engine.HeaderText = String.Join(";", fieldNames);
engine.WriteFile(MapPath("MyClass.csv"), myList);

Just to include a more complete example, which would have saved me some time, for version 3.4.1 of the FileHelpers NuGet package....
Given
[DelimitedRecord(",")]
public class Person
{
[FieldCaption("First")]
public string FirstName { get; set; }
[FieldCaption("Last")]
public string LastName { get; set; }
public int Age { get; set; }
}
and this code to create it
static void Main(string[] args)
{
var people = new List<Person>();
people.Add(new Person() { FirstName = "James", LastName = "Bond", Age = 38 });
people.Add(new Person() { FirstName = "George", LastName = "Washington", Age = 43 });
people.Add(new Person() { FirstName = "Robert", LastName = "Redford", Age = 28 });
CreatePeopleFile(people);
}
private static void CreatePeopleFile(List<Person> people)
{
var engine = new FileHelperEngine<Person>();
using (var fs = File.Create(#"c:\temp\people.csv"))
using (var sw = new StreamWriter(fs))
{
engine.HeaderText = engine.GetFileHeader();
engine.WriteStream(sw, people);
sw.Flush();
}
}
You get this
First,Last,Age
James,Bond,38
George,Washington,43
Robert,Redford,28

I found that you can use the FileHelperAsyncEngine to accomplish this. Assuming your data is a list called "output" of type "outputData", then you can write code that looks like this:
FileHelperAsyncEngine outEngine = new FileHelperAsyncEngine(typeof(outputData));
outEngine.HeaderText = "Header1, Header2, Header3";
outEngine.BeginWriteFile(outputfile);
foreach (outputData line in output){
outEngine.WriteNext(line);
}
outEngine.Close();

You can simply use FileHelper's GetFileHeader function from base class
var engine = new FileHelperEngine<ExportType>();
engine.HeaderText = engine.GetFileHeader();
engine.WriteFile(exportFile, exportData);

Parsing a CSV formatted text file

I have a text file that looks like this:
1,Smith, 249.24, 6/10/2010
2,Johnson, 1332.23, 6/11/2010
3,Woods, 2214.22, 6/11/2010
1,Smith, 219.24, 6/11/2010
I need to be able to find the balance for a client on a given date.
I'm wondering if I should:
A. Start from the end and read each line into an Array, one at a time.
Check the last name index to see if it is the client we're looking for.
Then, display the balance index of the first match.
or
B. Use RegEx to find a match and display it.
I don't have much experience with RegEx, but I'll learn it if it's a no brainer in a situation like this.

I would recommend using the FileHelpers opensource project:
http://www.filehelpers.net/
Piece of cake:
Define your class:
[DelimitedRecord(",")]
public class Customer
{
public int CustId;
public string Name;
public decimal Balance;
[FieldConverter(ConverterKind.Date, "dd-MM-yyyy")]
public DateTime AddedDate;
}
Use it:
var engine = new FileHelperAsyncEngine<Customer>();
// Read
using(engine.BeginReadFile("TestIn.txt"))
{
// The engine is IEnumerable
foreach(Customer cust in engine)
{
// your code here
Console.WriteLine(cust.Name);
// your condition >> add balance
}
}

This looks like a pretty standard CSV type layout, which is easy enough to process. You can actually do it with ADO.Net and the Jet provider, but I think it is probably easier in the long run to process it yourself.
So first off, you want to process the actual text data. I assume it is reasonable to assume each record is seperated by some newline character, so you can utilize the ReadLine method to easily get each record:
StreamReader reader = new StreamReader("C:\Path\To\file.txt")
while(true)
{
var line = reader.ReadLine();
if(string.IsNullOrEmpty(line))
break;
// Process Line
}
And then to process each line, you can split the string on comma, and store the values into a data structure. So if you use a data structure like this:
public class MyData
{
public int Id { get; set; }
public string Name { get; set; }
public decimal Balance { get; set; }
public DateTime Date { get; set; }
}
And you can process the line data with a method like this:
public MyData GetRecord(string line)
{
var fields = line.Split(',');
return new MyData()
{
Id = int.Parse(fields[0]),
Name = fields[1],
Balance = decimal.Parse(fields[2]),
Date = DateTime.Parse(fields[3])
};
}
Now, this is the simplest example, and doesn't account for cases where the fields may be empty, in which case you would either need to support NULL for those fields (using nullable types int?, decimal? and DateTime?), or define some default value that would be assigned to those values.
So once you have that you can store the collection of MyData objects in a list, and easily perform calculations based on that. So given your example of finding the balance on a given date you could do something like:
var data = customerDataList.First(d => d.Name == customerNameImLookingFor
&& d.Date == dateImLookingFor);
Where customerDataList is the collection of MyData objects read from the file, customerNameImLookingFor is a variable containing the customer's name, and customerDateImLookingFor is a variable containing the date.
I've used this technique to process data in text files in the past for files ranging from a couple records, to tens of thousands of records, and it works pretty well.

I think the cleanest way is to load the entire file into an array of custom objects and work with that. For 3 MB of data, this won't be a problem. If you wanted to do completely different search later, you could reuse most of the code. I would do it this way:
class Record
{
public int Id { get; protected set; }
public string Name { get; protected set; }
public decimal Balance { get; protected set; }
public DateTime Date { get; protected set; }
public Record (int id, string name, decimal balance, DateTime date)
{
Id = id;
Name = name;
Balance = balance;
Date = date;
}
}
…
Record[] records = from line in File.ReadAllLines(filename)
let fields = line.Split(',')
select new Record(
int.Parse(fields[0]),
fields[1],
decimal.Parse(fields[2]),
DateTime.Parse(fields[3])
).ToArray();
Record wantedRecord = records.Single
(r => r.Name = clientName && r.Date = givenDate);

Note that both your options will scan the file. That is fine if you only want to search in the file for 1 item.
If you need to search for multiple client/date combinations in the same file, you could parse the file into a Dictionary<string, Dictionary <date, decimal>> first.
A direct answer: for a one-off, a RegEx will probably be faster.

If you're just reading it I'd consider reading in the whole file in memory using StreamReader.ReadToEnd and then treating it as one long string to search through and when you find a record you want to look at just look for the previous and next line break and then you have the transaction row you want.
If it's on a server or the file can be refreshed all the time this might not be a good solution though.

If it's all well-formatted CSV like this then I'd use something like the Microsoft.VisualBasic.TextFieldParser class or the Fast CSV class over on code project to read it all in.
The data type is a little tricky because I imagine not every client has a record for every day. That means you can't just have a nested dictionary for your looksup. Instead, you want to "index" by name first and then date, but the form of the date record is a little different. I think I'd go for something like this as I read in each record:
Dictionary<string, SortedList<DateTime, double>>

hey, hey, hey!!! why not do it with this great project on codeproject Linq to CSV, way cool!
rock solid