Develop Reference

C# Reading CSV to DataTable and Invoke Rows/Columns

i am currently working on a small Project and i got stuck with a Problem i currently can not manage to solve...
I have multiple ".CSV" Files i want to read, they all have the same Data just with different Values.
Header1;Value1;Info1
Header2;Value2;Info2
Header3;Value3;Info3
While reading the first File i Need to Create the Headers. The Problem is they are not splited in Columns but in rows (as you can see above Header1-Header3).
Then it Needs to read the Value 1 - Value 3 (they are listed in the 2nd Column) and on top of that i Need to create another Header -> Header4 with the data of "Info2" which is always placed in Column 3 and Row 2 (the other values of Column 3 i can ignore).
So the Outcome after the first File should look like this:
Header1;Header2;Header3;Header4;
Value1;Value2;Value3;Info2;
And after multiple files it sohuld be like this:
Header1;Header2;Header3;Header4;
Value1;Value2;Value3;Value4;
Value1b;Value2b;Value3b;Value4b;
Value1c;Value2c;Value3c;Value4c;
I tried it with OleDB but i get the Error "missing ISAM" which i cant mange to fix. The Code i Used is the following:
public DataTable ReadCsv(string fileName)
{
DataTable dt = new DataTable("Data");
/* using (OleDbConnection cn = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0;Data Source=\"" +
Path.GetDirectoryName(fileName) + "\";Extendet Properties ='text;HDR=yes;FMT=Delimited(,)';"))
*/
using (OleDbConnection cn = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" +
Path.GetDirectoryName(fileName) + ";Extendet Properties ='text;HDR=yes;FMT=Delimited(,)';"))
{
using(OleDbCommand cmd = new OleDbCommand(string.Format("select *from [{0}]", new FileInfo(fileName).Name,cn)))
{
cn.Open();
using(OleDbDataAdapter adapter = new OleDbDataAdapter(cmd))
{
adapter.Fill(dt);
}
}
}
return dt;
}
Another attempt i did was using StreamReader. But the Headers are in the wrong place and i dont know how to Change this + do this for every file. the Code i tried is the following:
public static DataTable ReadCsvFilee(string path)
{
DataTable oDataTable = new DataTable();
var fileNames = Directory.GetFiles(path);
foreach (var fileName in fileNames)
{
//initialising a StreamReader type variable and will pass the file location
StreamReader oStreamReader = new StreamReader(fileName);
// CONTROLS WHETHER WE SKIP A ROW OR NOT
int RowCount = 0;
// CONTROLS WHETHER WE CREATE COLUMNS OR NOT
bool hasColumns = false;
string[] ColumnNames = null;
string[] oStreamDataValues = null;
//using while loop read the stream data till end
while (!oStreamReader.EndOfStream)
{
String oStreamRowData = oStreamReader.ReadLine().Trim();
if (oStreamRowData.Length > 0)
{
oStreamDataValues = oStreamRowData.Split(';');
//Bcoz the first row contains column names, we will poluate
//the column name by
//reading the first row and RowCount-0 will be true only once
// CHANGE TO CHECK FOR COLUMNS CREATED
if (!hasColumns)
{
ColumnNames = oStreamRowData.Split(';');
//using foreach looping through all the column names
foreach (string csvcolumn in ColumnNames)
{
DataColumn oDataColumn = new DataColumn(csvcolumn.ToUpper(), typeof(string));
//setting the default value of empty.string to newly created column
oDataColumn.DefaultValue = string.Empty;
//adding the newly created column to the table
oDataTable.Columns.Add(oDataColumn);
}
// SET COLUMNS CREATED
hasColumns = true;
// SET RowCount TO 0 SO WE KNOW TO SKIP COLUMNS LINE
RowCount = 0;
}
else
{
// IF RowCount IS 0 THEN SKIP COLUMN LINE
if (RowCount++ == 0) continue;
//creates a new DataRow with the same schema as of the oDataTable
DataRow oDataRow = oDataTable.NewRow();
//using foreach looping through all the column names
for (int i = 0; i < ColumnNames.Length; i++)
{
oDataRow[ColumnNames[i]] = oStreamDataValues[i] == null ? string.Empty : oStreamDataValues[i].ToString();
}
//adding the newly created row with data to the oDataTable
oDataTable.Rows.Add(oDataRow);
}
}
}
//close the oStreamReader object
oStreamReader.Close();
//release all the resources used by the oStreamReader object
oStreamReader.Dispose();
}
return oDataTable;
}
I am thankful for everyone who is willing to help. And Thanks for reading this far!
Sincerely yours

If I understood you right, there is a strict parsing there like this:
string OpenAndParse(string filename, bool firstFile=false)
{
var lines = File.ReadAllLines(filename);
var parsed = lines.Select(l => l.Split(';')).ToArray();
var header = $"{parsed[0][0]};{parsed[1][0]};{parsed[2][0]};{parsed[1][0]}\n";
var data = $"{parsed[0][1]};{parsed[1][1]};{parsed[2][1]};{parsed[1][2]}\n";
return firstFile
? $"{header}{data}"
: $"{data}";
}
Where it would return - if first file:
Header1;Header2;Header3;Header2
Value1;Value2;Value3;Value4
if not first file:
Value1;Value2;Value3;Value4
If I am correct, rest is about running this against a list file of files and joining the results in an output file.
EDIT: Against a directory:
void ProcessFiles(string folderName, string outputFileName)
{
bool firstFile = true;
foreach (var f in Directory.GetFiles(folderName))
{
File.AppendAllText(outputFileName, OpenAndParse(f, firstFile));
firstFile = false;
}
}
Note: I missed you want a DataTable and not an output file. Then you could simply create a list and put the results into that list making the list the datasource for your datatable (then why would you use semicolons in there? Probably all you need is to simply attach the array values to a list).

(Adding as another answer just to make it uncluttered)
void ProcessMyFiles(string folderName)
{
List<MyData> d = new List<MyData>();
var files = Directory.GetFiles(folderName);
foreach (var file in files)
{
OpenAndParse(file, d);
}
string[] headers = GetHeaders(files[0]);
DataGridView dgv = new DataGridView {Dock=DockStyle.Fill};
dgv.DataSource = d;
dgv.ColumnAdded += (sender, e) => {e.Column.HeaderText = headers[e.Column.Index];};
Form f = new Form();
f.Controls.Add(dgv);
f.Show();
}
string[] GetHeaders(string filename)
{
var lines = File.ReadAllLines(filename);
var parsed = lines.Select(l => l.Split(';')).ToArray();
return new string[] { parsed[0][0], parsed[1][0], parsed[2][0], parsed[1][0] };
}
void OpenAndParse(string filename, List<MyData> d)
{
var lines = File.ReadAllLines(filename);
var parsed = lines.Select(l => l.Split(';')).ToArray();
var data = new MyData
{
Col1 = parsed[0][1],
Col2 = parsed[1][1],
Col3 = parsed[2][1],
Col4 = parsed[1][2]
};
d.Add(data);
}
public class MyData
{
public string Col1 { get; set; }
public string Col2 { get; set; }
public string Col3 { get; set; }
public string Col4 { get; set; }
}

I don't know if this is the best way to do this. But what i would have done in your case, is to rewrite the CSV's the conventionnal way while reading all the files, then create a stream containing the new CSV created.
It would look like something like this :
var csv = new StringBuilder();
csv.AppendLine("Header1;Header2;Header3;Header4");
foreach (var item in file)
{
var newLine = string.Format("{0},{1},{2},{3}", item.value1, item.value2, item.value3, item.value4);
csv.AppendLine(newLine);
}
//Create Stream
MemoryStream stream = new MemoryStream();
StreamReader reader = new StreamReader(stream);
//Fill your data table here with your values
Hope this will help.

Given string array of column names, how do I read a .csv file to a DataTable?

Assume I have a .csv file with 70 columns, but only 5 of the columns are what I need. I want to be able to pass a method a string array of the columns names that I want, and for it to return a datatable.
private void method(object sender, EventArgs e) {
string[] columns =
{
#"Column21",
#"Column48"
};
DataTable myDataTable = Get_DT(columns);
}
public DataTable Get_DT(string[] columns) {
DataTable ret = new DataTable();
if (columns.Length > 0)
{
foreach (string column in columns)
{
ret.Columns.Add(column);
}
string[] csvlines = File.ReadAllLines(#"path to csv file");
csvlines = csvlines.Skip(1).ToArray(); //ignore the columns in the first line of the csv file
//this is where i need help... i want to use linq to read the fields
//of the each row with only the columns name given in the string[]
//named columns
}
return ret;
}

Read the first line of the file, line.Split(',') (or whatever your delimiter is), then get the index of each column name and store that.
Then for each other line, again do a var values = line.Split(','), then get the values from the columns.
Quick and dirty version:
string[] csvlines = File.ReadAllLines(#"path to csv file");
//select the indices of the columns we want
var cols = csvlines[0].Split(',').Select((val,i) => new { val, i }).Where(x => columns.Any(c => c == x.val)).Select(x => x.i).ToList();
//now go through the remaining lines
foreach (var line in csvlines.Skip(1))
{
var line_values = line.Split(',').ToList();
var dt_values = line_values.Where(x => cols.Contains(line_values.IndexOf(x)));
//now do something with the values you got for this row, add them to your datatable
}

You can look at https://joshclose.github.io/CsvHelper/
Think Reading individual fields is what you are looking for
var csv = new CsvReader( textReader );
while( csv.Read() )
{
var intField = csv.GetField<int>( 0 );
var stringField = csv.GetField<string>( 1 );
var boolField = csv.GetField<bool>( "HeaderName" );
}

We can easily do this without writing much code.
Exceldatareader is an awesome dll for that, it will directly as a datable from the excel sheet with just one method.
here is the links for example:http://www.c-sharpcorner.com/blogs/using-iexceldatareader1
http://exceldatareader.codeplex.com/
Hope it was useful kindly let me know your thoughts or feedbacks
Thanks
Karthik

var data = File.ReadAllLines(#"path to csv file");
// the expenses row
var query = data.Single(d => d[0] == "Expenses");
//third column
int column21 = 3;
return query[column21];

As others have stated a library like CsvReader can be used for this. As for linq, I don't think its suitable for this kind of job.
I haven't tested this but it should get you through
using (TextReader textReader = new StreamReader(filePath))
{
using (var csvReader = new CsvReader(textReader))
{
var headers = csvReader.FieldHeaders;
for (int rowIndex = 0; csvReader.Read(); rowIndex++)
{
var dataRow = dataTable.NewRow();
for (int chosenColumnIndex = 0; chosenColumnIndex < columns.Count(); chosenColumnIndex++)
{
for (int headerIndex = 0; headerIndex < headers.Length; headerIndex++)
{
if (headers[headerIndex] == columns[chosenColumnIndex])
{
dataRow[chosenColumnIndex] = csvReader.GetField<string>(headerIndex);
}
}
}
dataTable.Rows.InsertAt(dataRow, rowIndex);
}
}
}

Why does my ajax call takes so long?

when I make an AJAX call to receive the JSON-object defined in this c# code I have to 46+ seconds to receive this object. The object itself is only 12kb large. Is it because of my c# code(which doesn't take that long to execute?) of is it something else. Im testing it on my localhost IIS server.
This is my code:
[AcceptVerbs(HttpVerbs.Post)]
public JsonResult LoadAudit(int id, string sheet)
{
FactsCustomerRepository<Cateringaudit> repo = new FactsCustomerRepository<Cateringaudit>();
//IQueryable<Cateringaudit> result = repo.GetAll(i => i.State == 1);
Cateringaudit result1 = repo.Get(id);
WorkBook wb = new WorkBook();
wb.read(new MemoryStream(result1.ExcelData));
object[] jsonObjects = new object[3];
//sheetnames to collect data from
string[] sheetNames = { "contract", "proces", "output" };
//itterate trough all sheets in excel file
for (int sheetCount = 0; sheetCount < sheetNames.Length; sheetCount++)
{
wb.Sheet = wb.findSheetByName(sheetNames[sheetCount]);
//Create new array with same lenght as rows with data
Dictionary<string, string[]> excelData = new Dictionary<string, string[]>();
//iterate trough all rows in worksheet
for (int i = 1; i < wb.LastRow + 2; i++)
{
excelData.Add(blabla);
jsonObjects[sheetCount] = excelData;
}
}
return Json(jsonObjects);
}

Going on a limb here: opening Excelsheets in C# is ridiculously slow. There are great libraries out there which are way faster:
EEPLUS: http://epplus.codeplex.com/
But to rule out Excel first: have you tried returning a static JsonObject and not use excel?
[AcceptVerbs(HttpVerbs.Post)]
public JsonResult LoadAudit(int id, string sheet)
{
return Json(); // something like this
}

Should I Convert excel to csv or write htmltable to csv

I have 10 reports in my application which I let users to export to excel. I have never written CSV files. In my existing application, I convert the results from the stored procedure to an HTML table and write it to Excel. Some of my results from the stored procedures have dynamic columns so I use dapper. My new requirement is to provide CSV export as well.
So should I first convert html datatable to excel and convert it to CSV or write HTML datatable to CSV. I dont want to manually parse because there are 10 different reports with different columns and some of the reports have dynamic columns so I cant manually parse.
Stored procs returning Dapper, Dynamic Columns
EFDbContext db = new EFDbContext();
var recordDate = StartDate.Date;
var cnn = new SqlConnection(db.Database.Connection.ConnectionString);
cnn.Open();
var p = new DynamicParameters();
p.Add("#StartDate", StartDate);
p.Add("#UserRoleID", UserRoleID);
p.Add("#SelectedSystemIDs", SelectedSystemIDs);
p.Add("#SelectedPartIDs", SelectedPartIDs);
p.Add("#SelectedSubSystems", SelectedSubsystems);
p.Add("#SelectedServiceTypes", SelectedServiceTypes);
var obs = cnn.Query(sql: "spExportInstrumentConfigAll", param: p, commandType: CommandType.StoredProcedure);
var dt = ToDataTable(obs);
return ExportDatatableToHtml(dt);
public static DataTable ToDataTable(IEnumerable<dynamic> items)
{
if (items == null) return null;
var data = items.ToArray();
if (data.Length == 0) return null;
var dt = new DataTable();
foreach (var pair in ((IDictionary<string, object>)data[0]))
{
dt.Columns.Add(pair.Key, (pair.Value ?? string.Empty).GetType());
}
foreach (var d in data)
{
dt.Rows.Add(((IDictionary<string, object>)d).Values.ToArray());
}
return dt;
}
public static string ExportDatatableToHtml(DataTable dt)
{
StringBuilder strHTMLBuilder = new StringBuilder();
strHTMLBuilder.Append("<html >");
strHTMLBuilder.Append("<head>");
strHTMLBuilder.Append("</head>");
strHTMLBuilder.Append("<body>");
strHTMLBuilder.Append("<table border='1px' cellpadding='1' cellspacing='1 style='font-family:Garamond; font-size:medium'>");
strHTMLBuilder.Append("<tr >");
foreach (DataColumn myColumn in dt.Columns)
{
strHTMLBuilder.Append("<td >");
strHTMLBuilder.Append(myColumn.ColumnName);
strHTMLBuilder.Append("</td>");
}
strHTMLBuilder.Append("</tr>");
foreach (DataRow myRow in dt.Rows)
{
strHTMLBuilder.Append("<tr >");
foreach (DataColumn myColumn in dt.Columns)
{
strHTMLBuilder.Append("<td >");
strHTMLBuilder.Append(myRow[myColumn.ColumnName].ToString());
strHTMLBuilder.Append("</td>");
}
strHTMLBuilder.Append("</tr>");
}
//Close tags.
strHTMLBuilder.Append("</table>");
strHTMLBuilder.Append("</body>");
strHTMLBuilder.Append("</html>");
string Htmltext = strHTMLBuilder.ToString();
return Htmltext;
}
Non-Dynamic Columns mapped to entity
return db.Database.SqlQuery<ServiceEntryPartExportDataRow>("[dbo].[spExportServiceParts] #parm1, #parm2, #parm3, #parm4,#parm5,#parm6",
new SqlParameter("parm1", StartDate),
new SqlParameter("parm2", EndDate),
new SqlParameter("parm3", Reconciled),
new SqlParameter("parm4", ServiceTypes),
new SqlParameter("parm5", SelectedSystemIDs),
new SqlParameter("parm6", UserRoleID)
).ToList().ToHTMLTable();
public static string ToHTMLTable<T>(this IList<T> data)
{
PropertyDescriptorCollection props =
TypeDescriptor.GetProperties(typeof(T));
StringBuilder builder = new StringBuilder();
builder.Append("<table border=\"1\">");
builder.Append("<tr>");
for (int i = 0; i < props.Count; i++)
{
builder.Append("<td>");
PropertyDescriptor prop = props[i];
builder.Append(prop.Name);
builder.Append("</td>");
}
builder.Append("</tr>");
object[] values = new object[props.Count];
foreach (T item in data)
{
builder.Append("<tr>");
for (int i = 0; i < values.Length; i++)
{
builder.Append("<td>");
builder.Append(props[i].GetValue(item));
builder.Append("</td>");
}
builder.Append("</tr>");
}
builder.Append("</table>");
return "<html><body>" + builder.ToString() + "</body></html";
}
Current code Sending to Excel
return new PostActionResult(htmlTable, "ServiceEntryHistory", submit);
public PostActionResult(string htmlTable, string typeName, string submit) { this.htmlTable = htmlTable; this.typeName = typeName; this.submit = submit; }
public PostActionResult(DataTable dataTable, string typeName, string submit) { this.dataTable = dataTable; this.typeName = typeName; this.submit = submit; }
public override void ExecuteResult(ControllerContext context)
{
if (submit == "Excel")
{
ExcelHelpers.ExportToExcel(context.HttpContext, typeName, htmlTable);
}
if (submit == "CSV")
{
ExcelHelpers.ExportToExcelCSV(context.HttpContext, typeName, htmlTable);
}
}
public static void ExportToExcel(HttpContextBase httpBase, string fileNamePrefix, string table)
{
string TimeStamp = DateTime.Now.ToLocalTime().ToString();
string fileName = string.Format("attachment;filename={0}_{1}.xls", fileNamePrefix, TimeStamp);
httpBase.Response.ClearHeaders();
httpBase.Response.ClearContent();
httpBase.Response.Clear();
httpBase.Response.AddHeader("content-disposition", fileName);
httpBase.Response.ContentType = "application/vnd.ms-excel";
httpBase.Response.Write(table);
httpBase.Response.End();
}

You already have code to build an HTML table from the data. Building a CSV is very nearly identical. For brevity, let's simplify the HTML table pseudo-code:
builder.Append("<table>");
// header
builder.Append("<tr>");
foreach (var column in columns)
builder.Append("<th>" + column.name + "</th>");
builder.Append("</tr>");
// rows
foreach (var row in rows)
{
builder.Append("<tr>");
foreach (var column in row.columns)
builder.Append("<td>" + column.value + "</td>");
builder.Append("</tr>");
}
builder.Append("</table>");
Building a CSV is the exact same structure:
// header
foreach (var column in columns)
builder.Append("\"" + column.name + "\",");
// there's now an extra comma at the end. remove it, or use a
// different method to have built the row, such as string.Join.
// rows
foreach (var row in rows)
{
foreach (var column in row.columns)
builder.Append("\"" + column.value + "\",");
// there's now an extra comma at the end. remove it, or use a
// different method to have built the row, such as string.Join.
builder.Append(Environment.NewLine);
}
Remember that this is free-hand pseudo-code, there are some clean-ups you can employ. You might also check the column types to determine if you need those escaped quotes or not, since numeric types wouldn't want them. But the point is that the structure is the same. A CSV is text in the same way that HTML is text. It's only the dressing around the values that's different.
Side note: This is actually a classic example of the Template Method Pattern.

Use Display Name attribute for column header in exported Excel data

I am trying to use a DisplayName attribute to modify the column headers in data exported to an Excel file. I want "My Name" but get "MyName". The data is data returned from a query, converted to a List, then to a DataTable, then exported to Excel.
What else can I try?
There is some discussion here saying that functionality for this was being added.
Using .NET Framework 4
public class MyExcelViewModel
{
private string myName;
//[System.ComponentModel.DisplayName("Mine Name")] // no effect
//[System.ComponentModel.DataAnnotations("My Name")] // doesn't compile
//[System.ComponentModel.DisplayNameAttribute("My Name")] // no effect
[Display(Name = "My Name")] // no effect
public string MyName { get; set; }
}
public class MyController : Controller
// initialize exportList
IList<MyExcelViewModel> exportList = new List<MyExcelViewModel>(fromQueryList.Count);
foreach (var value in fromQueryList)
{
exportList.Add(new MyExcelViewModel());
}
DataTable dataTable = ConvertListToDataTable(exportList); // convert to DataTable
Export_EPPlus(dataTable); // export
private void Export_EPPlus(DataTable dataTable)
{
using (ExcelPackage package = new ExcelPackage())
{
ExcelWorksheet worksheet = package.Workbook.Worksheets.Add(" Results");
// Load the datatable into the sheet, starting from cell A1.
worksheet.Cells["A1"].LoadFromDataTable(dataTable, true);
worksheet.View.FreezePanes(2, 1); // freeze first row & column
worksheet.Row(1).Style.Font.Bold = true;
worksheet.Cells[worksheet.Dimension.Address].AutoFitColumns(); // autofit column width
for (int rowNumber = worksheet.Dimension.Start.Row; rowNumber <= worksheet.Dimension.End.Row; rowNumber++)
{
int rowIndex = rowNumber % 2; // even => 0, odd => 1
ExcelRow excelRow = worksheet.Row(rowNumber);
ExcelFill excelFillRow = excelRow.Style.Fill;
excelFillRow.PatternType = ExcelFillStyle.Solid;
switch (rowIndex)
{
case 0:
excelFillRow.BackgroundColor.SetColor(System.Drawing.Color.White);
break;
case 1:
excelFillRow.BackgroundColor.SetColor(System.Drawing.Color.LightGray);
break;
}
}
try
{
// Write to client
Response.ContentType = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
Response.AddHeader("content-disposition", "attachment; filename="Results-EPPlus.xlsx");
Response.BinaryWrite(package.GetAsByteArray());
}
catch (Exception ex)
{
System.Diagnostics.Trace.WriteLine("MyController.Export_EPPlus(): exception: " + ex);
}
}
}
public static DataTable ConvertListToDataTable<T>(this IList<T> data)
{
PropertyDescriptorCollection props = TypeDescriptor.GetProperties(typeof(T));
DataTable table = new DataTable();
for (int i = 0; i < props.Count; i++)
{
PropertyDescriptor prop = props[i];
table.Columns.Add(prop.Name, prop.PropertyType);
}
object[] values = new object[props.Count];
foreach (T item in data)
{
for (int i = 0; i < values.Length; i++)
{
values[i] = props[i].GetValue(item);
}
table.Rows.Add(values);
}
return table;
}
}

The DisplayName attribute thing only works in 4.0, not with 3.1 unfortunately. The only way to do it in version 3.1 is to manually set the Caption of each DataColumn in the DataTable record, as seen here.
I wrote a blog post on that and other new 4.0 features in practice that can be found here.
As proof, here is the relevant code from EPPlus 3.1 that prints the column headers in the LoadFromDataTable method:
if (PrintHeaders)
{
foreach (DataColumn dataColumn in (InternalDataCollectionBase) Table.Columns)
this._worksheet.Cell(row, num++).Value = (object) dataColumn.Caption;
++row;
num = this._fromCol;
}