I am trying to write into a csv file row by row using C# language. Here is my function
string first = reader[0].ToString();
string second=image.ToString();
string csv = string.Format("{0},{1}\n", first, second);
File.WriteAllText(filePath, csv);
The whole function runs inside a loop, and every row should be written to the csv file. In my case, next row overwrites the existing row and in the end, I am getting an only single record in the csv file which is the last one. How can I write all the rows in the csv file?
UPDATE
Back in my naïve days, I suggested doing this manually (it was a simple solution to a simple question), however due to this becoming more and more popular, I'd recommend using the library CsvHelper that does all the safety checks, etc.
CSV is way more complicated than what the question/answer suggests.
Original Answer
As you already have a loop, consider doing it like this:
//before your loop
var csv = new StringBuilder();
//in your loop
var first = reader[0].ToString();
var second = image.ToString();
//Suggestion made by KyleMit
var newLine = string.Format("{0},{1}", first, second);
csv.AppendLine(newLine);
//after your loop
File.WriteAllText(filePath, csv.ToString());
Or something to this effect.
My reasoning is: you won't be need to write to the file for every item, you will only be opening the stream once and then writing to it.
You can replace
File.WriteAllText(filePath, csv.ToString());
with
File.AppendAllText(filePath, csv.ToString());
if you want to keep previous versions of csv in the same file
C# 6
If you are using c# 6.0 then you can do the following
var newLine = $"{first},{second}"
EDIT
Here is a link to a question that explains what Environment.NewLine does.
I would highly recommend you to go the more tedious route. Especially if your file size is large.
using(var w = new StreamWriter(path))
{
for( /* your loop */)
{
var first = yourFnToGetFirst();
var second = yourFnToGetSecond();
var line = string.Format("{0},{1}", first, second);
w.WriteLine(line);
w.Flush();
}
}
File.AppendAllText() opens a new file, writes the content and then closes the file. Opening files is a much resource-heavy operation, than writing data into open stream. Opening\closing a file inside a loop will cause performance drop.
The approach suggested by Johan solves that problem by storing all the output in memory and then writing it once. However (in case of big files) you program will consume a large amount of RAM and even crash with OutOfMemoryException
Another advantage of my solution is that you can implement pausing\resuming by saving current position in input data.
upd. Placed using in the right place
Writing csv files by hand can be difficult because your data might contain commas and newlines. I suggest you use an existing library instead.
This question mentions a few options.
Are there any CSV readers/writer libraries in C#?
I use a two parse solution as it's very easy to maintain
// Prepare the values
var allLines = (from trade in proposedTrades
select new object[]
{
trade.TradeType.ToString(),
trade.AccountReference,
trade.SecurityCodeType.ToString(),
trade.SecurityCode,
trade.ClientReference,
trade.TradeCurrency,
trade.AmountDenomination.ToString(),
trade.Amount,
trade.Units,
trade.Percentage,
trade.SettlementCurrency,
trade.FOP,
trade.ClientSettlementAccount,
string.Format("\"{0}\"", trade.Notes),
}).ToList();
// Build the file content
var csv = new StringBuilder();
allLines.ForEach(line =>
{
csv.AppendLine(string.Join(",", line));
});
File.WriteAllText(filePath, csv.ToString());
Instead of calling every time AppendAllText() you could think about opening the file once and then write the whole content once:
var file = #"C:\myOutput.csv";
using (var stream = File.CreateText(file))
{
for (int i = 0; i < reader.Count(); i++)
{
string first = reader[i].ToString();
string second = image.ToString();
string csvRow = string.Format("{0},{1}", first, second);
stream.WriteLine(csvRow);
}
}
You can use AppendAllText instead:
File.AppendAllText(filePath, csv);
As the documentation of WriteAllText says:
If the target file already exists, it is overwritten
Also, note that your current code is not using proper new lines, for example in Notepad you'll see it all as one long line. Change the code to this to have proper new lines:
string csv = string.Format("{0},{1}{2}", first, image, Environment.NewLine);
Instead of reinventing the wheel a library could be used. CsvHelper is great for creating and reading csv files. It's read and write operations are stream based and therefore also support operations with a big amount of data.
You can write your csv like the following.
using(var textWriter = new StreamWriter(#"C:\mypath\myfile.csv"))
{
var writer = new CsvWriter(textWriter, CultureInfo.InvariantCulture);
writer.Configuration.Delimiter = ",";
foreach (var item in list)
{
writer.WriteField( "a" );
writer.WriteField( 2 );
writer.WriteField( true );
writer.NextRecord();
}
}
As the library is using reflection it will take any type and parse it directly.
public class CsvRow
{
public string Column1 { get; set; }
public bool Column2 { get; set; }
public CsvRow(string column1, bool column2)
{
Column1 = column1;
Column2 = column2;
}
}
IEnumerable<CsvRow> rows = new [] {
new CsvRow("value1", true),
new CsvRow("value2", false)
};
using(var textWriter = new StreamWriter(#"C:\mypath\myfile.csv")
{
var writer = new CsvWriter(textWriter, CultureInfo.InvariantCulture);
writer.Configuration.Delimiter = ",";
writer.WriteRecords(rows);
}
value1,true
value2,false
If you want to read more about the librarys configurations and possibilities you can do so here.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Data;
using System.Configuration;
using System.Data.SqlClient;
public partial class CS : System.Web.UI.Page
{
protected void ExportCSV(object sender, EventArgs e)
{
string constr = ConfigurationManager.ConnectionStrings["constr"].ConnectionString;
using (SqlConnection con = new SqlConnection(constr))
{
using (SqlCommand cmd = new SqlCommand("SELECT * FROM Customers"))
{
using (SqlDataAdapter sda = new SqlDataAdapter())
{
cmd.Connection = con;
sda.SelectCommand = cmd;
using (DataTable dt = new DataTable())
{
sda.Fill(dt);
//Build the CSV file data as a Comma separated string.
string csv = string.Empty;
foreach (DataColumn column in dt.Columns)
{
//Add the Header row for CSV file.
csv += column.ColumnName + ',';
}
//Add new line.
csv += "\r\n";
foreach (DataRow row in dt.Rows)
{
foreach (DataColumn column in dt.Columns)
{
//Add the Data rows.
csv += row[column.ColumnName].ToString().Replace(",", ";") + ',';
}
//Add new line.
csv += "\r\n";
}
//Download the CSV file.
Response.Clear();
Response.Buffer = true;
Response.AddHeader("content-disposition", "attachment;filename=SqlExport.csv");
Response.Charset = "";
Response.ContentType = "application/text";
Response.Output.Write(csv);
Response.Flush();
Response.End();
}
}
}
}
}
}
Handling Commas
For handling commas inside of values when using string.Format(...), the following has worked for me:
var newLine = string.Format("\"{0}\",\"{1}\",\"{2}\"",
first,
second,
third
);
csv.AppendLine(newLine);
So to combine it with Johan's answer, it'd look like this:
//before your loop
var csv = new StringBuilder();
//in your loop
var first = reader[0].ToString();
var second = image.ToString();
//Suggestion made by KyleMit
var newLine = string.Format("\"{0}\",\"{1}\"", first, second);
csv.AppendLine(newLine);
//after your loop
File.WriteAllText(filePath, csv.ToString());
Returning CSV File
If you simply wanted to return the file instead of writing it to a location, this is an example of how I accomplished it:
From a Stored Procedure
public FileContentResults DownloadCSV()
{
// I have a stored procedure that queries the information I need
SqlConnection thisConnection = new SqlConnection("Data Source=sv12sql;User ID=UI_Readonly;Password=SuperSecure;Initial Catalog=DB_Name;Integrated Security=false");
SqlCommand queryCommand = new SqlCommand("spc_GetInfoINeed", thisConnection);
queryCommand.CommandType = CommandType.StoredProcedure;
StringBuilder sbRtn = new StringBuilder();
// If you want headers for your file
var header = string.Format("\"{0}\",\"{1}\",\"{2}\"",
"Name",
"Address",
"Phone Number"
);
sbRtn.AppendLine(header);
// Open Database Connection
thisConnection.Open();
using (SqlDataReader rdr = queryCommand.ExecuteReader())
{
while (rdr.Read())
{
// rdr["COLUMN NAME"].ToString();
var queryResults = string.Format("\"{0}\",\"{1}\",\"{2}\"",
rdr["Name"].ToString(),
rdr["Address"}.ToString(),
rdr["Phone Number"].ToString()
);
sbRtn.AppendLine(queryResults);
}
}
thisConnection.Close();
return File(new System.Text.UTF8Encoding().GetBytes(sbRtn.ToString()), "text/csv", "FileName.csv");
}
From a List
/* To help illustrate */
public static List<Person> list = new List<Person>();
/* To help illustrate */
public class Person
{
public string name;
public string address;
public string phoneNumber;
}
/* The important part */
public FileContentResults DownloadCSV()
{
StringBuilder sbRtn = new StringBuilder();
// If you want headers for your file
var header = string.Format("\"{0}\",\"{1}\",\"{2}\"",
"Name",
"Address",
"Phone Number"
);
sbRtn.AppendLine(header);
foreach (var item in list)
{
var listResults = string.Format("\"{0}\",\"{1}\",\"{2}\"",
item.name,
item.address,
item.phoneNumber
);
sbRtn.AppendLine(listResults);
}
}
return File(new System.Text.UTF8Encoding().GetBytes(sbRtn.ToString()), "text/csv", "FileName.csv");
}
Hopefully this is helpful.
This is a simple tutorial on creating csv files using C# that you will be able to edit and expand on to fit your own needs.
First you’ll need to create a new Visual Studio C# console application, there are steps to follow to do this.
The example code will create a csv file called MyTest.csv in the location you specify. The contents of the file should be 3 named columns with text in the first 3 rows.
https://tidbytez.com/2018/02/06/how-to-create-a-csv-file-with-c/
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO;
namespace CreateCsv
{
class Program
{
static void Main()
{
// Set the path and filename variable "path", filename being MyTest.csv in this example.
// Change SomeGuy for your username.
string path = #"C:\Users\SomeGuy\Desktop\MyTest.csv";
// Set the variable "delimiter" to ", ".
string delimiter = ", ";
// This text is added only once to the file.
if (!File.Exists(path))
{
// Create a file to write to.
string createText = "Column 1 Name" + delimiter + "Column 2 Name" + delimiter + "Column 3 Name" + delimiter + Environment.NewLine;
File.WriteAllText(path, createText);
}
// This text is always added, making the file longer over time
// if it is not deleted.
string appendText = "This is text for Column 1" + delimiter + "This is text for Column 2" + delimiter + "This is text for Column 3" + delimiter + Environment.NewLine;
File.AppendAllText(path, appendText);
// Open the file to read from.
string readText = File.ReadAllText(path);
Console.WriteLine(readText);
}
}
}
public static class Extensions
{
public static void WriteCSVLine(this StreamWriter writer, IEnumerable<string> fields)
{
const string q = #"""";
writer.WriteLine(string.Join(",",
fields.Select(
v => (v.Contains(',') || v.Contains('"') || v.Contains('\n') || v.Contains('\r')) ? $"{q}{v.Replace(q, q + q)}{q}" : v
)));
}
public static void WriteCSVLine(this StreamWriter writer, params string[] fields) => WriteCSVLine(writer, (IEnumerable<string>)fields);
}
This should allow you to write a csv file quite simply. Usage:
StreamWriter writer = new ("myfile.csv");
writer.WriteCSVLine("A", "B"); // A,B
Here is another open source library to create CSV file easily, Cinchoo ETL
List<dynamic> objs = new List<dynamic>();
dynamic rec1 = new ExpandoObject();
rec1.Id = 10;
rec1.Name = #"Mark";
rec1.JoinedDate = new DateTime(2001, 2, 2);
rec1.IsActive = true;
rec1.Salary = new ChoCurrency(100000);
objs.Add(rec1);
dynamic rec2 = new ExpandoObject();
rec2.Id = 200;
rec2.Name = "Tom";
rec2.JoinedDate = new DateTime(1990, 10, 23);
rec2.IsActive = false;
rec2.Salary = new ChoCurrency(150000);
objs.Add(rec2);
using (var parser = new ChoCSVWriter("emp.csv").WithFirstLineHeader())
{
parser.Write(objs);
}
For more information, please read the CodeProject article on usage.
One simple way to get rid of the overwriting issue is to use File.AppendText to append line at the end of the file as
void Main()
{
using (System.IO.StreamWriter sw = System.IO.File.AppendText("file.txt"))
{
string first = reader[0].ToString();
string second=image.ToString();
string csv = string.Format("{0},{1}\n", first, second);
sw.WriteLine(csv);
}
}
enter code here
string string_value= string.Empty;
for (int i = 0; i < ur_grid.Rows.Count; i++)
{
for (int j = 0; j < ur_grid.Rows[i].Cells.Count; j++)
{
if (!string.IsNullOrEmpty(ur_grid.Rows[i].Cells[j].Text.ToString()))
{
if (j > 0)
string_value= string_value+ "," + ur_grid.Rows[i].Cells[j].Text.ToString();
else
{
if (string.IsNullOrEmpty(string_value))
string_value= ur_grid.Rows[i].Cells[j].Text.ToString();
else
string_value= string_value+ Environment.NewLine + ur_grid.Rows[i].Cells[j].Text.ToString();
}
}
}
}
string where_to_save_file = #"d:\location\Files\sample.csv";
File.WriteAllText(where_to_save_file, string_value);
string server_path = "/site/Files/sample.csv";
Response.ContentType = ContentType;
Response.AppendHeader("Content-Disposition", "attachment; filename=" + Path.GetFileName(server_path));
Response.WriteFile(server_path);
Response.End();
You might just have to add a line feed "\n\r".
Hi I have a problem with importing a csv file into a sql server, this csv file contains articles that need to be saved in the sql server database. Once the import (done with the code c # written below) is finished, some fields imported as (Descrizione and CodArt) are not written correctly in the database and have strange characters. To download the csv file click here.
SqlServer improper import over blue line:
Import C# Code:
using (var rd = new StreamReader(labelPercorso.Text))
{
Articolo a = new Articolo();
a.db = this.db;
while (!rd.EndOfStream)
{
//setto codean e immagine =null ad ogni giro
CodEAN = "";
Immagine = "";
try
{
var splits = rd.ReadLine().Split(';');
CodArt = splits[0];
Descrizione = splits[1];
String Price = splits[2];
Prezzo = decimal.Parse(Price);
}
catch (Exception ex)
{
Console.WriteLine("Non è presente nè immagine nè codean");
}
a.Prezzo = Prezzo;
a.CodiceArticolo = CodArt;
a.Descrizione = Descrizione;
a.Fornitore = fornitore;
//manca da controllare se l'articolo è presente e nel caso aggiornalo
a.InserisciArticoloCSV();
}
}
Code of function: InserisciArticoloCSV
try
{
SqlConnection conn = db.apriconnessione();
String query = "INSERT INTO Articolo(CodArt,Descrizione,Prezzo,PrezzoListino,Fornitore,Importato,TipoArticolo) VALUES(#CodArt,#Descrizione,#Prezzo,#PrezzoListino,#Fornitore,#Importato,#TipoArticolo)";
String Importato = "CSV";
String TipoArticolo = "A";
SqlCommand cmd = new SqlCommand(query, conn);
// MessageBox.Show("CodArt: " + CodiceArticolo + "\n Descrizione :" + Descrizione + "\n Prezzo: " + Prezzo);
cmd.Parameters.AddWithValue("#CodArt", CodiceArticolo.ToString());
cmd.Parameters.AddWithValue("#Descrizione", Descrizione.ToString());
cmd.Parameters.AddWithValue("#Prezzo", Prezzo);
cmd.Parameters.AddWithValue("#PrezzoListino", Prezzo);
cmd.Parameters.AddWithValue("#Fornitore", Fornitore.ToString());
cmd.Parameters.AddWithValue("#Importato", Importato.ToString());
cmd.Parameters.AddWithValue("#TipoArticolo", TipoArticolo.ToString());
cmd.ExecuteNonQuery();
db.chiudiconnessione();
conn.Close();
return true;
}
catch (Exception ex)
{
Console.WriteLine("Errore nell'inserimento dell'articolo " + ex);
//MessageBox.Show("Errore nel inserimento dell'articolo: " + ex);
return false;
}
Your CSV file is not well formated , there are intermediatory Carriage Returns in between , which screws up the parsing. See the file in Notepad++ and turn on the Line Breaks , this is what you find.
So for the lines which are in format the data import is working fine , for others the logic is not working.
As others have pointed out, you have numerous problems, encoding, carriage returns and a lot of white space. In addition you are using single inserts into your database, which is very slow. I show below some sample code, which illustrates how to deal with all of these points.
IFormatProvider fP = new CultureInfo("it");
DataTable tmp = new DataTable();
tmp.Columns.Add("CodArt", typeof(string));
tmp.Columns.Add("Descrizione", typeof(string));
tmp.Columns.Add("Prezzo", typeof(decimal));
using (var rd = new StreamReader("yourFileName", Encoding.GetEncoding("iso-8859-1")))
{
while (!rd.EndOfStream)
{
try
{
var nextLine = Regex.Replace(rd.ReadLine(), #"\s+", " ");
while (nextLine.Split(';').Length < 3)
{
nextLine = nextLine.Replace("\r\n", "") + Regex.Replace(rd.ReadLine(), #"\s+", " ");
}
var splits = nextLine.Split(';');
DataRow dR = tmp.NewRow();
dR[0] = splits[0];
dR[1] = splits[1];
string Price = splits[2];
dR[2] = decimal.Parse(Price, fP);
tmp.Rows.Add(dR);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
}
using (var conn = db.apriconnessione())
{
var sBC = new SqlBulkCopy(conn);
conn.Open();
sBC.DestinationTableName = "yourTableName";
sBC.WriteToServer(tmp);
conn.Close();
}
Now for some explanation:
Firstly I am storing the parsed values in a DataTable. Please note that I have only included the three fields that are in the CSV. In practice you must supply the other columns and fill the extra columns with the correct values for each row. I was simply being lazy, but I am sure you will get the idea.
I do not know what encoding your csv file is, but iso-8859-1 worked for me!
I use Regex to replace multiple white space with a single space.
If any line does not have the required number of splits, I keep adding further lines (having deleted the carriage return) until I hit success!
Once I have a complete line, I can now split it, and assign it to the new DataRow (please see my comments above for extra columns).
Finally once the file has been read, the DataTable will have all the rows and can be uploaded to your database using BulkCopy. This is very fast!
HTH
PS Some of your lines have double quotes. You probably want to get rid of these as well!
You should specify the correct encoding when you read your file. Is it utf? Is it ascii with a specific code page? You should also specify the SqlDbType of your Sql parameters, especially the string parameters which will be either varchar or nvarchar and there is a big difference between them.
// what is the encoding of your file? This is an example using code page windows-1252
var encoding = Encoding.GetEncoding("windows-1252");
using (var file = File.Open(labelPercorso.Text, FileMode.Open))
using (var reader = new StreamReader(file, encoding))
{
// rest of code unchanged
}
Sql Code. Note that I added using blocks for the types that implement IDisposable like Connection and Command.
try
{
String query = "INSERT INTO Articolo(CodArt,Descrizione,Prezzo,PrezzoListino,Fornitore,Importato,TipoArticolo) VALUES(#CodArt,#Descrizione,#Prezzo,#PrezzoListino,#Fornitore,#Importato,#TipoArticolo)";
String Importato = "CSV";
String TipoArticolo = "A";
using(SqlConnection conn = db.apriconnessione())
using(SqlCommand cmd = new SqlCommand(query, conn))
{
// -1 indicates you used MAX like nvarchar(max), otherwise use the maximum number of characters in the schema
cmd.Parameters.Add(new SqlDbParameter("#CodArt", SqlDbType.NVarChar, -1)).Value = CodiceArticolo.ToString();
cmd.Parameters.Add(new SqlDbParameter("#Descrizione", SqlDbType.NVarChar, -1)).Value = Descrizione.ToString();
/*
Rest of your parameters created in the same manner
*/
cmd.ExecuteNonQuery();
db.chiudiconnessione();
}
return true;
}
catch (Exception ex)
{
Console.WriteLine("Errore nell'inserimento dell'articolo " + ex);
//MessageBox.Show("Errore nel inserimento dell'articolo: " + ex);
return false;
}
Just in case if you are interested in exploring library to handle all parsing needs with few lines of code, you can check out the Cinchoo ETL - an open source library. Here is sample to parse the csv file and shows how to get either datatable or list of records for later to load them to database.
System.Threading.Thread.CurrentThread.CurrentCulture = new CultureInfo("it");
using (var p = new ChoCSVReader("Bosch Luglio 2017.csv")
.Configure((c) => c.MayContainEOLInData = true) //Handle newline chars in data
.Configure(c => c.Encoding = Encoding.GetEncoding("iso-8859-1")) //Specify the encoding for reading
.WithField("CodArt", 1) //first column
.WithField("Descrizione", 2) //second column
.WithField("Prezzo", 3, fieldType: typeof(decimal)) //third column
.Setup(c => c.BeforeRecordLoad += (o, e) =>
{
e.Source = e.Source.CastTo<string>().Replace(#"""", String.Empty); //Remove the quotes
}) //Scrub the data
)
{
var dt = p.AsDataTable();
//foreach (var rec in p)
// Console.WriteLine(rec.Prezzo);
}
Disclaimer: I'm the author of this library.
I am using below code to export data from a csv file to datatable.
As the values are of mixed text i.e. both numbers and Alphabets, some of the columns are not getting exported to Datatable.
I have done some research here and found that we need to set ImportMixedType = Text and TypeGuessRows = 0 in registry which even did not solve the problem.
Below code is working for some files even with mixed text.
Could someone tell me what is wrong with below code. Do I miss some thing here.
if (isFirstRowHeader)
{
header = "Yes";
}
using (OleDbConnection connection = new OleDbConnection(#"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + pathOnly +
";Extended Properties=\"text;HDR=" + header + ";FMT=Delimited\";"))
{
using (OleDbCommand command = new OleDbCommand(sql, connection))
{
using (OleDbDataAdapter adapter = new OleDbDataAdapter(command))
{
adapter.Fill(table);
}
connection.Close();
}
}
for comma delimited file this worked for me
public DataTable CSVtoDataTable(string inputpath)
{
DataTable csvdt = new DataTable();
string Fulltext;
if (File.Exists(inputpath))
{
using (StreamReader sr = new StreamReader(inputpath))
{
while (!sr.EndOfStream)
{
Fulltext = sr.ReadToEnd().ToString();//read full content
string[] rows = Fulltext.Split('\n');//split file content to get the rows
for (int i = 0; i < rows.Count() - 1; i++)
{
var regex = new Regex("\\\"(.*?)\\\"");
var output = regex.Replace(rows[i], m => m.Value.Replace(",", "\\c"));//replace commas inside quotes
string[] rowValues = output.Split(',');//split rows with comma',' to get the column values
{
if (i == 0)
{
for (int j = 0; j < rowValues.Count(); j++)
{
csvdt.Columns.Add(rowValues[j].Replace("\\c",","));//headers
}
}
else
{
try
{
DataRow dr = csvdt.NewRow();
for (int k = 0; k < rowValues.Count(); k++)
{
if (k >= dr.Table.Columns.Count)// more columns may exist
{ csvdt .Columns.Add("clmn" + k);
dr = csvdt .NewRow();
}
dr[k] = rowValues[k].Replace("\\c", ",");
}
csvdt.Rows.Add(dr);//add other rows
}
catch
{
Console.WriteLine("error");
}
}
}
}
}
}
}
return csvdt;
}
The main thing that would probably help is to first stop using OleDB objects for reading a delimited file. I suggest using the 'TextFieldParser' which is what I have successfully used for over 2 years now for a client.
http://www.dotnetperls.com/textfieldparser
There may be other issues, but without seeing your .CSV file, I can't tell you where your problem may lie.
The TextFieldParser is specifically designed to parse comma delimited files. The OleDb objects are not. So, start there and then we can determine what the problem may be, if it persists.
If you look at an example on the link I provided, they are merely writing lines to the console. You can alter this code portion to add rows to a DataTable object, as I do, for sorting purposes.
I have a Excel File (xls) that has a column called Money. In the Money column all the columns are formatted as number, except for some that have that marker saying formatted as text against them. I convert the Excel file to CSV using a c# script that uses IMEX=1 in the connection string to open it. The fields that are marked with stored as text do not come through to the csv file. The file is large, about 20MB. So this means 100 values like 33344 etc do not come thro the csv file.
I tried to put a delay in where I open the Excel File. This worked on my PC but not the Development machine.
Have any idea how to get round this without manually intervention, like format all columns with mixed data types as number etc ? I am looking for an automated solution that works every time . This is on SSIS 2008.
static void ConvertExcelToCsv(string excelFilePath, string csvOutputFile, int worksheetNumber = 1) {
if (!File.Exists(excelFilePath)) throw new FileNotFoundException(excelFilePath);
if (File.Exists(csvOutputFile)) throw new ArgumentException("File exists: " + csvOutputFile);
// connection string
var cnnStr = String.Format("Provider=Microsoft.Jet.OLEDB.4.0;Data Source={0};Extended Properties=\"Excel 8.0;IMEX=1;HDR=NO\"", excelFilePath);
var cnn = new OleDbConnection(cnnStr);
// get schema, then data
var dt = new DataTable();
try {
cnn.Open();
var schemaTable = cnn.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, null);
if (schemaTable.Rows.Count < worksheetNumber) throw new ArgumentException("The worksheet number provided cannot be found in the spreadsheet");
string worksheet = schemaTable.Rows[worksheetNumber - 1]["table_name"].ToString().Replace("'", "");
string sql = String.Format("select * from [{0}]", worksheet);
var da = new OleDbDataAdapter(sql, cnn);
da.Fill(dt);
}
catch (Exception e) {
// ???
throw e;
}
finally {
// free resources
cnn.Close();
}
// write out CSV data
using (var wtr = new StreamWriter(csvOutputFile)) {
foreach (DataRow row in dt.Rows) {
bool firstLine = true;
foreach (DataColumn col in dt.Columns) {
if (!firstLine) { wtr.Write(","); } else { firstLine = false; }
var data = row[col.ColumnName].ToString().Replace("\"", "\"\"");
wtr.Write(String.Format("\"{0}\"", data));
}
wtr.WriteLine();
}
}
}
My solution was to specify a format for the incoming files which said no columns with mixed data types. Solution was from business side and not technology.
This has been killing me - I have a massive file that I need to read in as a DataTable.
After a lot of messing about I am using this:
using (OleDbConnection connection = new OleDbConnection(connString))
{
using (OleDbCommand command = new OleDbCommand(sql, connection))
{
using (OleDbDataAdapter adapter = new OleDbDataAdapter(command))
{
dataTable = new DataTable();
dataTable.Locale = CultureInfo.CurrentCulture;
adapter.Fill(dataTable);
}
}
}
which works if the text file is comma seperated but does not work if it is tab delimited - Can anyone please help??
My connection string looks like :
string connString = #"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + pathOnly + #";Extended Properties='text;HDR=YES'";
I ve tried to set the FMT property with no luck....
Here is a good library use it.
http://www.codeproject.com/KB/database/CsvReader.aspx
Here is the code which use the library.
TextReader tr = new StreamReader(HttpContext.Current.Server.MapPath(Filename));
string data = tr.ReadToEnd();
tr.Close();
for comma delimited;
CachedCsvReader cr = new CachedCsvReader(new StringReader(csv), true);
for tab delimited;
CachedCsvReader cr = new CachedCsvReader(new StringReader(csv), true, '\t');
And here you can load it into DataTable by having this code
DataTable dt = new DataTable();
dt.Load(cr);
Hope you find it helpful. Thanks
Manually: You can use String.Split() method to split your entire file. Here is an example of what i use in my code. In this example, i read the data line by line and split it. I then place the data directly into columns.
//Open and read the file
System.IO.FileStream fs = new System.IO.FileStream("myfilename", System.IO.FileMode.Open);
System.IO.StreamReader sr = new System.IO.StreamReader(fs);
string line = "";
line = sr.ReadLine();
string[] colVal;
try
{
//clear of previous data
//myDataTable.Clear();
//for each reccord insert it into a row
while (!sr.EndOfStream)
{
line = sr.ReadLine();
colVal = line.Split('\t');
DataRow dataRow = myDataTable.NewRow();
//associate values with the columns
dataRow["col1"] = colVal[0];
dataRow["col2"] = colVal[1];
dataRow["col3"] = colVal[2];
//add the row to the table
myDataTable.Rows.Add(dataRow);
}
//close the stream
sr.Close();
//binds the dataset tothe grid view.
BindingSource bs = new BindingSource();
bs.DataSource = myDataSet;
bs.DataMember = myDataTable.TableName;
myGridView.DataSource = bs;
}
You could modify it to do some loops for columns if you have many and they are numbered. Also, i recommend checking the integrity first, by checking that the number of columns read is correct.
This should work: (from http://www.hotblue.com/article0000.aspx?a=0006)
just replace the comma part with:
if ((postdata || !quoted) && (c == ',' || c == '\t'))
to make it tab delimited.
using System.Data;
using System.IO;
using System.Text.RegularExpressions;
public DataTable ParseCSV(string inputString) {
DataTable dt=new DataTable();
// declare the Regular Expression that will match versus the input string
Regex re=new Regex("((?<field>[^\",\\r\\n]+)|\"(?<field>([^\"]|\"\")+)\")(,|(?<rowbreak>\\r\\n|\\n|$))");
ArrayList colArray=new ArrayList();
ArrayList rowArray=new ArrayList();
int colCount=0;
int maxColCount=0;
string rowbreak="";
string field="";
MatchCollection mc=re.Matches(inputString);
foreach(Match m in mc) {
// retrieve the field and replace two double-quotes with a single double-quote
field=m.Result("${field}").Replace("\"\"","\"");
rowbreak=m.Result("${rowbreak}");
if (field.Length > 0) {
colArray.Add(field);
colCount++;
}
if (rowbreak.Length > 0) {
// add the column array to the row Array List
rowArray.Add(colArray.ToArray());
// create a new Array List to hold the field values
colArray=new ArrayList();
if (colCount > maxColCount)
maxColCount=colCount;
colCount=0;
}
}
if (rowbreak.Length == 0) {
// this is executed when the last line doesn't
// end with a line break
rowArray.Add(colArray.ToArray());
if (colCount > maxColCount)
maxColCount=colCount;
}
// create the columns for the table
for(int i=0; i < maxColCount; i++)
dt.Columns.Add(String.Format("col{0:000}",i));
// convert the row Array List into an Array object for easier access
Array ra=rowArray.ToArray();
for(int i=0; i < ra.Length; i++) {
// create a new DataRow
DataRow dr=dt.NewRow();
// convert the column Array List into an Array object for easier access
Array ca=(Array)(ra.GetValue(i));
// add each field into the new DataRow
for(int j=0; j < ca.Length; j++)
dr[j]=ca.GetValue(j);
// add the new DataRow to the DataTable
dt.Rows.Add(dr);
}
// in case no data was parsed, create a single column
if (dt.Columns.Count == 0)
dt.Columns.Add("NoData");
return dt;
}
Now that we have a parser for converting a string into a DataTable, all we need now is a function that will read the content from a CSV file and pass it to our ParseCSV function:
public DataTable ParseCSVFile(string path) {
string inputString="";
// check that the file exists before opening it
if (File.Exists(path)) {
StreamReader sr = new StreamReader(path);
inputString = sr.ReadToEnd();
sr.Close();
}
return ParseCSV(inputString);
}
And now you can easily fill a DataGrid with data coming off the CSV file:
protected System.Web.UI.WebControls.DataGrid DataGrid1;
private void Page_Load(object sender, System.EventArgs e) {
// call the parser
DataTable dt=ParseCSVFile(Server.MapPath("./demo.csv"));
// bind the resulting DataTable to a DataGrid Web Control
DataGrid1.DataSource=dt;
DataGrid1.DataBind();
}