Related
I am trying to write into a csv file row by row using C# language. Here is my function
string first = reader[0].ToString();
string second=image.ToString();
string csv = string.Format("{0},{1}\n", first, second);
File.WriteAllText(filePath, csv);
The whole function runs inside a loop, and every row should be written to the csv file. In my case, next row overwrites the existing row and in the end, I am getting an only single record in the csv file which is the last one. How can I write all the rows in the csv file?
UPDATE
Back in my naïve days, I suggested doing this manually (it was a simple solution to a simple question), however due to this becoming more and more popular, I'd recommend using the library CsvHelper that does all the safety checks, etc.
CSV is way more complicated than what the question/answer suggests.
Original Answer
As you already have a loop, consider doing it like this:
//before your loop
var csv = new StringBuilder();
//in your loop
var first = reader[0].ToString();
var second = image.ToString();
//Suggestion made by KyleMit
var newLine = string.Format("{0},{1}", first, second);
csv.AppendLine(newLine);
//after your loop
File.WriteAllText(filePath, csv.ToString());
Or something to this effect.
My reasoning is: you won't be need to write to the file for every item, you will only be opening the stream once and then writing to it.
You can replace
File.WriteAllText(filePath, csv.ToString());
with
File.AppendAllText(filePath, csv.ToString());
if you want to keep previous versions of csv in the same file
C# 6
If you are using c# 6.0 then you can do the following
var newLine = $"{first},{second}"
EDIT
Here is a link to a question that explains what Environment.NewLine does.
I would highly recommend you to go the more tedious route. Especially if your file size is large.
using(var w = new StreamWriter(path))
{
for( /* your loop */)
{
var first = yourFnToGetFirst();
var second = yourFnToGetSecond();
var line = string.Format("{0},{1}", first, second);
w.WriteLine(line);
w.Flush();
}
}
File.AppendAllText() opens a new file, writes the content and then closes the file. Opening files is a much resource-heavy operation, than writing data into open stream. Opening\closing a file inside a loop will cause performance drop.
The approach suggested by Johan solves that problem by storing all the output in memory and then writing it once. However (in case of big files) you program will consume a large amount of RAM and even crash with OutOfMemoryException
Another advantage of my solution is that you can implement pausing\resuming by saving current position in input data.
upd. Placed using in the right place
Writing csv files by hand can be difficult because your data might contain commas and newlines. I suggest you use an existing library instead.
This question mentions a few options.
Are there any CSV readers/writer libraries in C#?
I use a two parse solution as it's very easy to maintain
// Prepare the values
var allLines = (from trade in proposedTrades
select new object[]
{
trade.TradeType.ToString(),
trade.AccountReference,
trade.SecurityCodeType.ToString(),
trade.SecurityCode,
trade.ClientReference,
trade.TradeCurrency,
trade.AmountDenomination.ToString(),
trade.Amount,
trade.Units,
trade.Percentage,
trade.SettlementCurrency,
trade.FOP,
trade.ClientSettlementAccount,
string.Format("\"{0}\"", trade.Notes),
}).ToList();
// Build the file content
var csv = new StringBuilder();
allLines.ForEach(line =>
{
csv.AppendLine(string.Join(",", line));
});
File.WriteAllText(filePath, csv.ToString());
Instead of calling every time AppendAllText() you could think about opening the file once and then write the whole content once:
var file = #"C:\myOutput.csv";
using (var stream = File.CreateText(file))
{
for (int i = 0; i < reader.Count(); i++)
{
string first = reader[i].ToString();
string second = image.ToString();
string csvRow = string.Format("{0},{1}", first, second);
stream.WriteLine(csvRow);
}
}
You can use AppendAllText instead:
File.AppendAllText(filePath, csv);
As the documentation of WriteAllText says:
If the target file already exists, it is overwritten
Also, note that your current code is not using proper new lines, for example in Notepad you'll see it all as one long line. Change the code to this to have proper new lines:
string csv = string.Format("{0},{1}{2}", first, image, Environment.NewLine);
Instead of reinventing the wheel a library could be used. CsvHelper is great for creating and reading csv files. It's read and write operations are stream based and therefore also support operations with a big amount of data.
You can write your csv like the following.
using(var textWriter = new StreamWriter(#"C:\mypath\myfile.csv"))
{
var writer = new CsvWriter(textWriter, CultureInfo.InvariantCulture);
writer.Configuration.Delimiter = ",";
foreach (var item in list)
{
writer.WriteField( "a" );
writer.WriteField( 2 );
writer.WriteField( true );
writer.NextRecord();
}
}
As the library is using reflection it will take any type and parse it directly.
public class CsvRow
{
public string Column1 { get; set; }
public bool Column2 { get; set; }
public CsvRow(string column1, bool column2)
{
Column1 = column1;
Column2 = column2;
}
}
IEnumerable<CsvRow> rows = new [] {
new CsvRow("value1", true),
new CsvRow("value2", false)
};
using(var textWriter = new StreamWriter(#"C:\mypath\myfile.csv")
{
var writer = new CsvWriter(textWriter, CultureInfo.InvariantCulture);
writer.Configuration.Delimiter = ",";
writer.WriteRecords(rows);
}
value1,true
value2,false
If you want to read more about the librarys configurations and possibilities you can do so here.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Data;
using System.Configuration;
using System.Data.SqlClient;
public partial class CS : System.Web.UI.Page
{
protected void ExportCSV(object sender, EventArgs e)
{
string constr = ConfigurationManager.ConnectionStrings["constr"].ConnectionString;
using (SqlConnection con = new SqlConnection(constr))
{
using (SqlCommand cmd = new SqlCommand("SELECT * FROM Customers"))
{
using (SqlDataAdapter sda = new SqlDataAdapter())
{
cmd.Connection = con;
sda.SelectCommand = cmd;
using (DataTable dt = new DataTable())
{
sda.Fill(dt);
//Build the CSV file data as a Comma separated string.
string csv = string.Empty;
foreach (DataColumn column in dt.Columns)
{
//Add the Header row for CSV file.
csv += column.ColumnName + ',';
}
//Add new line.
csv += "\r\n";
foreach (DataRow row in dt.Rows)
{
foreach (DataColumn column in dt.Columns)
{
//Add the Data rows.
csv += row[column.ColumnName].ToString().Replace(",", ";") + ',';
}
//Add new line.
csv += "\r\n";
}
//Download the CSV file.
Response.Clear();
Response.Buffer = true;
Response.AddHeader("content-disposition", "attachment;filename=SqlExport.csv");
Response.Charset = "";
Response.ContentType = "application/text";
Response.Output.Write(csv);
Response.Flush();
Response.End();
}
}
}
}
}
}
Handling Commas
For handling commas inside of values when using string.Format(...), the following has worked for me:
var newLine = string.Format("\"{0}\",\"{1}\",\"{2}\"",
first,
second,
third
);
csv.AppendLine(newLine);
So to combine it with Johan's answer, it'd look like this:
//before your loop
var csv = new StringBuilder();
//in your loop
var first = reader[0].ToString();
var second = image.ToString();
//Suggestion made by KyleMit
var newLine = string.Format("\"{0}\",\"{1}\"", first, second);
csv.AppendLine(newLine);
//after your loop
File.WriteAllText(filePath, csv.ToString());
Returning CSV File
If you simply wanted to return the file instead of writing it to a location, this is an example of how I accomplished it:
From a Stored Procedure
public FileContentResults DownloadCSV()
{
// I have a stored procedure that queries the information I need
SqlConnection thisConnection = new SqlConnection("Data Source=sv12sql;User ID=UI_Readonly;Password=SuperSecure;Initial Catalog=DB_Name;Integrated Security=false");
SqlCommand queryCommand = new SqlCommand("spc_GetInfoINeed", thisConnection);
queryCommand.CommandType = CommandType.StoredProcedure;
StringBuilder sbRtn = new StringBuilder();
// If you want headers for your file
var header = string.Format("\"{0}\",\"{1}\",\"{2}\"",
"Name",
"Address",
"Phone Number"
);
sbRtn.AppendLine(header);
// Open Database Connection
thisConnection.Open();
using (SqlDataReader rdr = queryCommand.ExecuteReader())
{
while (rdr.Read())
{
// rdr["COLUMN NAME"].ToString();
var queryResults = string.Format("\"{0}\",\"{1}\",\"{2}\"",
rdr["Name"].ToString(),
rdr["Address"}.ToString(),
rdr["Phone Number"].ToString()
);
sbRtn.AppendLine(queryResults);
}
}
thisConnection.Close();
return File(new System.Text.UTF8Encoding().GetBytes(sbRtn.ToString()), "text/csv", "FileName.csv");
}
From a List
/* To help illustrate */
public static List<Person> list = new List<Person>();
/* To help illustrate */
public class Person
{
public string name;
public string address;
public string phoneNumber;
}
/* The important part */
public FileContentResults DownloadCSV()
{
StringBuilder sbRtn = new StringBuilder();
// If you want headers for your file
var header = string.Format("\"{0}\",\"{1}\",\"{2}\"",
"Name",
"Address",
"Phone Number"
);
sbRtn.AppendLine(header);
foreach (var item in list)
{
var listResults = string.Format("\"{0}\",\"{1}\",\"{2}\"",
item.name,
item.address,
item.phoneNumber
);
sbRtn.AppendLine(listResults);
}
}
return File(new System.Text.UTF8Encoding().GetBytes(sbRtn.ToString()), "text/csv", "FileName.csv");
}
Hopefully this is helpful.
This is a simple tutorial on creating csv files using C# that you will be able to edit and expand on to fit your own needs.
First you’ll need to create a new Visual Studio C# console application, there are steps to follow to do this.
The example code will create a csv file called MyTest.csv in the location you specify. The contents of the file should be 3 named columns with text in the first 3 rows.
https://tidbytez.com/2018/02/06/how-to-create-a-csv-file-with-c/
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO;
namespace CreateCsv
{
class Program
{
static void Main()
{
// Set the path and filename variable "path", filename being MyTest.csv in this example.
// Change SomeGuy for your username.
string path = #"C:\Users\SomeGuy\Desktop\MyTest.csv";
// Set the variable "delimiter" to ", ".
string delimiter = ", ";
// This text is added only once to the file.
if (!File.Exists(path))
{
// Create a file to write to.
string createText = "Column 1 Name" + delimiter + "Column 2 Name" + delimiter + "Column 3 Name" + delimiter + Environment.NewLine;
File.WriteAllText(path, createText);
}
// This text is always added, making the file longer over time
// if it is not deleted.
string appendText = "This is text for Column 1" + delimiter + "This is text for Column 2" + delimiter + "This is text for Column 3" + delimiter + Environment.NewLine;
File.AppendAllText(path, appendText);
// Open the file to read from.
string readText = File.ReadAllText(path);
Console.WriteLine(readText);
}
}
}
public static class Extensions
{
public static void WriteCSVLine(this StreamWriter writer, IEnumerable<string> fields)
{
const string q = #"""";
writer.WriteLine(string.Join(",",
fields.Select(
v => (v.Contains(',') || v.Contains('"') || v.Contains('\n') || v.Contains('\r')) ? $"{q}{v.Replace(q, q + q)}{q}" : v
)));
}
public static void WriteCSVLine(this StreamWriter writer, params string[] fields) => WriteCSVLine(writer, (IEnumerable<string>)fields);
}
This should allow you to write a csv file quite simply. Usage:
StreamWriter writer = new ("myfile.csv");
writer.WriteCSVLine("A", "B"); // A,B
Here is another open source library to create CSV file easily, Cinchoo ETL
List<dynamic> objs = new List<dynamic>();
dynamic rec1 = new ExpandoObject();
rec1.Id = 10;
rec1.Name = #"Mark";
rec1.JoinedDate = new DateTime(2001, 2, 2);
rec1.IsActive = true;
rec1.Salary = new ChoCurrency(100000);
objs.Add(rec1);
dynamic rec2 = new ExpandoObject();
rec2.Id = 200;
rec2.Name = "Tom";
rec2.JoinedDate = new DateTime(1990, 10, 23);
rec2.IsActive = false;
rec2.Salary = new ChoCurrency(150000);
objs.Add(rec2);
using (var parser = new ChoCSVWriter("emp.csv").WithFirstLineHeader())
{
parser.Write(objs);
}
For more information, please read the CodeProject article on usage.
One simple way to get rid of the overwriting issue is to use File.AppendText to append line at the end of the file as
void Main()
{
using (System.IO.StreamWriter sw = System.IO.File.AppendText("file.txt"))
{
string first = reader[0].ToString();
string second=image.ToString();
string csv = string.Format("{0},{1}\n", first, second);
sw.WriteLine(csv);
}
}
enter code here
string string_value= string.Empty;
for (int i = 0; i < ur_grid.Rows.Count; i++)
{
for (int j = 0; j < ur_grid.Rows[i].Cells.Count; j++)
{
if (!string.IsNullOrEmpty(ur_grid.Rows[i].Cells[j].Text.ToString()))
{
if (j > 0)
string_value= string_value+ "," + ur_grid.Rows[i].Cells[j].Text.ToString();
else
{
if (string.IsNullOrEmpty(string_value))
string_value= ur_grid.Rows[i].Cells[j].Text.ToString();
else
string_value= string_value+ Environment.NewLine + ur_grid.Rows[i].Cells[j].Text.ToString();
}
}
}
}
string where_to_save_file = #"d:\location\Files\sample.csv";
File.WriteAllText(where_to_save_file, string_value);
string server_path = "/site/Files/sample.csv";
Response.ContentType = ContentType;
Response.AppendHeader("Content-Disposition", "attachment; filename=" + Path.GetFileName(server_path));
Response.WriteFile(server_path);
Response.End();
You might just have to add a line feed "\n\r".
Hi I have a problem with importing a csv file into a sql server, this csv file contains articles that need to be saved in the sql server database. Once the import (done with the code c # written below) is finished, some fields imported as (Descrizione and CodArt) are not written correctly in the database and have strange characters. To download the csv file click here.
SqlServer improper import over blue line:
Import C# Code:
using (var rd = new StreamReader(labelPercorso.Text))
{
Articolo a = new Articolo();
a.db = this.db;
while (!rd.EndOfStream)
{
//setto codean e immagine =null ad ogni giro
CodEAN = "";
Immagine = "";
try
{
var splits = rd.ReadLine().Split(';');
CodArt = splits[0];
Descrizione = splits[1];
String Price = splits[2];
Prezzo = decimal.Parse(Price);
}
catch (Exception ex)
{
Console.WriteLine("Non è presente nè immagine nè codean");
}
a.Prezzo = Prezzo;
a.CodiceArticolo = CodArt;
a.Descrizione = Descrizione;
a.Fornitore = fornitore;
//manca da controllare se l'articolo è presente e nel caso aggiornalo
a.InserisciArticoloCSV();
}
}
Code of function: InserisciArticoloCSV
try
{
SqlConnection conn = db.apriconnessione();
String query = "INSERT INTO Articolo(CodArt,Descrizione,Prezzo,PrezzoListino,Fornitore,Importato,TipoArticolo) VALUES(#CodArt,#Descrizione,#Prezzo,#PrezzoListino,#Fornitore,#Importato,#TipoArticolo)";
String Importato = "CSV";
String TipoArticolo = "A";
SqlCommand cmd = new SqlCommand(query, conn);
// MessageBox.Show("CodArt: " + CodiceArticolo + "\n Descrizione :" + Descrizione + "\n Prezzo: " + Prezzo);
cmd.Parameters.AddWithValue("#CodArt", CodiceArticolo.ToString());
cmd.Parameters.AddWithValue("#Descrizione", Descrizione.ToString());
cmd.Parameters.AddWithValue("#Prezzo", Prezzo);
cmd.Parameters.AddWithValue("#PrezzoListino", Prezzo);
cmd.Parameters.AddWithValue("#Fornitore", Fornitore.ToString());
cmd.Parameters.AddWithValue("#Importato", Importato.ToString());
cmd.Parameters.AddWithValue("#TipoArticolo", TipoArticolo.ToString());
cmd.ExecuteNonQuery();
db.chiudiconnessione();
conn.Close();
return true;
}
catch (Exception ex)
{
Console.WriteLine("Errore nell'inserimento dell'articolo " + ex);
//MessageBox.Show("Errore nel inserimento dell'articolo: " + ex);
return false;
}
Your CSV file is not well formated , there are intermediatory Carriage Returns in between , which screws up the parsing. See the file in Notepad++ and turn on the Line Breaks , this is what you find.
So for the lines which are in format the data import is working fine , for others the logic is not working.
As others have pointed out, you have numerous problems, encoding, carriage returns and a lot of white space. In addition you are using single inserts into your database, which is very slow. I show below some sample code, which illustrates how to deal with all of these points.
IFormatProvider fP = new CultureInfo("it");
DataTable tmp = new DataTable();
tmp.Columns.Add("CodArt", typeof(string));
tmp.Columns.Add("Descrizione", typeof(string));
tmp.Columns.Add("Prezzo", typeof(decimal));
using (var rd = new StreamReader("yourFileName", Encoding.GetEncoding("iso-8859-1")))
{
while (!rd.EndOfStream)
{
try
{
var nextLine = Regex.Replace(rd.ReadLine(), #"\s+", " ");
while (nextLine.Split(';').Length < 3)
{
nextLine = nextLine.Replace("\r\n", "") + Regex.Replace(rd.ReadLine(), #"\s+", " ");
}
var splits = nextLine.Split(';');
DataRow dR = tmp.NewRow();
dR[0] = splits[0];
dR[1] = splits[1];
string Price = splits[2];
dR[2] = decimal.Parse(Price, fP);
tmp.Rows.Add(dR);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
}
using (var conn = db.apriconnessione())
{
var sBC = new SqlBulkCopy(conn);
conn.Open();
sBC.DestinationTableName = "yourTableName";
sBC.WriteToServer(tmp);
conn.Close();
}
Now for some explanation:
Firstly I am storing the parsed values in a DataTable. Please note that I have only included the three fields that are in the CSV. In practice you must supply the other columns and fill the extra columns with the correct values for each row. I was simply being lazy, but I am sure you will get the idea.
I do not know what encoding your csv file is, but iso-8859-1 worked for me!
I use Regex to replace multiple white space with a single space.
If any line does not have the required number of splits, I keep adding further lines (having deleted the carriage return) until I hit success!
Once I have a complete line, I can now split it, and assign it to the new DataRow (please see my comments above for extra columns).
Finally once the file has been read, the DataTable will have all the rows and can be uploaded to your database using BulkCopy. This is very fast!
HTH
PS Some of your lines have double quotes. You probably want to get rid of these as well!
You should specify the correct encoding when you read your file. Is it utf? Is it ascii with a specific code page? You should also specify the SqlDbType of your Sql parameters, especially the string parameters which will be either varchar or nvarchar and there is a big difference between them.
// what is the encoding of your file? This is an example using code page windows-1252
var encoding = Encoding.GetEncoding("windows-1252");
using (var file = File.Open(labelPercorso.Text, FileMode.Open))
using (var reader = new StreamReader(file, encoding))
{
// rest of code unchanged
}
Sql Code. Note that I added using blocks for the types that implement IDisposable like Connection and Command.
try
{
String query = "INSERT INTO Articolo(CodArt,Descrizione,Prezzo,PrezzoListino,Fornitore,Importato,TipoArticolo) VALUES(#CodArt,#Descrizione,#Prezzo,#PrezzoListino,#Fornitore,#Importato,#TipoArticolo)";
String Importato = "CSV";
String TipoArticolo = "A";
using(SqlConnection conn = db.apriconnessione())
using(SqlCommand cmd = new SqlCommand(query, conn))
{
// -1 indicates you used MAX like nvarchar(max), otherwise use the maximum number of characters in the schema
cmd.Parameters.Add(new SqlDbParameter("#CodArt", SqlDbType.NVarChar, -1)).Value = CodiceArticolo.ToString();
cmd.Parameters.Add(new SqlDbParameter("#Descrizione", SqlDbType.NVarChar, -1)).Value = Descrizione.ToString();
/*
Rest of your parameters created in the same manner
*/
cmd.ExecuteNonQuery();
db.chiudiconnessione();
}
return true;
}
catch (Exception ex)
{
Console.WriteLine("Errore nell'inserimento dell'articolo " + ex);
//MessageBox.Show("Errore nel inserimento dell'articolo: " + ex);
return false;
}
Just in case if you are interested in exploring library to handle all parsing needs with few lines of code, you can check out the Cinchoo ETL - an open source library. Here is sample to parse the csv file and shows how to get either datatable or list of records for later to load them to database.
System.Threading.Thread.CurrentThread.CurrentCulture = new CultureInfo("it");
using (var p = new ChoCSVReader("Bosch Luglio 2017.csv")
.Configure((c) => c.MayContainEOLInData = true) //Handle newline chars in data
.Configure(c => c.Encoding = Encoding.GetEncoding("iso-8859-1")) //Specify the encoding for reading
.WithField("CodArt", 1) //first column
.WithField("Descrizione", 2) //second column
.WithField("Prezzo", 3, fieldType: typeof(decimal)) //third column
.Setup(c => c.BeforeRecordLoad += (o, e) =>
{
e.Source = e.Source.CastTo<string>().Replace(#"""", String.Empty); //Remove the quotes
}) //Scrub the data
)
{
var dt = p.AsDataTable();
//foreach (var rec in p)
// Console.WriteLine(rec.Prezzo);
}
Disclaimer: I'm the author of this library.
How can I use OLEDB to parse and import a CSV file that each cell is encased in double quotes because some rows contain commas in them?? I am unable to change the format as it is coming from a vendor.
I am trying the following and it is failing with an IO error:
public DataTable ConvertToDataTable(string fileToImport, string fileDestination)
{
string fullImportPath = fileDestination + #"\" + fileToImport;
OleDbDataAdapter dAdapter = null;
DataTable dTable = null;
try
{
if (!File.Exists(fullImportPath))
return null;
string full = Path.GetFullPath(fullImportPath);
string file = Path.GetFileName(full);
string dir = Path.GetDirectoryName(full);
//create the "database" connection string
string connString = "Provider=Microsoft.Jet.OLEDB.4.0;"
+ "Data Source=\"" + dir + "\\\";"
+ "Extended Properties=\"text;HDR=No;FMT=Delimited\"";
//create the database query
string query = "SELECT * FROM " + file;
//create a DataTable to hold the query results
dTable = new DataTable();
//create an OleDbDataAdapter to execute the query
dAdapter = new OleDbDataAdapter(query, connString);
//fill the DataTable
dAdapter.Fill(dTable);
}
catch (Exception ex)
{
throw new Exception(CLASS_NAME + ".ConvertToDataTable: Caught Exception: " + ex);
}
finally
{
if (dAdapter != null)
dAdapter.Dispose();
}
return dTable;
}
When I use a normal CSV it works fine. Do I need to change something in the connString??
Use a dedicated CSV parser.
There are many out there. A popular one is FileHelpers, though there is one hidden in the Microsoft.VisualBasic.FileIO namespace - TextFieldParser.
Have a look at FileHelpers.
You can use this code : MS office required
private void ConvertCSVtoExcel(string filePath = #"E:\nucc_taxonomy_140.csv", string tableName = "TempTaxonomyCodes")
{
string tempPath = System.IO.Path.GetDirectoryName(filePath);
string strConn = #"Driver={Microsoft Text Driver (*.txt; *.csv)};Dbq=" + tempPath + #"\;Extensions=asc,csv,tab,txt";
OdbcConnection conn = new OdbcConnection(strConn);
OdbcDataAdapter da = new OdbcDataAdapter("Select * from " + System.IO.Path.GetFileName(filePath), conn);
DataTable dt = new DataTable();
da.Fill(dt);
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(ConfigurationSettings.AppSettings["dbConnectionString"]))
{
bulkCopy.DestinationTableName = tableName;
bulkCopy.BatchSize = 50;
bulkCopy.WriteToServer(dt);
}
}
There is a lot to consider when handling CSV files. However you extract them from the file, you should know how you are handling the parsing. There are classes out there that can get you part way, but most don't handle the nuances that Excel does with embedded commas, quotes and line breaks. However, loading Excel or the MS classes seems a lot of freaking overhead if you just want parse a txt file like a CSV.
One thing you can consider is doing the parsing in your own Regex, which will also make your code a little more platform independent, in case you need to port it to another server or application at some point. Using regex has the benefit of also being accessible in virtually every language. That said, there are some good regex patterns out there that handle the CSV puzzle. Here is my shot at it, which does cover embedded commas, quotes and line breaks. Regex code/pattern and explanation :
http://www.kimgentes.com/worshiptech-web-tools-page/2008/10/14/regex-pattern-for-parsing-csv-files-with-embedded-commas-dou.html
Hope that is of some help..
Try the code from my answer here:
Reading CSV files in C#
It handles quoted csv just fine.
private static void Mubashir_CSVParser(string s)
{
// extract the fields
Regex RegexCSVParser = new Regex(",(?=(?:[^\"]*\"[^\"]*\")*(?![^\"]*\"))");
String[] Fields = RegexCSVParser.Split(s);
// clean up the fields (remove " and leading spaces)
for (int i = 0; i < Fields.Length; i++)
{
Fields[i] = Fields[i].TrimStart(' ', '"');
Fields[i] = Fields[i].TrimEnd('"');// this line remove the quotes
//Fields[i] = Fields[i].Trim();
}
}
Just incase anyone has a similar issue, i wanted to post the code i used. i did end up using Textparser to get the file and parse ot the columns, but i am using recrusion to get the rest done and substrings.
/// <summary>
/// Parses each string passed as a "row".
/// This routine accounts for both double quotes
/// as well as commas currently, but can be added to
/// </summary>
/// <param name="row"> string or row to be parsed</param>
/// <returns></returns>
private List<String> ParseRowToList(String row)
{
List<String> returnValue = new List<String>();
if (row[0] == '\"')
{// Quoted String
if (row.IndexOf("\",") > -1)
{// There are more columns
returnValue = ParseRowToList(row.Substring(row.IndexOf("\",") + 2));
returnValue.Insert(0, row.Substring(1, row.IndexOf("\",") - 1));
}
else
{// This is the last column
returnValue.Add(row.Substring(1, row.Length - 2));
}
}
else
{// Unquoted String
if (row.IndexOf(",") > -1)
{// There are more columns
returnValue = ParseRowToList(row.Substring(row.IndexOf(",") + 1));
returnValue.Insert(0, row.Substring(0, row.IndexOf(",")));
}
else
{// This is the last column
returnValue.Add(row.Substring(0, row.Length));
}
}
return returnValue;
}
Then the code for Textparser is:
// string pathFile = #"C:\TestFTP\TestCatalog.txt";
string pathFile = #"C:\TestFTP\SomeFile.csv";
List<String> stringList = new List<String>();
TextFieldParser fieldParser = null;
DataTable dtable = new DataTable();
/* Set up TextFieldParser
* use the correct delimiter provided
* and path */
fieldParser = new TextFieldParser(pathFile);
/* Set that there are quotes in the file for fields and or column names */
fieldParser.HasFieldsEnclosedInQuotes = true;
/* delimiter by default to be used first */
fieldParser.SetDelimiters(new string[] { "," });
// Build Full table to be imported
dtable = BuildDataTable(fieldParser, dtable);
This is what I used in a project, parses a single line of data.
private string[] csvParser(string csv, char separator = ',')
{
List <string> parsed = new List<string>();
string[] temp = csv.Split(separator);
int counter = 0;
string data = string.Empty;
while (counter < temp.Length)
{
data = temp[counter].Trim();
if (data.Trim().StartsWith("\""))
{
bool isLast = false;
while (!isLast && counter < temp.Length)
{
data += separator.ToString() + temp[counter + 1];
counter++;
isLast = (temp[counter].Trim().EndsWith("\""));
}
}
parsed.Add(data);
counter++;
}
return parsed.ToArray();
}
http://zamirsblog.blogspot.com/2013/09/c-csv-parser-csvparser.html
The following code writes the data and is working fine, but I want to add more than one client (maybe 10) in the .csv file. How can I achieve this. Thanks in advance.
private void createFileButton_Click(object sender, EventArgs e)
{
string newFileName = "C:\\client_20100913.csv";
string clientDetails = clientNameTextBox.Text + "," + mIDTextBox.Text + "," + billToTextBox.Text;
//Header of the .csv File
string clientHeader = "Client Name(ie. Billto_desc)" + "," + "Mid_id,billing number(ie billto_id)" + "," + "business unit id" + Environment.NewLine;
File.WriteAllText(newFileName, clientHeader);
File.AppendAllText(newFileName, clientDetails);
MessageBox.Show("Client Added", "Added", MessageBoxButtons.OK);
}
If you want to append the client information to an existing file, how about:
string newFileName = "C:\\client_20100913.csv";
string clientDetails = clientNameTextBox.Text + "," + mIDTextBox.Text + "," + billToTextBox.Text;
if (!File.Exists(newFileName))
{
string clientHeader = "Client Name(ie. Billto_desc)" + "," + "Mid_id,billing number(ie billto_id)" + "," + "business unit id" + Environment.NewLine;
File.WriteAllText(newFileName, clientHeader);
}
File.AppendAllText(newFileName, clientDetails);
This way the header line is only written the first time, when the file is created.
Although it would probably be even nicer to provide a list-detail view that lets you view all clients, add and remove clients, select a client to edit details, and save the complete file with all clients.
It looks to me like you want a new client to be added every time you click the button.
If that's the case, the reason why it doesn't work currently is that the file is being cleared by the line
File.WriteAllText(newFileName, clientHeader);
The simplest change would be to check if the file exists before writing over it:
if (!File.Exists(newFileName))
{
//Header of the .csv File
string clientHeader = "Client Name(ie. Billto_desc)" + "," + "Mid_id,billing number(ie billto_id)" + "," + "business unit id" + Environment.NewLine;
File.WriteAllText(newFileName, clientHeader);
}
Although you could use other strategies, such as creating the file on startup of the application and keeping it open (using something like a StreamWriter). You would then close the writer when your application exited. This would pretty much guarantee that the file couldn't be messed with while your application is open.
You might want to do this because there is a race condition in that code - after you check the file exists, and before you write to the file, a user could delete it. Keeping the file open helps to avoid this, but you may or may not want to do it.
The underlying problem here seems to be where you're getting the data from to append to your CSV file. Your example code looks like it gets the various pieces of data from text boxes on the page, so if you want multiple clients, are they all going to have their data on the screen in text boxes? My instinct is probably not.
It sounds to me like you should be handling this client data using a class of some sort (perhaps persisted in a database) and then implement a method in the class called something like void AppendToCSV(string filename), which appends that client data to the CSV file. Then you can loop over your client objects, appending each one in turn.
How you produce/store your client objects, in relation to the text boxes you have on the screen, depends on what your app is trying to achieve.
I know this has been answered but there is what i did to create a "log" of subscribers. This uses reflection to get the properties and values of the object. Hope this helps someone in the future.
internal static bool UpdateSubscriberList(MailingListEmail subscriber)
{
PropertyInfo[] propertyinfo;
propertyinfo = typeof(MailingListEmail).GetProperties();
var values = string.Empty;
try
{
string fileName = #"C:\Development\test.csv";
if (!File.Exists(fileName))
{
var header = string.Empty;
foreach (var prop in propertyinfo)
{
if (string.IsNullOrEmpty(header))
header += prop.Name;
else
header = string.Format("{0},{1}", header, prop.Name);
}
header = string.Format("{0},{1}", header, "CreatedDate");
header += Environment.NewLine;
File.WriteAllText(fileName, header);
}
foreach (var prop in propertyinfo)
{
var value = prop.GetValue(subscriber, null);
if (string.IsNullOrEmpty(values))
values += value;
else
values = string.Format("{0},{1}", values, value);
}
values = string.Format("{0},{1}", values, DateTime.Now.ToShortDateString());
values += Environment.NewLine;
File.AppendAllText(fileName, values);
}
catch (Exception ex)
{
Elmah.ErrorSignal.FromCurrentContext().Raise(ex);
return false;
}
return true;
}
here is what i have done, and it works for me perfectly :
first you need to creat DataTable from your listview, or just put data from textboxes:
`public Boolean PreparCVS(string DateOne, string DataTwo)
{
try
{
// Create the `DataTable` structure according to your data source
DataTable table = new DataTable();
table.Columns.Add("HeaderOne", typeof(string));
table.Columns.Add("HeaderTwo", typeof(String));
// Iterate through data source object and fill the table
table.Rows.Add(HeaderOne, HeaderTwo);
//Creat CSV File
CreateCSVFile(table, sCsvFilePath);
return true;
}
catch (Exception ex)
{
throw new System.Exception(ex.Message);
}
}`
once dataTable is created you can generate CSV file by this method :
in the streamwriter constructor you must specify in the second parameter True, by this, you can append data to you existing .csv file :
public void CreateCSVFile(DataTable dt, string strFilePath)
{
StreamWriter sw = new StreamWriter(strFilePath, true);
int iColCount = dt.Columns.Count;
for (int i = 0; i < iColCount; i++)
{
sw.Write(dt.Columns[i]);
if (i < iColCount - 1)
{
sw.Write(",");
}
}
sw.Write(sw.NewLine);
foreach (DataRow dr in dt.Rows)
{
for (int i = 0; i < iColCount; i++)
{
if (!Convert.IsDBNull(dr[i]))
{
sw.Write(dr[i].ToString());
}
if (i < iColCount - 1)
{
sw.Write(",");
}
}
sw.Write(sw.NewLine);
}
sw.Close();
}
// At first read all the data from your first CSV
StreamReader oStreamReader = new StreamReader(#"d:\test\SourceFile.csv");
string recordsFromFirstFile = oStreamReader.ReadToEnd();
oStreamReader.Close();
// Now read the new records from your another csv file
oStreamReader = new StreamReader(#"d:\test\DestinationFile.csv");
string recordsFromSecondFile = oStreamReader.ReadToEnd();
oStreamReader.Close();
oStreamReader.Dispose();
// Here Records from second file will also contain column headers so we need to truncate them using Substring() method
recordsFromSecondFile = recordsFromSecondFile.Substring(recordsFromSecondFile.IndexOf('\n') + 1);
// Now merge the records either in SourceFile.csv or in Targetfile.csv or as per your required file
StreamWriter oStreamWriter= new StreamWriter(#"d:\testdata\TargetFile.csv");
oStreamWriter.Write(recordsFromFirstFile + recordsFromSecondFile);
oStreamWriter.Close();
oStreamWriter.Dispose();
Happy Coding.....
c#csv
using CsvHelper;
public void WriteDataToCsv(MsgEnvironmentData[] data, string csvPath)
{
if (!File.Exists(csvPath))
{
using (var writer = new StreamWriter(csvPath))
using (var csvWriter = new CsvWriter(writer,firstConfiguration))
{
csvWriter.WriteHeader<MsgEnvironmentData>();
csvWriter.NextRecord();
csvWriter.WriteRecords(data);
}
}
else
{
using (var stream = new FileStream(csvPath, FileMode.Append))
using (var writer = new StreamWriter(stream))
using (var csvWriter = new CsvWriter(writer, secondConfiguration))
{
csvWriter.WriteRecords(data);
}
}
}
Jeramy's answer writing the contents on last cell and from their horizontally in a row in csv file. I mixed and matched his solution with answer given here. I know this questions been asked long before but for the ones who doing research I'm posting the answer here.
string newFileName = #"C:\.NET\test.csv"; //filepath
var csv = new StringBuilder();
string clientDetails = "content1,content2,content3" + Environment.NewLine;
csv.Append(clientDetails);
File.AppendAllText(newFileName, csv.ToString());
I use this simple piece of code to append data to an existing CSV file:
string[] data = { "John", "Doe", "25" };
string csvFilePath = "example.csv";
// Open the file for writing
using (StreamWriter writer = File.AppendText(csvFilePath))
{
// Write the data row
writer.WriteLine(string.Join(",", data));
}
My program is now still running to import data from a log file into a remote SQL Server Database. The log file is about 80MB in size and contains about 470000 lines, with about 25000 lines of data. My program can import only 300 rows/second, which is really bad. :(
public static int ImportData(string strPath)
{
//NameValueCollection collection = ConfigurationManager.AppSettings;
using (TextReader sr = new StreamReader(strPath))
{
sr.ReadLine(); //ignore three first lines of log file
sr.ReadLine();
sr.ReadLine();
string strLine;
var cn = new SqlConnection(ConnectionString);
cn.Open();
while ((strLine = sr.ReadLine()) != null)
{
{
if (strLine.Trim() != "") //if not a blank line, then import into database
{
InsertData(strLine, cn);
_count++;
}
}
}
cn.Close();
sr.Close();
return _count;
}
}
InsertData is just a normal insert method using ADO.NET. It uses a parsing method:
public Data(string strLine)
{
string[] list = strLine.Split(new[] {'\t'});
try
{
Senttime = DateTime.Parse(list[0] + " " + list[1]);
}
catch (Exception)
{
}
Clientip = list[2];
Clienthostname = list[3];
Partnername = list[4];
Serverhostname = list[5];
Serverip = list[6];
Recipientaddress = list[7];
Eventid = Convert.ToInt16(list[8]);
Msgid = list[9];
Priority = Convert.ToInt16(list[10]);
Recipientreportstatus = Convert.ToByte(list[11]);
Totalbytes = Convert.ToInt32(list[12]);
Numberrecipient = Convert.ToInt16(list[13]);
DateTime temp;
if (DateTime.TryParse(list[14], out temp))
{
OriginationTime = temp;
}
else
{
OriginationTime = null;
}
Encryption = list[15];
ServiceVersion = list[16];
LinkedMsgid = list[17];
MessageSubject = list[18];
SenderAddress = list[19];
}
InsertData method:
private static void InsertData(string strLine, SqlConnection cn)
{
var dt = new Data(strLine); //parse the log line into proper fields
const string cnnStr =
"INSERT INTO LOGDATA ([SentTime]," + "[client-ip]," +
"[Client-hostname]," + "[Partner-Name]," + "[Server-hostname]," +
"[server-IP]," + "[Recipient-Address]," + "[Event-ID]," + "[MSGID]," +
"[Priority]," + "[Recipient-Report-Status]," + "[total-bytes]," +
"[Number-Recipients]," + "[Origination-Time]," + "[Encryption]," +
"[service-Version]," + "[Linked-MSGID]," + "[Message-Subject]," +
"[Sender-Address]) " + " VALUES ( " + "#Senttime," + "#Clientip," +
"#Clienthostname," + "#Partnername," + "#Serverhostname," + "#Serverip," +
"#Recipientaddress," + "#Eventid," + "#Msgid," + "#Priority," +
"#Recipientreportstatus," + "#Totalbytes," + "#Numberrecipient," +
"#OriginationTime," + "#Encryption," + "#ServiceVersion," +
"#LinkedMsgid," + "#MessageSubject," + "#SenderAddress)";
var cmd = new SqlCommand(cnnStr, cn) {CommandType = CommandType.Text};
cmd.Parameters.AddWithValue("#Senttime", dt.Senttime);
cmd.Parameters.AddWithValue("#Clientip", dt.Clientip);
cmd.Parameters.AddWithValue("#Clienthostname", dt.Clienthostname);
cmd.Parameters.AddWithValue("#Partnername", dt.Partnername);
cmd.Parameters.AddWithValue("#Serverhostname", dt.Serverhostname);
cmd.Parameters.AddWithValue("#Serverip", dt.Serverip);
cmd.Parameters.AddWithValue("#Recipientaddress", dt.Recipientaddress);
cmd.Parameters.AddWithValue("#Eventid", dt.Eventid);
cmd.Parameters.AddWithValue("#Msgid", dt.Msgid);
cmd.Parameters.AddWithValue("#Priority", dt.Priority);
cmd.Parameters.AddWithValue("#Recipientreportstatus", dt.Recipientreportstatus);
cmd.Parameters.AddWithValue("#Totalbytes", dt.Totalbytes);
cmd.Parameters.AddWithValue("#Numberrecipient", dt.Numberrecipient);
if (dt.OriginationTime != null)
cmd.Parameters.AddWithValue("#OriginationTime", dt.OriginationTime);
else
cmd.Parameters.AddWithValue("#OriginationTime", DBNull.Value);
//if OriginationTime was null, then insert with null value to this column
cmd.Parameters.AddWithValue("#Encryption", dt.Encryption);
cmd.Parameters.AddWithValue("#ServiceVersion", dt.ServiceVersion);
cmd.Parameters.AddWithValue("#LinkedMsgid", dt.LinkedMsgid);
cmd.Parameters.AddWithValue("#MessageSubject", dt.MessageSubject);
cmd.Parameters.AddWithValue("#SenderAddress", dt.SenderAddress);
cmd.ExecuteNonQuery();
}
How can my program run faster?
Thank you so much!
Use SqlBulkCopy.
Edit: I created a minimal implementation of IDataReader and created a Batch type so that I could insert arbitrary in-memory data using SqlBulkCopy. Here is the important bit:
IDataReader dr = batch.GetDataReader();
using (SqlTransaction tx = _connection.BeginTransaction())
{
try
{
using (SqlBulkCopy sqlBulkCopy =
new SqlBulkCopy(_connection, SqlBulkCopyOptions.Default, tx))
{
sqlBulkCopy.DestinationTableName = TableName;
SetColumnMappings(sqlBulkCopy.ColumnMappings);
sqlBulkCopy.WriteToServer(dr);
tx.Commit();
}
}
catch
{
tx.Rollback();
throw;
}
}
The rest of the implementation is left as an exercise for the reader :)
Hint: the only bits of IDataReader you need to implement are Read, GetValue and FieldCount.
Hmmm, let's break this down a little bit.
In pseudocode what you did is the ff:
Open the file
Open a connection
For every line that has data:
Parse the string
Save the data in SQL Server
Close the connection
Close the file
Now the fundamental problems in doing it this way are:
You are keeping a SQL connection open while waiting for your line parsing (pretty susceptible to timeouts and stuff)
You might be saving the data line by line, each in its own transaction. We won't know until you show us what the InsertData method is doing
Consequently you are keeping the file open while waiting for SQL to finish inserting
The optimal way of doing this is to parse the file as a whole, and then insert them in bulk. You can do this with SqlBulkCopy (as suggested by Matt Howells), or with SQL Server Integration Services.
If you want to stick with ADO.NET, you can pool together your INSERT statements and then pass them off into one large SQLCommand, instead of doing it this way e.g., setting up one SQLCommand object per insert statement.
You create the SqlCommand object for every row of data. The simplest improvement would therefore to create a
private static SqlCommand cmdInsert
and declare the parameters with the Parameters.Add() method. Then for each data row, set the parameter values using
cmdInsert.Parameters["#paramXXX"].Value = valueXXX;
A second performance improvement might be to skip creation of Data objects for each row, and assign Parameter values directly from the list[] array.