Develop Reference

C# - Reading from CSV file producing two entries at once?

I am trying to read in a list of several hundred thousand values from what once was a spreadsheet, but for the sake of simplicity, I have turned into a CSV file.
My problem is that while testing it to make sure it reads properly, the code is for some reason ignoring the comma after the second position, and combining the value in that spot with the value next to it, despite being, you know, separated by a comma. It also begins to combine the final value with the first value from the next set.
For Example:
CSV File:
0,0,0,104672
0,1,6,51971
0,1,36,80212
0,2,5,51972
0,2,13,51973
...
Program Output:
0
00
1046720
00
16
519710
136
...
I think the example probably does a better job describing what's going on than I did in words. It continues like that, displaying the wrong information until it reaches the end of the file.
My code is as follows:
static void Main()
{
using(var fs = File.OpenRead(#"C:\path\to\file.csv"))
using(var read = new StreamReader(fs))
{
while (!read.EndOfStream)
{
int i = 0;
var line = read.ReadLine();
while (i < 4)
{
var values = line.Split(',');
Console.Write(values[i]);
Console.Read();
i++;
}
}
}
}
EDIT: Sorry, I got lost in my understanding of what the code should do and forgot to explain the goal here.
This program is made to take these values and rename a file from the 4th value (for example, 104672) to the first three values, separated by dashes (ex. 0-0-0). What I want from my output right now is to be able to see the program give me the values back, one at a time, so that I know when I go to rename the files, I'm not getting improper results.
EDIT 2: I also realize, a day later, that the answer I got was one of significance to making my program work, rather than actually discovering why I was getting the output I got. For those curious in the future, the answer is essentially that Console.Read(); is not a true pause, and causes more writes to happen upon key press than expected.

A more clear and easy-to-understand approach would be:
using (StreamReader sr = new StreamReader(#"C:\path\to\file.csv"))
{
string currentLine;
while((currentLine = sr.ReadLine()) != null)
{
string[] lineArr = line.Split(',');
foreach(string subLine in lineArr)
{
Console.WriteLine(subline);
}
Console.Read(); // Awaits user input in order to proceed
}
}

if you need those values for later use, why don't you put them into List for later... like this:
List<string[]> listOfValues = new List<string[]>();
using (var fs = File.OpenRead(#"C:\temp\csv.txt"))
using (var read = new StreamReader(fs))
{
while (!read.EndOfStream)
{
var line = read.ReadLine();
listOfValues.Add(line.Split(','));
}
}
later, you can use data from list:
for (int i = 0; i<listOfValues.Count; i++)
{
Console.WriteLine("line number: {0}, contents: {1}", i + 1, string.Join(" ", listOfValues[i]));
}
which gives you
line number: 1, contents: 0 0 0 104672
line number: 2, contents: 0 1 6 51971
line number: 3, contents: 0 1 36 80212
line number: 4, contents: 0 2 5 51972
line number: 5, contents: 0 2 13 51973

It's hard to tell from your code what you think it's supposed to do. Here is a version that will read each line, split it on the commas, and iterate through the values, printing each value. After printing all the values for a line, it prints a new line. Hopefully that's something like what you were trying to achieve.
static void Main()
{
using(var fs = File.OpenRead(#"C:\path\to\file.csv"))
using(var read = new StreamReader(fs))
{
while (!read.EndOfStream)
{
int i = 0;
var line = read.ReadLine();
var values = line.Split(',');
while (i < values.Length)
{
Console.Write(values[i]);
//Console.Read();
i++;
}
Console.WriteLine();
}
}
}

As #rory.ap said, you have plenty of libraries to read CSV right out of the box. But event if you still want to do it on your own, it seems that it is taking a great effort on doing a simple task. Try this:
using (StreamReader reader = new StreamReader("C:/yourpath/yourfile.csv"))
{
while (reader.Peek() >= 0)
{
string line = reader.ReadLine();
string[] yourData = line.Split(',');
}
}

Why not just
foreach (var line in File.ReadLines(#"C:\path\to\file.csv"))
foreach (var segment in line.Split(','))
Console.WriteLine(segment);

C# removing last delimited field from a string split

I am a beginner c# programmer and just had a quick question on an application I am building. My process reads in multiple files with the purpose of stripping out specific records based on a 1 or 0 pipe delimited field in the text file. It is the last delimited field in the file actually. If it is a 0, I write it to a temp file (which will later replace the original that I read), if it is anything else I do not. And not to try to get it too confusing but there are two types of records in the file, a header row, and then that is followed by a few supp rows. The header row is the only one that has the flag, so as you can tell from below, if the bool gets set to a good record by being 0, it writes the header record along with all supp records below it until it hits a bad one in which case it will negate writing them until the next good one.
However, what I am trying to do now (and would like to know the easiest way), is how to write the header record without the last pipe delimited field (IE the flag). Since it should always be the last 2 characters of the row (for example "0|" or "1|" as the preceeding pipe is needed), should it be a string trim on my inputrecord string? Is there an easier way? Is there a way to do a split on the record but not actually include the last field (in this case, field 36)? Any advice would be appreciated. Thank you,
static void Main(string[] args)
{
try
{
string executionDirectory = RemoveFlaggedRecords.Properties.Settings.Default.executionDirectory;
string workDirectory = RemoveFlaggedRecords.Properties.Settings.Default.workingDirectory;
string[] files = Directory.GetFiles(executionDirectory, "FilePrefix*");
foreach (string file in files)
{
string tempFile = Path.Combine(workDirectory,Path.GetFileName(file));
using (StreamReader sr = new StreamReader(file,Encoding.Default))
{
StreamWriter sw = new StreamWriter(tempFile);
string inputRecord = sr.ReadLine();
bool goodRecord = false;
bool isheaderRecord = false;
while (inputRecord != null)
{
string[] fields = inputRecord.Split('|');
if (fields[0].ToString().ToUpper() == "HEADER")
{
goodRecord = Convert.ToInt32(fields[36]) == 0;
isheaderRecord = true;
}
if (goodRecord == true && isheaderRecord == true)
{
// I'm not sure what to do here to write the string without the 36th field***
}
else if (goodRecord == true)
{
sw.WriteLine(inputRecord);
}
inputRecord = sr.ReadLine();
}
sr.Close();
sw.Close();
sw = null;
}
}
string[] newFiles = Directory.GetFiles(workDirectory, "fileprefix*");
foreach (string file in newFiles)
{
string tempFile = Path.Combine(workDirectory, Path.GetFileName(file));
string destFile = Path.Combine(executionDirectory, Path.GetFileName(file));
File.Copy(tempFile, destFile, true);
if (File.Exists(destFile))
{
File.Delete(tempFile);
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex);
}
finally
{
// not done
}
}

One way you could do this - if what you want at that point in the code is to always write all but the final element in your string[] - is construct a for loop that terminates before the last item:
for (int i = 0; i < fields.Length - 1; i++)
{
// write your field here
}
This is assuming that you want to write each field individually, and that you want to iterate through fields in the first place. If all you want to do is just write a single string to a single line without using a loop, you could do this:
var truncatedFields = fields.Take(fields.Length - 1);
And then just write the truncatedFields string[] as you see fit. One way you could accomplish all this in a single line might look like so:
sw.WriteLine(String.Join("|", fields.Take(fields.Length - 1)));

goodRecord = fields.Last().Trim() == "0";
if (inputRecord.Contains("|") string outputRecord = inputRecord.Substring(1, inputRecord.LastIndexOf("|"));

Read and extract from file

I have a huge file with ~3 mill rows. Every line contains record like this:
1|2|3|4|5|6|7|8|9
Exactly 8 separators like '|' on every line. I am looking for a way to read this file then extract last '9' number only from every line and store it into another file.
edit:
Ok here is what i done already.
using (StreamReader sr = new StreamReader(filepath))
using (StreamWriter sw = new StreamWriter(filepath1))
{
string line = null;
while ((line = sr.ReadLine()) != null)
sw.WriteLine(line.Split('|')[8]);
}
File.WriteAllLines("filepath", File.ReadAllLines(filepath).Where(l => !string.IsNullOrWhiteSpace(l)));
Read file, extract last digits then write in new file and clear blank lines. Last digit is 10-15 symbols and I want to extract first 6. I continue to read and try some and when I'm done or have some question I'll edit again.
Thanks
Edit 2:
Ok, here I take first 8 digits from the number:
sw.WriteLine(line.Substring(0, Math.Min(line.Length, 8)));
Edit 3:
I have no idea how can I match now every numbers that left in file. I want to match them and to see witch number how many times is in the file.
Any help?

I am looking for a way to read this file then extract last [..] number only from every line and store it into another file.
What part exactly are you having trouble with? In psuedo code, this is what you want:
fileReader = OpenFile("input")
fileWriter = OpenFile("output")
while !fileReader.EndOfFile
line = fileReader.ReadLine
records[] = line.Split('|')
value = records[8]
fileWriter.WriteLine(value)
do
So start implementing it and feel free to ask a question on any specific line you're having trouble with. Each line of code I posted contains enough pointers to figure out the C# code or the terms to do a web search for it.

You don't say where you are stuck. Break the problem down:
Write and run minimal C# program
Read lines from file
Break up one line
write result line to a file
Are you stuck on any one of those? Then ask a specific question about that. This decomposition technique is key to many programming tasks, and indeed complex tasks in general.
You might find the string split capability useful.

Because it's a huge file you must read it line by line!
public IEnumerable ReadFileIterator(String filePath)
{
using (StreamReader streamReader = new StreamReader(filePath, Encoding.Default))
{
String line;
while ((line = streamReader.ReadLine()) != null)
{
yield return line;
}
yield break;
}
}
public void WriteToFile(String inputFilePath, String outputFilePath)
{
using (StreamWriter streamWriter = new StreamWriter(outputFilePath, true, Encoding.Default))
{
foreach (String line in ReadFileIterator(inputFilePath))
{
String[] subStrings = line.Split('|');
streamWriter.WriteLine(subStrings[8]);
}
streamWriter.Flush();
streamWriter.Close();
}
}

using (StreamReader sr = new StreamReader("input"))
using (StreamWriter sw = new StreamWriter("output"))
{
string line = null;
while ((line=sr.ReadLine())!=null)
sw.WriteLine(line.Split('|')[8]);
}

Some pointer to start from: StreamReader.Readline() and String.Split(). There are examples on both pages.

With LINQ you could do a thing like the following to filter the numbers:
var numbers = from l in File.ReadLines(fileName)
let p = l.Split('|')
select p[8];
and then write them into a new file like that:
File.WriteAllText(newFileName, String.Join("\r\n", numbers));

Use String.Split() to get the line inside an array and get the last element and store it into another file. Repeat the process for each line.

Try this...
// Read the file and display it line by line.
System.IO.StreamReader file =
new System.IO.StreamReader("c:\\test.txt");
while((line = file.ReadLine()) != null)
{
string[] words = s.Split('|');
string value = words [8]
Console.WriteLine (value);
}
file.Close();

Issue with adding a Double to a List<double>

I'm getting an issue with adding a Double to a List<double> from a CSV file. Now I've done this before, with the exact same file, and added it to a Double Array.
Here is my code that works with an array:
double[] ch1Array = new double[arraySize];
if (openFileDialog1.ShowDialog() == DialogResult.OK)
{
string fileName = Path.Combine(filePath, openFileDialog1.FileName);
var reader = new StreamReader(File.OpenRead(fileName));
while(!reader.EndOfStream)
{
var line = reader.ReadLine(); //Get through the column titles
var values = line.Split(',');
if (dataSize > 0)
{
try
{
ch1Array[dataSize] = Convert.ToDouble(values[1]);
//etc...
This code works perfectly and the only thing I change is that I am not using an Array anymore and instead am using a List<double>:
List<double> ch1Array = new List<double>();
if (openFileDialog1.ShowDialog() == DialogResult.OK)
{
//Create stream reader and open file
string fileName = Path.Combine(mainFilePath, openFileDialog1.FileName);
var reader = new StreamReader(File.OpenRead(fileName));
int counter = 0;
//Read document until end of stream
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
var values = line.Split(',');
try
{
ch1Array.Add(Convert.ToDouble(values[1]));
//etc..
Now I am assuming I am using the List wrong (it's the first time I have used Lists). Once the program gets to that section (it compiles and runs fine) it tells me I have an error and that my input string was not in the correct format. But since I can use the variable values[1] in other contexts as a Double I don't know what I am doing that is throwing the error.
Can anyone tell me what I am doing wrong?
EDIT: Here is the exception thrown:

The problem is that you are not parsing the CSV file correctly. It looks something like this:
"Some value","3276",Something
When you split on ,, you end up with values[1] being "3276" (with the string having double quotes in it, not just 3276), which cannot be parsed as a number. I recommend you use an existing CSV library, e.g. FileHelpers or google for something else.
Another problem with splitting on , is if a value contains a comma, e.g.:
"Some, value","3276",Something
Will be split into "Some, value", "3276", and Something. You'd then be trying to parse value", which obviously won't work. For this reason I wouldn't recommend just doing a Replace to remove the quotes from your number.

You said your message box was showing the value to be "3276". This means you're trying to do Convert.ToDouble("\"3276\"") which will throw the exception you're getting.
Change the following line:
Convert.ToDouble(values[1])
To:
Convert.ToDouble(values[1].Replace("\"",""));

is there any way to ignore reading in certain lines in a text file?

I'm trying to read in a text file in a c# application, but I don't want to read the first two lines, or the last line. There's 8 lines in the file, so effectivly I just want to read in lines, 3, 4, 5, 6 and 7.
Is there any way to do this?
example file
_USE [Shelley's Other Database]
CREATE TABLE db.exmpcustomers(
fName varchar(100) NULL,
lName varchar(100) NULL,
dateOfBirth date NULL,
houseNumber int NULL,
streetName varchar(100) NULL
) ON [PRIMARY]_
EDIT
Okay, so, I've implemented Callum Rogers answer into my code and for some reason it works with my edited text file (I created a text file with the lines I didn't want to use omitted) and it does exactly what it should, but whenever I try it with the original text file (above) it throws an exception. I display this information in a DataGrid and I think that's where the exception is being thrown.
Any ideas?

The Answer by Rogers is good, I am just providing another way of doing this.
Try this,
List<string> list = new List<string>();
using (StreamReader reader = new StreamReader(FilePath))
{
string text = "";
while ((text = reader.ReadLine()) != null)
{
list.Add(text);
}
list.RemoveAt(0);
list.RemoveAt(0);
}
Hope this helps

Why do you want to ignore exactly the first two and the last line?
Depending on what your file looks like you might want to analyze the line, e.g. look at the first character whether it is a comment sign, or ignore everything until you find the first empty line, etc.
Sometimes, hardcoding "magic" numbers isn't such a good idea. What if the file format needs to be changed to contain 3 header lines?
As the other answers demonstrate: Nothing keeps you from doing what you ever want with a line you have read, so of course, you can ignore it, too.
Edit, now that you've provided an example of your file: For your case I'd definitely not use the hardcoded numbers approach. What if some day the SQL statement should contain another field, or if it appears on one instead of 8 lines?
My suggestion: Read in the whole string at once, then analyze it. Safest way would be to use a grammar, but if you presume the SQL statement is never going to be more complicated, you can use a regular expression (still much better than using line numbers etc.):
string content = File.ReadAllText(filename);
Regex r = new Regex(#"CREATE TABLE [^\(]+\((.*)\) ON");
string whatYouWant = r.Match(content).Groups[0].Value;

Why not just use File.ReadAllLines() and then remove the first 2 lines and the last line? With such a small file speed differences will not be noticeable.
string[] allLines = File.ReadAllLines("file.ext");
string[] linesWanted = new string[allLines.Length-3];
Array.Copy(allLines, 2, linesWanted, 0, allLines.Length-3);

If you have a TextReader object wrapping the filestream you could just call ReadLine() two times.
StreamReader inherits from TextReader, which is abstract.
Non-fool proof example:
using (var fs = new FileStream("blah", FileMode.Open))
using (var reader = new StreamReader(fs))
{
reader.ReadLine();
reader.ReadLine();
// Do stuff.
}

string filepath = #"C:\whatever.txt";
using (StreamReader rdr = new StreamReader(filepath))
{
rdr.ReadLine(); // ignore 1st line
rdr.ReadLine(); // ignore 2nd line
string fileContents = "";
while (true)
{
string line = rdr.ReadLine();
if (rdr.EndOfStream)
break; // finish without processing last line
fileContents += line + #"\r\n";
}
Console.WriteLine(fileContents);
}

How about a general solution?
To me, the first step is to enumerate over the lines of a file (already provided by ReadAllLines, but that has a performance cost due to populating an entire string[] array; there's also ReadLines, but that's only available as of .NET 4.0).
Implementing this is pretty trivial:
public static IEnumerable<string> EnumerateLines(this FileInfo file)
{
using (var reader = file.OpenText())
{
while (!reader.EndOfStream)
{
yield return reader.ReadLine();
}
}
}
The next step is to simply skip the first two lines of this enumerable sequence. This is straightforward using the Skip extension method.
The last step is to ignore the last line of the enumerable sequence. Here's one way you could implement this:
public static IEnumerable<T> IgnoreLast<T>(this IEnumerable<T> source, int ignoreCount)
{
if (ignoreCount < 0)
{
throw new ArgumentOutOfRangeException("ignoreCount");
}
var buffer = new Queue<T>();
foreach (T value in source)
{
if (buffer.Count < ignoreCount)
{
buffer.Enqueue(value);
continue;
}
T buffered = buffer.Dequeue();
buffer.Enqueue(value);
yield return buffered;
}
}
OK, then. Putting it all together, we have:
var file = new FileInfo(#"path\to\file.txt");
var lines = file.EnumerateLines().Skip(2).IgnoreLast(1);
Test input (contents of file):
This is line number 1.
This is line number 2.
This is line number 3.
This is line number 4.
This is line number 5.
This is line number 6.
This is line number 7.
This is line number 8.
This is line number 9.
This is line number 10.
Output (of Skip(2).IgnoreLast(1)):
This is line number 3.
This is line number 4.
This is line number 5.
This is line number 6.
This is line number 7.
This is line number 8.
This is line number 9.

You can do this:
var valid = new int[] { 3, 4, 5, 6, 7 };
var lines = File.ReadAllLines("file.txt").
Where((line, index) => valid.Contains(index + 1));
Or the opposite:
var invalid = new int[] { 1, 2, 8 };
var lines = File.ReadAllLines("file.txt").
Where((line, index) => !invalid.Contains(index + 1));
If you're looking for a general way to remove the last and the first 2, you can use this:
var allLines = File.ReadAllLines("file.txt");
var lines = allLines
.Take(allLines.Length - 1)
.Skip(2);
But from your example it seems that you're better off looking for the string pattern that you want to read from the file. Try using regexes.