Develop Reference

Delimit a string by character unless within quotation marks C#

I need to demilitarise text by a single character, a comma. But I want to only use that comma as a delimiter if it is not encapsulated by quotation marks.
An example:
Method,value1,value2
Would contain three values: Method, value1 and value2
But:
Method,"value1,value2"
Would contain two values: Method and "value1,value2"
I'm not really sure how to go about this as when splitting a string I would use:
String.Split(',');
But that would demilitarise based on ALL commas. Is this possible without getting overly complicated and having to manually check every character of the string.
Thanks in advance

Copied from my comment: Use an available csv parser like VisualBasic.FileIO.TextFieldParser or this or this.
As requested, here is an example for the TextFieldParser:
var allLineFields = new List<string[]>();
string sampleText = "Method,\"value1,value2\"";
var reader = new System.IO.StringReader(sampleText);
using (var parser = new Microsoft.VisualBasic.FileIO.TextFieldParser(reader))
{
parser.Delimiters = new string[] { "," };
parser.HasFieldsEnclosedInQuotes = true; // <--- !!!
string[] fields;
while ((fields = parser.ReadFields()) != null)
{
allLineFields.Add(fields);
}
}
This list now contains a single string[] with two strings. I have used a StringReader because this sample uses a string, if the source is a file use a StreamReader(f.e. via File.OpenText).

You can try Regex.Split() to split the data up using the pattern
",|(\"[^\"]*\")"
This will split by commas and by characters within quotes.
Code Sample:
using System;
using System.Linq;
using System.Text.RegularExpressions;
public class Program
{
public static void Main()
{
string data = "Method,\"value1,value2\",Method2";
string[] pieces = Regex.Split(data, ",|(\"[^\"]*\")").Where(exp => !String.IsNullOrEmpty(exp)).ToArray();
foreach (string piece in pieces)
{
Console.WriteLine(piece);
}
}
}
Results:
Method
"value1,value2"
Method2
Demo

Remove a specific column from a delimited file

I've been working with some big delimited text (~1GB) files these days. It looks like somewhat below
COlumn1 #COlumn2#COlumn3#COlumn4
COlumn1#COlumn2#COlumn3 #COlumn4
where # is the delimiter.
In case a column is invalid I might have to remove it from the whole text file. The output file when Column 3 is invalid should look like this.
COlumn1 #COlumn2#COlumn4
COlumn1#COlumn2#COlumn4
string line = "COlumn1# COlumn2 #COlumn3# COlumn4";
int junk =3;
int columncount = line.Split(new char[] { '#' }, StringSplitOptions.None).Count();
//remove the [junk-1]th '#' and the value till [junk]th '#'
//"COlumn1# COlumn2 # COlumn4"
I's not able to find a c# version of this in SO. Is there a way I can do that? Please help.
EDIT:
The solution which I found myself is like below which does the job. Is there a way I could modify this to a better way so that it narrows down the performance impact it might have in case of large text files?
int junk = 3;
string line = "COlumn1#COlumn2#COlumn3#COlumn4";
int counter = 0;
int colcount = line.Split(new char[] { '#' }, StringSplitOptions.None).Length - 1;
string[] linearray = line.Split(new char[] { '#' }, StringSplitOptions.None);
List<string> linelist = linearray.ToList();
linelist.RemoveAt(junk - 1);
string finalline = string.Empty;
foreach (string s in linelist)
{
counter++;
finalline += s;
if (counter < colcount)
finalline += "#";
}
Console.WriteLine(finalline);

EDITED
This method can be very memory expensive, as your can read in this post, the suggestion should be:
If you need to run complex queries against the data in the file, the right thing to do is to load the data to database and let DBMS to take care of data retrieval and memory management.
To avoid memory consumption you should use a StreamReader to read file line by line
This could be a start for your task, missing your invalid match logic
using System.Collections.Generic;
using System.IO;
using System.Text;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
const string fileName = "temp.txt";
var results = FindInvalidColumns(fileName);
using (var reader = File.OpenText(fileName))
{
while (!reader.EndOfStream)
{
var builder = new StringBuilder();
var line = reader.ReadLine();
if (line == null) continue;
var split = line.Split(new[] { "#" }, 0);
for (var i = 0; i < split.Length; i++)
if (!results.Contains(i))
builder.Append(split[i]);
using (var fs = new FileStream("new.txt", FileMode.Append, FileAccess.Write))
using (var sw = new StreamWriter(fs))
{
sw.WriteLine(builder.ToString());
}
}
}
}
private static List<int> FindInvalidColumns(string fileName)
{
var invalidColumnIndexes = new List<int>();
using (var reader = File.OpenText(fileName))
{
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
if (line == null) continue;
var split = line.Split(new[] { "#" }, 0);
for (var i = 0; i < split.Length; i++)
{
if (IsInvalid(split[i]) && !invalidColumnIndexes.Contains(i))
invalidColumnIndexes.Add(i);
}
}
}
return invalidColumnIndexes;
}
private static bool IsInvalid(string s)
{
return false;
}
}
}

First, what you will do is re-write the line to a text file using a 0-length string for COlumn3. Therefore the line after being written correctly would look like this:
COlumun1#COlumn2##COlumn4
As you can see, there are two delimiters between COlumn2 and COlumn4. This is a cell with no data in it. (By "cell" I mean one column of a certain, single row.) Later, when some other process reads this using the Split function, it will still create a new value for Column 3, but in the array generated by Split, the 3rd position would be an empty string:
String[] columns = stream_reader.ReadLine().Split('#');
int lengthOfThirdItem = columns[2].Length; // for proof
// lengthOfThirdItem = 0
This reduces invalid values to null and persists them back in the text file.
For more on String.Split see C# StreamReader save to Array with separator.
It is not possible to write to lines internal to a text file while it is also open for read. This article discusses it some (simultaneous read-write a file in C#), but it looks like that question-asker just wants to be able to write lines to the end. You want to be able to write lines at any point in the interior. I think this is not possible without buffering the data in some way.
The simplest way to buffer the data is rename the file to a temp file first (using File.CoMovepy() // http://msdn.microsoft.com/en-us/library/system.io.file.move(v=vs.110).aspx). Then use the temp file as the data source. Just open the temp file that to read in the data which may have corrupt entries, and write the data afresh to the original file name using the approach I describe above to represent empty columns. After this is complete, then you should delete the temp file.
Important
Deleting the temp file may leave you vulnerable to power and data transients (or software 'transients'). (I.e., a power drop that interrupts part of the process could leave the data in an unusable state.) So you may also want to leave the temp file on the drive as an emergency backup in case of some problem.

Writing List<String> contents to text file after deleting string

I'm trying to get the contents of a Text File, delete a line of string, and re-write back to the Text File, deleting the line of string. I'm using StreamReader to get the text, importing into a List, removing the string, then rewriting using StreamWriter. My problems arises somewhere around the removing or writing of the string. Instead of writing back the existing, non deleted contents to the text file, all the text is replaced with :
System.Collections.Generic.List`1[System.String]
My code for this function is as follows:
{
for (int i = deleteDevice.Count - 1; i >= 0; i--)
{
string split = "";
//deleteDevice[i].Split(',').ToString();
List<string> parts = split.Split(',').ToList();
if (parts.Contains(deviceList.SelectedItem.ToString()))
{
deleteDevice.Remove(i.ToString());
}
}
if (deleteDevice.Count != 0) //Error Handling
{
writer.WriteLine(deleteDevice);
}
}
deviceList.Items.Remove(deviceList.SelectedItem);
}
I would just like the script to write back any string that isn't deleted (If there is any), without replacing it. Any help is appreciated, Cheers

You can read all the info from the text file into a list and then remove from the list and rewrite that to the text file.
I would change the list 'deleteDevice' to store a string array instead and use the code below to determine which item to remove.
List<int> toRemove = new List<int>();
int i = 0;
/*build a list of indexes to remove*/
foreach (string[] x in deleteDevice)
{
if (x[0].Contains(deviceList.SelectedItem.ToString()))
{
toRemove.Add(i);
}
i++;
}
/*Remove items from list*/
foreach (int fd in toRemove)
deleteDevice.RemoveAt(fd);
/*write to text file*/
using (StreamWriter writer = new StreamWriter("Devices.txt"))
{
if (deleteDevice.Count != 0) //Error Handling
{
foreach (string[] s in deleteDevice)
{
StringBuilder sb = new StringBuilder();
for (int fds = 0; fds < s.Length; fds++ )
{
sb.Append(s[fds] + ",");
}
string line = sb.ToString();
writer.WriteLine(line.Substring(0, line.Length - 1));
}
}
}
This isn't the best solution but should work for your needs. There's probably a much easier way of doing this.

The problem is in the following line:
writer.WriteLine(deleteDevice);
You're writing deleteDevice (I assume this is of type List). List.ToString() returns the type name of the list, because this has no specific implementation. What you want is
foreach(String s in deleteDevice)
{
writer.WriteLine(s);
}

Problems
deleteDevice is of type List<string>, and because it also doesn't overload ToString(), the default behaviour of List<string>.ToString() is to return the name of the type.
Hence your line writer.WriteLine(deleteDevice); writes the string System.Collections.Generic.List1[System.String]`.
Other than that, there are many things wrong with your code...
For example, you do this:
string split = "";
and then on the line afterwards you do this:
List<string> parts = split.Split(',').ToList();
But because split is "", this will always return an empty list.
Solution
To simplify the code, you could first write a helper method that will remove from a file all the lines that match a specified predicate:
public void RemoveUnwantedLines(string filename, Predicate<string> unwanted)
{
var lines = File.ReadAllLines(filename);
File.WriteAllLines(filename, lines.Where(line => !unwanted(line)));
}
Then you can write the predicate something like this (this might not be quite right; I don't really know exactly what your code is doing because it's not compilable and omits some of the types):
string filename = "My Filename";
string deviceToRemove= deviceList.SelectedItem.ToString();
Predicate<string> unwanted = line =>
line.Split(new [] {','})
.Contains(deviceToRemove);
RemoveUnwantedLines(filename, unwanted);

C# Template parsing and matching with text file

Need some ideas how to solve this problem.
I have a template file what describes the line in the text file. For example:
Template
[%f1%]|[%f2%]|[%f3%]"[%f4%]"[%f5%]"[%f6%]
Text file
1234|1234567|123"12345"12"123456
Now i need to read in the fields from the text file. In the template file fields are described with [%some name%]. Allso in the template file there is set what the field separators are, in this example here there are | and ". The lenght of the fields can change through different files but the separators will stay the same. What would be the best way to read in the template and by template read in the text file?
EDIT: Text file has multiple rows, like this:
1234|1234567|123"12345"12"123456"\r\n
1234|field|123"12345"12"asdasd"\r\n
123sd|1234567|123"asdsadf"12"123456"\r\n
45gg|somedata|123"12345"12"somefield"\r\n
EDIT2: Ok, lets make it even harder. Some fields can contain binary data and i know the starting and end position of the binary data field. I should be able to mark those fields in the template and then the parser will know that this field is binary. How to solve this problem?

I would create a regex based on the template and then parse the text file using that:
class Parser
{
private static readonly Regex TemplateRegex =
new Regex(#"\[%(?<field>[^]]+)%\](?<delim>[^[]+)?");
readonly List<string> m_fields = new List<string>();
private readonly Regex m_textRegex;
public Parser(string template)
{
var textRegexString = '^' + TemplateRegex.Replace(template, Evaluator) + '$';
m_textRegex = new Regex(textRegexString);
}
string Evaluator(Match match)
{
// add field name to collection and create regex for the field
var fieldName = match.Groups["field"].Value;
m_fields.Add(fieldName);
string result = "(.*?)";
// add delimiter to the regex, if it exists
// TODO: check, that only last field doesn't have delimiter
var delimGroup = match.Groups["delim"];
if (delimGroup.Success)
{
string delim = delimGroup.Value;
result += Regex.Escape(delim);
}
return result;
}
public IDictionary<string, string> Parse(string text)
{
var match = m_textRegex.Match(text);
var groups = match.Groups;
var result = new Dictionary<string, string>(m_fields.Count);
for (int i = 0; i < m_fields.Count; i++)
result.Add(m_fields[i], groups[i + 1].Value);
return result;
}
}

You can parse the template using regular expressions. An expression like this will match each field definition and separator:
Match m = Regex.Match(template, #"^(\[%(?<name>.+?)%\](?<separator>.)?)+$")
The match will contain two named groups for (name and separator), each of which will contain a number of captures for each time they matched in the input string. In your example, the separator group would have one less capture than the name group.
You can then iterate over the captures, and use the results to extract the fields from the input string and store the values, like this:
if( m.Success )
{
Group name = m.Groups["name"];
Group separator = m.Groups["separator"];
int index = 0;
Dictionary<string, string> fields = new Dictionary<string, string>();
for( int x = 0; x < name.Captures.Count; ++x )
{
int separatorIndex = input.Length;
if( x < separator.Captures.Count )
separatorIndex = input.IndexOf(separator.Captures[x].Value, index);
fields.Add(name.Captures[x].Value, input.Substring(index, separatorIndex - index));
index = separatorIndex + 1;
}
// Do something with results.
}
Obviously in a real program you'd have to account for invalid input and such, which I didn't do here.

I would do this with a few lines of code. Loop through your template row, grabbing all text between "[" as the variable name and everything else as a terminator. Read all the text to the terminal, assign it to the variable name, repeat.

1- Use API for that sscanf(line, format, __arglist) check here
2- Use string split Like:
public IEnumerable<int> GetDataFromLines(string[] lines)
{
//handle the output data
List<int> data = new List<int>();
foreach (string line in lines)
{
string[] seperators = new string[] { "|", "\"" };
string[] results = line.Split(seperators, StringSplitOptions.RemoveEmptyEntries);
foreach (string result in results)
{
data.Add(int.Parse(result));
}
}
return data;
}
Test it with line:
line = "1234|1234567|123\"12345\"12\"123456";
string[] lines = new string[] { line };
GetDataFromLines(lines);
//output list items are:
1234
1234567
123
12345
12
123456

asp.net Convert CSV string to string[]

Is there an easy way to convert a string from csv format into a string[] or list?
I can guarantee that there are no commas in the data.

String.Split is just not going to cut it, but a Regex.Split may - Try this one:
using System.Text.RegularExpressions;
string[] line;
line = Regex.Split( input, ",(?=(?:[^\"]*\"[^\"]*\")*(?![^\"]*\"))");
Where 'input' is the csv line. This will handle quoted delimiters, and should give you back an array of strings representing each field in the line.

If you want robust CSV handling, check out FileHelpers

string[] splitString = origString.Split(',');
(Following comment not added by original answerer)
Please keep in mind that this answer addresses the SPECIFIC case where there are guaranteed to be NO commas in the data.

Try:
Regex rex = new Regex(",(?=([^\"]*\"[^\"]*\")*(?![^\"]*\"))");
string[] values = rex.Split( csvLine );
Source: http://weblogs.asp.net/prieck/archive/2004/01/16/59457.aspx

You can take a look at using the Microsoft.VisualBasic assembly with the
Microsoft.VisualBasic.FileIO.TextFieldParser
It handles CSV (or any delimiter) with quotes. I've found it quite handy recently.

There isn't a simple way to do this well, if you want to account for quoted elements with embedded commas, especially if they are mixed with non-quoted fields.
You will also probably want to convert the lines to a dictionary, keyed by the column name.
My code to do this is several hundred lines long.
I think there are some examples on the web, open source projects, etc.

Try this;
static IEnumerable<string> CsvParse(string input)
{
// null strings return a one-element enumeration containing null.
if (input == null)
{
yield return null;
yield break;
}
// we will 'eat' bits of the string until it's gone.
String remaining = input;
while (remaining.Length > 0)
{
if (remaining.StartsWith("\"")) // deal with quotes
{
remaining = remaining.Substring(1); // pass over the initial quote.
// find the end quote.
int endQuotePosition = remaining.IndexOf("\"");
switch (endQuotePosition)
{
case -1:
// unclosed quote.
throw new ArgumentOutOfRangeException("Unclosed quote");
case 0:
// the empty quote
yield return "";
remaining = remaining.Substring(2);
break;
default:
string quote = remaining.Substring(0, endQuotePosition).Trim();
remaining = remaining.Substring(endQuotePosition + 1);
yield return quote;
break;
}
}
else // deal with commas
{
int nextComma = remaining.IndexOf(",");
switch (nextComma)
{
case -1:
// no more commas -- read to end
yield return remaining.Trim();
yield break;
case 0:
// the empty cell
yield return "";
remaining = remaining.Substring(1);
break;
default:
// get everything until next comma
string cell = remaining.Substring(0, nextComma).Trim();
remaining = remaining.Substring(nextComma + 1);
yield return cell;
break;
}
}
}
}

CsvString.split(',');

Get a string[] of all the lines:
string[] lines = System.IO.File.ReadAllLines("yourfile.csv");
Then loop through and split those lines (this error prone because it doesn't check for commas in quote-delimited fields):
foreach (string line in lines)
{
string[] items = line.Split({','}};
}

string s = "1,2,3,4,5";
string myStrings[] = s.Split({','}};
Note that Split() takes an array of characters to split on.

Some CSV files have double quotes around the values along with a comma. Therefore sometimes you can split on this string literal: ","

A Csv file with Quoted fields, is not a Csv file. Far more things (Excel) output without quotes rather than with quotes when you select "Csv" in a save as.
If you want one you can use, free, or commit to, here's mine that also does IDataReader/Record. It also uses DataTable to define/convert/enforce columns and DbNull.
http://github.com/claco/csvdatareader/
It doesn't do quotes.. yet. I just tossed it together a few days ago to scratch an itch.
Forgotten Semicolon: Nice link. Thanks.
cfeduke: Thanks for the tip to Microsoft.VisualBasic.FileIO.TextFieldParser. Going into CsvDataReader tonight.

http://github.com/claco/csvdatareader/ updated using TextFieldParser suggested by cfeduke.
Just a few props away from exposing separators/trimspaces/type ig you just need code to steal.

I was already splitting on tabs so this did the trick for me:
public static string CsvToTabDelimited(string line) {
var ret = new StringBuilder(line.Length);
bool inQuotes = false;
for (int idx = 0; idx < line.Length; idx++) {
if (line[idx] == '"') {
inQuotes = !inQuotes;
} else {
if (line[idx] == ',') {
ret.Append(inQuotes ? ',' : '\t');
} else {
ret.Append(line[idx]);
}
}
}
return ret.ToString();
}

string test = "one,two,three";
string[] okNow = test.Split(',');

separationChar[] = {';'}; // or '\t' ',' etc.
var strArray = strCSV.Split(separationChar);

string[] splitStrings = myCsv.Split(",".ToCharArray());