Develop Reference

How to merge .txt files in c#? [duplicate]

using (StreamWriter writer = File.CreateText(FinishedFile))
{
int lineNum = 0;
while (lineNum < FilesLineCount.Min())
{
for (int i = 0; i <= FilesToMerge.Count() - 1; i++)
{
if (i != FilesToMerge.Count() - 1)
{
var CurrentFile = File.ReadLines(FilesToMerge[i]).Skip(lineNum).Take(1);
string CurrentLine = string.Join("", CurrentFile);
writer.Write(CurrentLine + ",");
}
else
{
var CurrentFile = File.ReadLines(FilesToMerge[i]).Skip(lineNum).Take(1);
string CurrentLine = string.Join("", CurrentFile);
writer.Write(CurrentLine + "\n");
}
}
lineNum++;
}
}
The current way i am doing this is just too slow. I am merging files that are each 50k+ lines long with various amounts of data.
for ex:
File 1
1
2
3
4
File 2
4
3
2
1
i need this to merge into being a third fileFile 3
1,4
2,3
3,2
4,1P.S. The user can pick as many files as they want from any locations.
Thanks for the help.

You approach is slow because of the Skip and Take in the loops.
You could use a dictionary to collect all line-index' lines:
string[] allFileLocationsToMerge = { "filepath1", "filepath2", "..." };
var mergedLists = new Dictionary<int, List<string>>();
foreach (string file in allFileLocationsToMerge)
{
string[] allLines = File.ReadAllLines(file);
for (int lineIndex = 0; lineIndex < allLines.Length; lineIndex++)
{
bool indexKnown = mergedLists.TryGetValue(lineIndex, out List<string> allLinesAtIndex);
if (!indexKnown)
allLinesAtIndex = new List<string>();
allLinesAtIndex.Add(allLines[lineIndex]);
mergedLists[lineIndex] = allLinesAtIndex;
}
}
IEnumerable<string> mergeLines = mergedLists.Values.Select(list => string.Join(",", list));
File.WriteAllLines("targetPath", mergeLines);

Here's another approach - this implementation only stores in memory one set of lines from each file simultaneously, thus reducing memory pressure significantly (if that is an issue).
public static void MergeFiles(string output, params string[] inputs)
{
var files = inputs.Select(File.ReadLines).Select(iter => iter.GetEnumerator()).ToArray();
StringBuilder line = new StringBuilder();
bool any;
using (var outFile = File.CreateText(output))
{
do
{
line.Clear();
any = false;
foreach (var iter in files)
{
if (!iter.MoveNext())
continue;
if (line.Length != 0)
line.Append(", ");
line.Append(iter.Current);
any = true;
}
if (any)
outFile.WriteLine(line.ToString());
}
while (any);
}
foreach (var iter in files)
{
iter.Dispose();
}
}
This also handles files of different lengths.

Producer/consumer doesn't generate expected results

I've written such producer/consumer code, which should generate big file filled with random data
class Program
{
static void Main(string[] args)
{
Random random = new Random();
String filename = #"d:\test_out";
long numlines = 1000000;
var buffer = new BlockingCollection<string[]>(10); //limit to not get OOM.
int arrSize = 100; //size of each string chunk in buffer;
String[] block = new string[arrSize];
Task producer = Task.Factory.StartNew(() =>
{
long blockNum = 0;
long lineStopped = 0;
for (long i = 0; i < numlines; i++)
{
if (blockNum == arrSize)
{
buffer.Add(block);
blockNum = 0;
lineStopped = i;
}
block[blockNum] = random.Next().ToString();
//null is sign to stop if last block is not fully filled
if (blockNum < arrSize - 1)
{
block[blockNum + 1] = null;
}
blockNum++;
};
if (lineStopped < numlines)
{
buffer.Add(block);
}
buffer.CompleteAdding();
}, TaskCreationOptions.LongRunning);
Task consumer = Task.Factory.StartNew(() =>
{
using (var outputFile = new StreamWriter(filename))
{
foreach (string[] chunk in buffer.GetConsumingEnumerable())
{
foreach (string value in chunk)
{
if (value == null) break;
outputFile.WriteLine(value);
}
}
}
}, TaskCreationOptions.LongRunning);
Task.WaitAll(producer, consumer);
}
}
And it does what is intended to do. But for some unknown reason it produces only ~550000 strings, not 1000000 and I can not understand why this is happening.
Can someone point on my mistake? I really don't get what's wrong with this code.

The buffer
String[] block = new string[arrSize];
is declared outside the Lambda. That means it is captured and re-used.
That would normally go unnoticed (you would just write out the wrong random data) but because your if (blockNum < arrSize - 1) is placed inside the for loop you regularly write a null into the shared buffer.
Exercise, instead of:
block[blockNum] = random.Next().ToString();
use
block[blockNum] = i.ToString();
and predict and verify the results.

New to c#, not understanding online compiler

so I wrote some C# code and I am trying to test it incrementally, do to something that would take a while to explain, but bottom line, I'm new to c# and not understanding the online compiler error messages. Here is the error message I get when I try and compile, but the strings look good to me.
string solutionSet = "white|black|beige|pink|green|blue|red|yellow|orange|cyan|purple|brown";
string[] solutionSetArray = new string[12];
string ret = "";
string delimeter = "|";
int tempPos = 0;
int counter = 0;
int successFlag = 0;
int patternLength = 5;
for (int index = 0; index < solutionSet.Length; index++)
{
if (solutionSet[index] == delimeter)
{
solutionSetArray[counter] = solutionSet.Substring(tempPos, index);
tempPos = index + 1;
counter++;
}
if (solutionSet.Length - index == 1)
{
solutionSetArray[solutionSetArray.Length-1] = solutionSet.Substring(tempPos, solutionSet.Length);
}
}
for (int i = 0; i < patternLength; i++)
{
Random rnd = new Random();
int randIndex = rnd.Next(solutionSetArray.Length);
if (i != patternLength - 1)
{
ret += solutionSetArray[randIndex] + "|";
successFlag++;
}
else
{
ret += solutionSetArray[randIndex];
}
}
if (successFlag == patternLength - 1)
{
Console.WriteLine(ret);
}
else
{
Console.WriteLine("ERROR");
}

The error (which, according to the message, is on line 1, column 11) is being caused by your very first line of code, which begins string.
I can't tell the context from just your post, but my guess is that you are declaring solutionSet in a block that is not inside of a class or function. You should enclose your code in a class or method, e.g.
public class MyClass
{
static public void Main()
{
string solutionSet = "white|black|beige|pink|green|blue|red|yellow|orange|cyan|purple|brown";
//Rest of code goes here
}
}
By the way, if you're trying to convert solutionSet to an array, you can just write
var solutionSetArray = solutionSet.Split("|");

the problem with your code is
solutionSetArray[counter] = solutionSet.Substring(tempPos, index);
after 6 iterations tempPos=34 and index=37 which is running out of bounds of solutionSet. I would suggest to use var solutionSetArray = solutionSet.Split("|"); and also use LinqPad which can be easy for you to debug if possible,.

StreamReader using arraylist or array

Here's my code:
StreamReader reader = new StreamReader("war.txt");
string input = null;
while ((input = reader.ReadLine()) != null)
{
Console.WriteLine(input);
}
reader.Close();
The program above reads and prints out from the file “war.txt” line-by-line. I need to re-write the program so that it prints out in reverse order, i.e., last line first and first line last. For example, if “war.txt” contains the following:
Hello.
How are you?
Thank you.
Goodbye.
The program should prints out:
Goodbye.
Thank you.
How are you?
Hello.
I am very new in C# please help! Thanks!

To do that, you are going to have to buffer the data anyway (unless you do some tricky work with the FileStream API to read the file backwards). How about just:
var lines = File.ReadAllLines("war.txt");
for(int i = lines.Length - 1; i >= 0; i--)
Console.WriteLine(lines[i]);
which just loads the file (in lines) into an array, and then prints the array starting from the end.
A LINQ version of that would be:
foreach(var line in File.ReadLines("war.txt").Reverse())
Console.WriteLine(line);
but frankly the array version is more efficient.

You can do it using recursion with something like this:
void printReverse(int n)
{
String line = reader.readLine();
if (n > 0)
printReverse(n-1);
System.out.println(line);
}

Have a look at adding the lines to a List, then using Reverse on the list and then maybe the ForEach to output the items.

Another option: store each line into a Stack as you read them. After reading the file, pop the stack to print the lines in reverse order.

with the enumerable extension functions, this can be done shorter:
foreach(var l in File.ReadAllLines("war.txt").Reverse())
Console.WriteLine(l);

try
File.ReadAllLines(myFile)
.Reverse();
full code
var list = File.ReadAllLines(filepath).Reverse().ToList();
foreach (var l in list)
Console.WriteLine(l);
Implementation detail
Enumerable.Reverse Method - Inverts the order of the elements in a sequence
File.ReadAllLines Method (String) - Opens a text file, reads all lines of the file, and then closes the file.

here is a example mate, remember to add "using System.IO"
try
{
const int Size = 7;
decimal[] numbers = new decimal[Size];
decimal total = 0m;
int index = 0;
StreamReader inputfile;
inputfile = File.OpenText("Sales.txt");
while (index < numbers.Length && !inputfile.EndOfStream)
{
numbers[index] = decimal.Parse(inputfile.ReadLine());
index++;
}
inputfile.Close();
foreach (decimal Sales in numbers)
{
outputlistBox1.Items.Add(Sales);
total = total + Sales;
}
textBox1.Text = total.ToString();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}

Here is another example from a textbook i bought few years ago, this has highest/lowest/average scores..(remember to use 'using System IO;)
private double Average(int[] iArray)
{
int total = 0;
double Average;
for (int index = 0; index < iArray.Length;
index++)
{
total += iArray[index];
}
Average = (double) total / iArray.Length;
return Average;
}
private int Highest(int[] iArray)
{
int highest = iArray[0];
for (int index = 1; index < iArray.Length; index++)
{
if (iArray[index] > highest)
{
highest = iArray[index];
}
}
return highest;
}
private int Lowest(int[] iArray)
{
int lowest = iArray[0];
for (int index = 1; index < iArray.Length; index++)
{
if (iArray[index] < lowest)
{
lowest = iArray[index];
}
}
return lowest;
}
private void button1_Click(object sender, System.EventArgs e)
{
try
{
const int SIZE = 5;
int[] Scores = new int [SIZE];
int index = 0;
int highestScore;
int lowestScore;
double averageScore;
StreamReader inputFile;
inputFile = File.OpenText("C:\\Users\\Asus\\Desktop\\TestScores.txt");
while (!inputFile.EndOfStream && index < Scores.Length)
{
Scores[index] = int.Parse(inputFile.ReadLine());
index++;
}
inputFile.Close();
foreach (int value in Scores)
{
listBox1.Items.Add(value);
}
highestScore = Highest(Scores);
lowestScore = Lowest(Scores);
averageScore = Average(Scores);
textBox1.Text = highestScore.ToString();
textBox2.Text = lowestScore.ToString();
textBox3.Text = averageScore.ToString("n1");
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
private void button2_Click(object sender, EventArgs e)
{
this.Close();
}
}
}

Read double value from a file C#

I have a txt file that the format is:
0.32423 1.3453 3.23423
0.12332 3.1231 9.23432432
9.234324234 -1.23432 12.23432
...
Each line has three double value. There are more than 10000 lines in this file. I can use the ReadStream.ReadLine and use the String.Split, then convert it.
I want to know is there any faster method to do it.
Best Regards,

StreamReader.ReadLine, String.Split and Double.TryParse sounds like a good solution here.
No need for improvement.

There may be some little micro-optimisations you can perform, but the way you've suggested sounds about as simple as you'll get.
10000 lines shouldn't take very long - have you tried it and found you've actually got a performance problem? For example, here are two short programs - one creates a 10,000 line file and the other reads it:
CreateFile.cs:
using System;
using System.IO;
public class Test
{
static void Main()
{
Random rng = new Random();
using (TextWriter writer = File.CreateText("test.txt"))
{
for (int i = 0; i < 10000; i++)
{
writer.WriteLine("{0} {1} {2}", rng.NextDouble(),
rng.NextDouble(), rng.NextDouble());
}
}
}
}
ReadFile.cs:
using System;
using System.Diagnostics;
using System.IO;
using System.Linq;
public class Test
{
static void Main()
{
Stopwatch sw = Stopwatch.StartNew();
using (TextReader reader = File.OpenText("test.txt"))
{
string line;
while ((line = reader.ReadLine()) != null)
{
string[] bits = line.Split(' ');
foreach (string bit in bits)
{
double value;
if (!double.TryParse(bit, out value))
{
Console.WriteLine("Bad value");
}
}
}
}
sw.Stop();
Console.WriteLine("Total time: {0}ms",
sw.ElapsedMilliseconds);
}
}
On my netbook (which admittedly has an SSD in) it only takes 82ms to read the file. I would suggest that's probably not a problem :)

I would suggest reading all your lines at once with
string[] lines = System.IO.File.ReadAllLines(fileName);
This wold ensure that the I/O is done with the maximum efficiency. You woul have to measure (profile) but I would expect the conversions to take far less time.

your method is already good!
you can improve it by writing a readline function that returns an array of double and you reuse this function in other programs.

This solution is a little bit slower (see benchmarks at the end), but its nicer to read. It should also be more memory efficient because only the current character is buffered at the time (instead of the whole file or line).
Reading arrays is an additional feature in this reader which assumes that the size of the array always comes first as an int-value.
IParsable is another feature, that makes it easy to implement Parse methods for various types.
class StringSteamReader {
private StreamReader sr;
public StringSteamReader(StreamReader sr) {
this.sr = sr;
this.Separator = ' ';
}
private StringBuilder sb = new StringBuilder();
public string ReadWord() {
eol = false;
sb.Clear();
char c;
while (!sr.EndOfStream) {
c = (char)sr.Read();
if (c == Separator) break;
if (IsNewLine(c)) {
eol = true;
char nextch = (char)sr.Peek();
while (IsNewLine(nextch)) {
sr.Read(); // consume all newlines
nextch = (char)sr.Peek();
}
break;
}
sb.Append(c);
}
return sb.ToString();
}
private bool IsNewLine(char c) {
return c == '\r' || c == '\n';
}
public int ReadInt() {
return int.Parse(ReadWord());
}
public double ReadDouble() {
return double.Parse(ReadWord());
}
public bool EOF {
get { return sr.EndOfStream; }
}
public char Separator { get; set; }
bool eol;
public bool EOL {
get { return eol || sr.EndOfStream; }
}
public T ReadObject<T>() where T : IParsable, new() {
var obj = new T();
obj.Parse(this);
return obj;
}
public int[] ReadIntArray() {
int size = ReadInt();
var a = new int[size];
for (int i = 0; i < size; i++) {
a[i] = ReadInt();
}
return a;
}
public double[] ReadDoubleArray() {
int size = ReadInt();
var a = new double[size];
for (int i = 0; i < size; i++) {
a[i] = ReadDouble();
}
return a;
}
public T[] ReadObjectArray<T>() where T : IParsable, new() {
int size = ReadInt();
var a = new T[size];
for (int i = 0; i < size; i++) {
a[i] = ReadObject<T>();
}
return a;
}
internal void NextLine() {
eol = false;
}
}
interface IParsable {
void Parse(StringSteamReader r);
}
It can be used like this:
public void Parse(StringSteamReader r) {
double x = r.ReadDouble();
int y = r.ReadInt();
string z = r.ReadWord();
double[] arr = r.ReadDoubleArray();
MyParsableObject o = r.ReadObject<MyParsableObject>();
MyParsableObject [] oarr = r.ReadObjectArray<MyParsableObject>();
}
I did some benchmarking, comparing StringStreamReader with some other approaches, already proposed (StreamReader.ReadLine and File.ReadAllLines). Here are the methods I used for benchmarking:
private static void Test_StringStreamReader(string filename) {
var sw = new Stopwatch();
sw.Start();
using (var sr = new StreamReader(new FileStream(filename, FileMode.Open, FileAccess.Read))) {
var r = new StringSteamReader(sr);
r.Separator = ' ';
while (!r.EOF) {
var dbls = new List<double>();
while (!r.EOF) {
dbls.Add(r.ReadDouble());
}
}
}
sw.Stop();
Console.WriteLine("elapsed: {0}", sw.Elapsed);
}
private static void Test_ReadLine(string filename) {
var sw = new Stopwatch();
sw.Start();
using (var sr = new StreamReader(new FileStream(filename, FileMode.Open, FileAccess.Read))) {
var dbls = new List<double>();
while (!sr.EndOfStream) {
string line = sr.ReadLine();
string[] bits = line.Split(' ');
foreach(string bit in bits) {
dbls.Add(double.Parse(bit));
}
}
}
sw.Stop();
Console.WriteLine("elapsed: {0}", sw.Elapsed);
}
private static void Test_ReadAllLines(string filename) {
var sw = new Stopwatch();
sw.Start();
string[] lines = System.IO.File.ReadAllLines(filename);
var dbls = new List<double>();
foreach(var line in lines) {
string[] bits = line.Split(' ');
foreach (string bit in bits) {
dbls.Add(double.Parse(bit));
}
}
sw.Stop();
Console.WriteLine("Test_ReadAllLines: {0}", sw.Elapsed);
}
I used a file with 1.000.000 lines of double values (3 values each line). File is located on a SSD disk and each test was repeated multiple times in release-mode. These are the results (on average):
Test_StringStreamReader: 00:00:01.1980975
Test_ReadLine: 00:00:00.9117553
Test_ReadAllLines: 00:00:01.1362452
So, as mentioned StringStreamReader is a bit slower than the other approaches. For 10.000 lines, the performance is around (120ms / 95ms / 100ms).