Develop Reference

OutOfMemoryException while trying to read a txt file that is over 2GB

In my code I recover the file, extract the text, manipulate it and write the modified string in the file, I have not had any problems to date, the file I had to manipulate today weighed over 2GB, with over 1 million lines
public static void ModifyFile(string directory, string filename)
{
string input = string.Empty;
using (StreamReader reader = new StreamReader(directory + filename))
{
input = reader.ReadToEnd();
}
string output = Manipulate(input);
File.WriteAllText($"{directory}{filename}", String.Empty);
WriteFile(directory, filename, output);
}
private static void WriteFile(string directory, string filename, string output)
{
using (StreamWriter writer = new StreamWriter(directory + filename, true))
{
{
writer.Write(output);
}
writer.Close();
}
}
private static string Manipulate(string input)
{
var counter = 1;
StringBuilder output = new StringBuilder();
string[] subs = input.Split(new string[] { Environment.NewLine }, StringSplitOptions.None);
foreach (var x in subs)
{
if (subs[subs.Length - 1] != x && subs[subs.Length - 2] != x)
{
var column = x.Substring(121, 2);
if (column.Equals("NA"))
{
var c = x.Substring(22, 9);
output.Append(ManipulateStringElement(x, counter, 22)
.Replace("\r\n", "\n").Replace("\r", "\n").Replace("\n", "\r\n"));
output.Append("\n");
counter++;
}
}
else if (subs[subs.Length - 2] == x)
{
output.Append(ManipulateStringElement(x, counter, 22)
.Replace("\r\n", "\n").Replace("\r", "\n").Replace("\n", "\r\n"));
}
}
return output.ToString();
}
private static string ManipulateStringElement(string item, int counter, int start)
{
return item.Replace(item.Substring(start, 9), GenerateProgressive(counter));
}
private static string GenerateProgressive(int counter)
{
return $"{counter}".PadLeft(9, '0');
}
But while running reader.ReadToEnd() I get "OutOfMemoryException" error, which makes me think the file is too big
The application is in .NET Framewrok 4.6.1, the operating system is 64bit (I had read that it could affect)

You need to do this in a streaming fashion in order to reduce memory consumption.
Open an input and an output file at the same time, and immediately output the result of a single line from Manipulate(). Ensure it ends with your custom newline character.
Finally replace the original file with the new one.
public static void ModifyFile(string directory, string filename)
{
string inputFile = Path.Combine(directory, filename);
string outputFile = Path.Combine(directory, filename + ".new");
using (var reader = new StreamReader(inputFile))
using (var reader = new StreamWriter(outputFile, true))
{
string input;
while((input = reader.ReadLine()) != null)
{
string output = Manipulate(input);
writer.Write(output);
}
}
File.Move(outputFile, inputFile, true);
}
You may also want to do this using async code, which could improve responsiveness.
I note that you are also retrieving the last two lines of the file. I suggest you do this separately, using this answer for example.
There are also other performance improvements you can make. For example:
private static string GenerateProgressive(int counter)
{
return counter.ToString("D9");
}
as well as:
private static string ManipulateStringElement(string item, int counter, int start)
{
return GenerateProgressive(counter) + item.Substring(9);
}

Writing a certain amount of lines in a file C#

I am currently writing a program that takes in a file, loops through all of the lines.
The file contains a lot of variables + values in this format:
Message =
"alfjawejf1ij4l2jr183fhaalfjawejf1ij4l2jr183fhahalfjawejf1ij4l2jr183fhahalfjawejf1ij4l2jr183fhahalfjawejf1ij4l2jr183fhahh" //the string will encompass multiple
lines of length
Answer = ?
My program will modify the value within message and write it in a new file.
How do I store multiple lines of the value into one string (so I can modify it)?
I need it so that it recognizes "message", starts storing the next few lines, detects "answer" and stops.
For the string holding the message value, I believe some sort of concatenation will be used (concatenating multiple lines).
string[] file = System.IO.File.ReadAllLines(#path); //file is read
string pathNew = Path.GetDirectoryName(path) + "\\completed_" + Path.GetFileName(path);
using (StreamWriter writer = File.CreateText(#pathNew))
{
foreach (string line in file)
{
for (int i = 0; i < line.Length; i++)
{
if (line.Substring(0,6).Equals("Msg = "))
{
foreach (string msg in file)
{
}
}
}
}
}

You could create it pretty easily, if you would provide a method that returns an IEnumerable<string>, and then use the File.WriteAllLines( string, IEnumerable<string> ) utility method
A way you could do it could be
private IEnumerable<string> ReturnAllMessages( IEnumerable<string> lines )
{
bool isMessage = false;
foreach (var line in lines)
{
if (line.StartsWith('Msg ='))
{
isMessage = true;
// set a flag that the next lines are part of the message
// this would exclude the rest of the line from the results
// if you want it, you could use:
// yield return line.Substring('Msg ='.Length));
continue;
}
if (line.StartsWith('Answer ='))
{
// remove the flag
isMessage = false;
continue;
}
if (isMessage)
{
// yield a line that is a message
yield return line;
}
}
}
and then use the method in the following way
File.WriteAllLines( #newPath, ReturnAllMessages( File.ReadAllLines( #path ) ) );
(didn't really test the code though, so just use it as a reference ;) )

One way to do this would be to read all the text as a single string using File.ReadAllText, and then split it on "Message =" to get all the messages, and then split each message on "Answer = " to get the answer for each message (assuming the format is "Message = some message Answer = some answer Message = some other message Answer = some other answer").
It would be helpful if you included an actual sample from the file, since your code clearly was not written for the sample you've provided (there is no line where line.Substring(0,6).Equals("Msg = ") is true).
I've included a method in my sample below that creates a text file with a multi-line message, and then showed how you can read the message into a variable and display it in the console window.
Hope this helps:
private static void CreateFile(string filePath)
{
if (File.Exists(filePath)) File.Delete(filePath);
Directory.CreateDirectory(Path.GetDirectoryName(filePath));
var fileLines = new List<string>
{
"Message = ",
"I weigh nothing, but you can still see me.",
"If you put me in a bucket, I make the bucket lighter.",
"What am I?",
"Answer = A hole",
"Message = ",
"What’s the difference between",
"a hippo and a Zippo?",
"Answer = ",
"A hippo is really heavy, ",
"and a Zippo is a little lighter."
};
File.WriteAllLines(filePath, fileLines);
}
private static void Main()
{
// Set this to a file that doesn't exist or that you don't care about
var filePath = #"f:\private\temp\temp.txt";
// Create a file with multi-line messages
CreateFile(filePath);
// Read all the file text
var fileText = File.ReadAllText(filePath);
// Split it into the message/answers
var messageAnswers = fileText.Split(new[] {"Message ="},
StringSplitOptions.RemoveEmptyEntries);
// Split each message into a message/answer array
foreach (var messageAnswer in messageAnswers)
{
var parts = messageAnswer.Split(new[] {"Answer ="},
StringSplitOptions.RemoveEmptyEntries);
var message = parts[0].Trim();
var answer = parts.Length > 1 ? parts[1].Trim() : "";
Console.WriteLine(message);
var userResponse = Console.ReadLine().Trim();
if (userResponse.Equals(answer, StringComparison.OrdinalIgnoreCase))
{
Console.WriteLine("Correct!");
}
else
{
Console.WriteLine("The actual answer is: " + answer);
}
}
GetKeyFromUser("\nDone! Press any key to exit...");
}

C# detect quotes in a text file

I try to detect quotes in a loaded text file but it is not working. I have tried with '"' and '\"' without success. Any suggestion? thanks
void read()
{
txt = File.ReadAllText("txt/txttst");
for(int i=0;i<txt.Length;i++)
{
if(txt[i]=='"')
{
Debug.Log("Quotes at "+i);
}
}
}

How about this
string[] lines = File.ReadAllLines(#"txt/txttst");
for (int i=0;i<lines.Length;i++)
{
string line = lines[i];
// ASCII Code of Quotes is 34
var bytes = Encoding.UTF8.GetBytes(line.ToCharArray()).ToList();
if(bytes.Count(b=> b.ToString()=="34")>0)
Console.WriteLine("\"" + "at line " + (i + 1));
}

This is how you can do it, please see the code and screenshot below. Hope it helps.
namespace TestConsoleApp
{
class Program
{
static void Main(string[] args)
{
string txt = File.ReadAllText(#"C:\Users\Public\TestFolder\test.txt");
string[] lines = File.ReadAllLines(#"C:\Users\Public\TestFolder\test.txt");
var reg = new Regex("\"");
Console.WriteLine("Contents of test.txt are; ");
foreach (string line in lines)
{
Console.WriteLine(line);
var matches = reg.Matches(line);
foreach (var item in matches)
{
Console.WriteLine("Quotes at "+ ((System.Text.RegularExpressions.Capture)item).Index);
}
}
}
}
}

Ok I found the problem, my text editor did a subtle auto-correct from " to “ . Cheers.

Update string at a specific line so it doesn't update duplicates in string?

I know how to go to a specific line but I don't know how to update that specific line in the string. I have tried the Replace functionality but it overwrites the duplicates as well. Any ideas?
static string GetLine(string text, int lineNo)
{
string[] lines = text.Replace("\r", "").Split('\n');
return lines.Length >= lineNo ? lines[lineNo - 1] : null;
}
static void Main(string[] args)
{
string file = "D:\\random.text";
string contents = "";
string text="random";
contents = File.ReadAllText(file);
finale=GetLine(contents,lines);
// Console.ReadLine();
if(finale.Contains(text))
{
finale.Replace(text,"Random");
System.Console.WriteLine(finale);
Console.ReadLine();
}
}

Strings are immutable type which means you cant alter an existing string. string.Replace returns the replaced string and you need to assign it back.
if(finale.Contains(text))
{
finale = finale.Replace(text,"Random"); //<- note here
System.Console.WriteLine(finale);
Console.ReadLine();
}
From there you need to rebuild the string from the string array as noted by Philippe. A complete example (but untested):
static string[] GetLines(string text)
{
return text.Replace("\r", "").Split('\n');
}
static string GetLine(string[] lines, int lineNo)
{
return lines.Length >= lineNo ? lines[lineNo - 1] : null;
}
static void Main(string[] args)
{
string file = "D:\\random.text";
string contents = "";
string text="random";
contents = File.ReadAllText(file);
var lines = GetLines(contents);
finale = GetLine(lines, lineNo);
//Console.ReadLine();
if (finale == null)
return;
if(finale.Contains(text))
{
finale = finale.Replace(text, "Random");
System.Console.WriteLine(finale);
Console.ReadLine();
}
lines[lineNo] = finale;
contents = string.Join('\n', lines);
}
And best of all, you don't need all that split function at all. .NET has that functionality and it does lazily (on demand) which is a bonus.
See for File.ReadLines if you're using .NET 4.0 and above.

The quickest solution would be to keep the array returned by Split and then use String.Join to rebuild what you started with.

Just rebuild the string with string builder as you read the file.
static void Main(string[] args)
{
string file = "D:\\random.txt";
string find = "random";
string replace = "Random";
StringBuilder resultList = new StringBuilder();
using (var stream = File.OpenText(file))
{
while (stream.Peek() >= 0)
{
string line = stream.ReadLine();
if(line == find)
{
line = replace;
}
resultList.AppendLine(line);
}
}
string result = resultList.ToString();
Console.WriteLine(result);
Console.Read();
}

C# text creation issue

This is whats going on. I have a huge text file that is suppose to be 1 line per entry. The issue is sometimes the line is broken with a new line.
I edit this entire file and wherever the file doesn't begin with ("\"A) i need to append the current line to the previous line ( replacing \n with " "). Everything I come up with keeps appending the line to a new line. Any help is appricated...
CODE:
public void step1a()
{
string begins = ("\"A");
string betaFilePath = #"C:\ext.txt";
string[] lines = File.ReadAllLines(betaFilePath);
foreach (string line in lines)
{
if (line.StartsWith(begins))
{
File.AppendAllText(#"C:\xt2.txt",line);
File.AppendAllText(#"C:\xt2.txt", "\n");
}
else
{
string line2 = line.Replace(Environment.NewLine, " ");
File.AppendAllText(#"C:\xt2.txt",line2);
}
}
}
Example:
Orig:
"\"A"Hero|apple|orange|for the fun of this
"\"A"Hero|apple|mango|lots of fun always
"\"A"Her|apple|fruit|no
pain is the way
"\"A"Hero|love|stackoverflowpeople|more fun
Resulting:
"\"A"Hero|apple|orange|for the fun of this
"\"A"Hero|apple|mango|lots of fun always
"\"A"Her|apple|fruit|no pain is the way
"\"A"Hero|love|stackoverflowpeople|more fun
my problem isnt the finding the if (line.StartsWith(begins)) its the else statement, it appends line2 to a new line

it seems like your string is not well formated...
try this "\"\\\"A\"" instead
public void step1a()
{
string begins = ("\"\\\"A\"");
string betaFilePath = #"C:\ext.txt";
string[] lines = File.ReadAllLines(betaFilePath);
foreach (string line in lines)
{
if (line.StartsWith(begins))
{
File.AppendAllText(#"C:\xt2.txt",line);
File.AppendAllText(#"C:\xt2.txt", "\n");
}
else
{
string line2 = line.Replace(Environment.NewLine, " ");
File.AppendAllText(#"C:\xt2.txt",line2);
}
}
}

This does what you want:
CopyFileRemovingStrayNewlines(#"C:\ext.txt", #"C:\xt2.txt", #"""\""A");
With this method:
public static void CopyFileRemovingStrayNewlines(string sourcePath, string destinationPath, string linePrefix)
{
string[] lines = File.ReadAllLines(sourcePath);
bool firstLine = true;
foreach (string line in lines)
{
if (line.StartsWith(linePrefix))
{
if (!firstLine)
File.AppendAllText(destinationPath, Environment.NewLine);
else
firstLine = false;
File.AppendAllText(destinationPath, line);
}
else
{
File.AppendAllText(destinationPath, " ");
File.AppendAllText(destinationPath, line);
}
}
}
It does have the problem of appending to an existing file, though. I suggest using a StreamWriter rather than AppendAllText. Like this:
public static void CopyFileRemovingStrayNewlines(string sourcePath, string destinationPath, string linePrefix)
{
string[] lines = File.ReadAllLines(sourcePath);
bool firstLine = true;
using (StreamWriter writer = new StreamWriter(destinationPath, false))
{
foreach (string line in lines)
{
if (line.StartsWith(linePrefix))
{
if (!firstLine)
writer.WriteLine();
else
firstLine = false;
writer.Write(line);
}
else
{
writer.Write(" ");
writer.Write(line);
}
}
}
}

Your problem is that the \ is a C# escape code.
Your string is parsed as "A, because \" is the escape code for a single ".
You should make the begins string an #-string, which does not use escape codes.
You will then need to escape the " by doubling it up.
For example:
const string begins = #"\""A";
Note that the best way to do this is to use a StreamWriter, like this:
using(StreamWriter writer = File.Create(#"C:\xt2.txt"))
{
foreach (string line in lines)
{
if (line.StartsWith(begins))
writer.WriteLine(); //Close the previous line
writer.Write(line);
}
}

Based on #SLaks's example here is some code that should do the trick:
public static void step1a()
{
string betaFilePath = #"C:\ext.txt";
string[] lines = File.ReadAllLines(betaFilePath);
using (StreamWriter writer = new StreamWriter(File.Create(#"C:\xt2.txt")))
{
string buffer = null;
foreach (string line in lines)
{
if (!line.StartsWith(begins))
{
writer.WriteLine(buffer + line);
buffer = null;
}
else
{
if (buffer != null)
writer.WriteLine(buffer);
buffer = line;
}
}
if(buffer != null)
Console.Out.WriteLine(buffer);
}
}