Develop Reference

OutOfMemoryException while trying to read a txt file that is over 2GB

In my code I recover the file, extract the text, manipulate it and write the modified string in the file, I have not had any problems to date, the file I had to manipulate today weighed over 2GB, with over 1 million lines
public static void ModifyFile(string directory, string filename)
{
string input = string.Empty;
using (StreamReader reader = new StreamReader(directory + filename))
{
input = reader.ReadToEnd();
}
string output = Manipulate(input);
File.WriteAllText($"{directory}{filename}", String.Empty);
WriteFile(directory, filename, output);
}
private static void WriteFile(string directory, string filename, string output)
{
using (StreamWriter writer = new StreamWriter(directory + filename, true))
{
{
writer.Write(output);
}
writer.Close();
}
}
private static string Manipulate(string input)
{
var counter = 1;
StringBuilder output = new StringBuilder();
string[] subs = input.Split(new string[] { Environment.NewLine }, StringSplitOptions.None);
foreach (var x in subs)
{
if (subs[subs.Length - 1] != x && subs[subs.Length - 2] != x)
{
var column = x.Substring(121, 2);
if (column.Equals("NA"))
{
var c = x.Substring(22, 9);
output.Append(ManipulateStringElement(x, counter, 22)
.Replace("\r\n", "\n").Replace("\r", "\n").Replace("\n", "\r\n"));
output.Append("\n");
counter++;
}
}
else if (subs[subs.Length - 2] == x)
{
output.Append(ManipulateStringElement(x, counter, 22)
.Replace("\r\n", "\n").Replace("\r", "\n").Replace("\n", "\r\n"));
}
}
return output.ToString();
}
private static string ManipulateStringElement(string item, int counter, int start)
{
return item.Replace(item.Substring(start, 9), GenerateProgressive(counter));
}
private static string GenerateProgressive(int counter)
{
return $"{counter}".PadLeft(9, '0');
}
But while running reader.ReadToEnd() I get "OutOfMemoryException" error, which makes me think the file is too big
The application is in .NET Framewrok 4.6.1, the operating system is 64bit (I had read that it could affect)

You need to do this in a streaming fashion in order to reduce memory consumption.
Open an input and an output file at the same time, and immediately output the result of a single line from Manipulate(). Ensure it ends with your custom newline character.
Finally replace the original file with the new one.
public static void ModifyFile(string directory, string filename)
{
string inputFile = Path.Combine(directory, filename);
string outputFile = Path.Combine(directory, filename + ".new");
using (var reader = new StreamReader(inputFile))
using (var reader = new StreamWriter(outputFile, true))
{
string input;
while((input = reader.ReadLine()) != null)
{
string output = Manipulate(input);
writer.Write(output);
}
}
File.Move(outputFile, inputFile, true);
}
You may also want to do this using async code, which could improve responsiveness.
I note that you are also retrieving the last two lines of the file. I suggest you do this separately, using this answer for example.
There are also other performance improvements you can make. For example:
private static string GenerateProgressive(int counter)
{
return counter.ToString("D9");
}
as well as:
private static string ManipulateStringElement(string item, int counter, int start)
{
return GenerateProgressive(counter) + item.Substring(9);
}

How I can access a file with c#? [duplicate]

This question already has answers here:
How to read an entire file to a string using C#?
(17 answers)
Closed 3 years ago.
There are some function that read all text from file without use FileStream class and more easy?
In microsoft doc found this code to read from file but I think is some complexed.
private async void Button_Click_1(object sender, RoutedEventArgs e)
{
string filename = #"C:\Example\existingfile.txt";
char[] result;
StringBuilder builder = new StringBuilder();
using (StreamReader reader = File.OpenText(filename))
{
result = new char[reader.BaseStream.Length];
await reader.ReadAsync(result, 0, (int)reader.BaseStream.Length);
}
foreach (char c in result)
{
if (char.IsLetterOrDigit(c) || char.IsWhiteSpace(c))
{
builder.Append(c);
}
}
FileOutput.Text = builder.ToString();
}

Please see the File.ReadAllText Method.
public static void Main()
{
string path = #"c:\temp\MyTest.txt";
// This text is added only once to the file.
if (!File.Exists(path))
{
// Create a file to write to.
string createText = "Hello and Welcome" + Environment.NewLine;
File.WriteAllText(path, createText, Encoding.UTF8);
}
// This text is always added, making the file longer over time
// if it is not deleted.
string appendText = "This is extra text" + Environment.NewLine;
File.AppendAllText(path, appendText, Encoding.UTF8);
// Open the file to read from.
string readText = File.ReadAllText(path);
Console.WriteLine(readText);
}

Counting the # of lines in a very large file gives System OutofMemory Exception [duplicate]

This question already has answers here:
What's the fastest way to read a text file line-by-line?
(9 answers)
Closed 5 years ago.
static void Main(string[] args)
{
string TheDataFile = "";
string ErrorMsg = "";
string lngTransDate = "";
ProcessDataFile ProcessTheDataFile = new ProcessDataFile();
string TheFile = "S:\\MIS\\Provider NPI file\\Processed\\npidata_20050523-20161009.csv";
string[] lines = File.ReadAllLines(TheFile, Encoding.UTF8);//Read all lines to an array
Console.WriteLine(lines.Length.ToString());
Console.ReadLine();
}
This throws an error because the file is very large (has 6 million lines). Is there a way to handle large files and count the # of lines?

Use a StreamReader:
string TheFile = "S:\\MIS\\Provider NPI file\\Processed\\npidata_20050523-20161009.csv";
int count = 0;
using (System.IO.StreamReader sr = new System.IO.StreamReader(TheFile))
{
while (sr.ReadLine() != null)
count++;
}

You need to do a lazy evaluation of the file so it isn't loaded into memory entirelly.
Helper method
public static class ToolsEx
{
public static IEnumerable<string> ReadAsLines(this string filename)
{
using (var streamReader = new StreamReader(filename))
while (!streamReader.EndOfStream)
yield return streamReader.ReadLine();
}
}
Usage
var lineCount = "yourfile.txt".ReadAsLines().Count();

According to this already accepted answer, this should do it.
using System;
using System.IO;
namespace CountLinesInFiles_45194927
{
class Program
{
static void Main(string[] args)
{
int counter = 0;
foreach (var line in File.ReadLines("c:\\Path\\To\\File.whatever"))
{
counter++;
}
Console.WriteLine(counter);
Console.ReadLine();
}
}
}

How to get the filename according to my requirement C# DirctoryInfo.GetFileSystemInfos(); [duplicate]

This question already has answers here:
C# Splitting Strings on `#` character
(3 answers)
Closed 9 years ago.
.
net winform
Now I get all the fileName under the specified directory.
the result is "0001_00001523_028155.sql"
but my goal is only to get "00001523"
how can I do that?
private void loadscriptfolder()
{
string folderName = this.textBoxScriptLocation.Text.Trim();
DirectoryInfo dir = new DirectoryInfo(folderName);
if (dir.Exists)
{
FileSystemInfo[] fs = dir.GetFileSystemInfos();
foreach (FileSystemInfo fs2 in fs)
{
FileInfo file = fs2 as FileInfo;
if (file != null)
{
listBoxResult.Items.Add(file);
}
}
}
}
Thank you everyone!!!

If you just really need to get that part of the file name, then you can just do a string manipulation on its file name.
FileInfo file = fs2 as FileInfo;
if (file != null)
{
listBoxResult.Items.Add(file.Name.Split('_')[1]);
}

Try
string str = #"0001_00001523_028155.sql";
var result = str.Split('_')[1];

SharpZipLib ~ How to extract specific files from a zip [duplicate]

This question already has answers here:
Using SharpZipLib to unzip specific files?
(5 answers)
Closed 9 years ago.
Ok,
I have a list of files (SourceFile objects which just contain the filename only) then I want to pull those specific files out of a zip and dump them into a temp directory so I can distribute them later.
I came up with this, but I am unsure on how to proceed next..
private List<string> ExtractSelectedFiles()
{
List<SourceFile> zipFilePaths = new List<SourceFile>();
List<string> tempFilePaths = new List<string>();
if (!File.Exists(this.txtSourceSVNBuildPackage.Text)) { return tempFilePaths; };
FileStream zipFileStream = File.OpenRead(this.txtSourceSVNBuildPackage.Text);
ZipInputStream inStream = new ZipInputStream(zipFileStream);
foreach (SourceFile currentFile in _selectedSourceFiles)
{
bool getNextEntry = true;
while (getNextEntry)
{
ZipEntry entry = inStream.GetNextEntry();
getNextEntry = (entry != null);
if (getNextEntry)
{
if (fileType == ".dll")
{
if (sourcefile.Name == Path.GetFileName(entry.Name))
{
//Extract file into a temp directory somewhere
//tempFilePaths.Add("extractedfilepath")
}
}
}
}
}
return tempFilePaths;
}
FYI:
public class SourceFile
{
public string Name { get; set; } //ex. name = "Fred.dll"
}

ok.. figured I'd update you all after I got together the missing piece I needed.
//in the code somewhere above:
string tempDirectory = Environment.GetEnvironmentVariable("TEMP");
string createPath = tempDirectory + "\\" + Path.GetFileName(entry.Name);
//my missing piece..
//Extract file into a temp directory somewhere
FileStream streamWriter = File.Create(createPath);
int size = 2048;
byte[] data = new byte[2048];
while (true)
{
size = inStream.Read(data, 0, data.Length);
if (size > 0)
{
streamWriter.Write(data, 0, size);
}
else
{
break;
}
}
streamWriter.Close();