I wrote a program that split txt files into smaller pieces
But my problem is that my method is slow
Because this file size is 1gb and I use a variable named "pagesize" which base on that amount of lines in splitted files will be calculated
The problem is that the foreach is slow?
Is there a better way?
private void button1_Click(object sender, EventArgs e)
{
string inputFile = #"G:\Programming\C#\c# tamrin reza\large-Orders.txt";
int seed = 1000;
const int pageSize = 5000;
int count = 1;
const string destinationFileName = #"F:\Output\";
string outputFile;
string baseName = "-CustomerApp";
string extension = Path.GetExtension(inputFile);
var lst = new List<string>();
//FileInfo fileInfo = new FileInfo(inputFile);
//long fileSize = fileInfo.Length / pageSize;
FileStream fs = new FileStream(inputFile, FileMode.Open);
StreamReader sr = new StreamReader(fs);
while (!sr.EndOfStream)
{
for (int j = 1; j <= pageSize; j++)
{
var line = sr.ReadLine();
lst.Add(line);
}
outputFile = destinationFileName + count + baseName + extension;
CopyLines(lst, outputFile);
lst.Clear();
count++;
}
}
private void CopyLines(List<string> line, string outputFile)
{
FileStream outputFileStream = new FileStream(outputFile, FileMode.Create, FileAccess.Write);
//StreamWriter writer = new StreamWriter(outputFile);
//for (int i = 1; i < line.Count; i++)
//{
//}
using (StreamWriter sw = new StreamWriter(outputFileStream))
{
foreach (var li in line)
{
sw.WriteLine(li);
}
}
}
Thanks
You are iterating over the entire collection twice. If you write to the output file while you are reading, that will save one iteration over the entire collection.
while (!sr.EndOfStream)
{
outputFile = destinationFileName + count + baseName + extension;
FileStream outputFileStream = new FileStream(outputFile, FileMode.Create, FileAccess.Write);
using (StreamWriter sw = new StreamWriter(outputFileStream))
{
for (int j = 1; j <= pageSize; j++)
{
var line = sr.ReadLine();
sw.WriteLine(li);
}
}
lst.Clear();
count++;
}
Related
I am trying to hide text file into image with c#
void HideTextFileIntoImage(string TextPath, string ImagePath, string NewFilePath)
{
string[] Text = File.ReadAllLines(TextPath);
string[] Image = File.ReadAllLines(ImagePath);
File.Create(NewFilePath).Close();
string[] NewFile = new string[Text.Length + Image.Length];
for (int i = 0; i < Image.Length; i++)
{
NewFile[i] = Image[i];
}
for (int i = 0; i < Text.Length; i++)
{
NewFile[i + Image.Length] = Text[i];
}
StreamWriter sw = new StreamWriter(NewFilePath);
for (int t = 0; t < NewFile.Length; t++)
{
sw.WriteLine(NewFile[t]);
}
sw.Close();
}
But I can't see image after using it. What wrong?
You're trying to treat binary data as though it were text.
Try this instead (completely untested):
void HideTextFileIntoImage(string TextPath, string ImagePath, string NewFilePath)
{
var textBytes = File.ReadAllBytes(TextPath);
var imageBytes = File.ReadAllBytes(ImagePath);
using (var stream = new FileStream(NewFilePath, FileMode.Create)) {
stream.Write(imageBytes, 0, imageBytes.Length);
stream.Write(textBytes, 0, textBytes.Length);
}
}
I'm trying to copy the contents of one Excel file to another Excel file while replacing a string inside of the file on the copy. It's working for the most part, but the file is losing 27 kb of data. Any suggestions?
public void ReplaceString(string what, string with, string path) {
List < string > doneContents = new List < string > ();
List < string > doneNames = new List < string > ();
using(ZipArchive archive = ZipFile.Open(_path, ZipArchiveMode.Read)) {
int count = archive.Entries.Count;
for (int i = 0; i < count; i++) {
ZipArchiveEntry entry = archive.Entries[i];
using(var entryStream = entry.Open())
using(StreamReader reader = new StreamReader(entryStream)) {
string txt = reader.ReadToEnd();
if (txt.Contains(what)) {
txt = txt.Replace(what, with);
}
doneContents.Add(txt);
string name = entry.FullName;
doneNames.Add(name);
}
}
}
using(MemoryStream zipStream = new MemoryStream()) {
using(ZipArchive newArchive = new ZipArchive(zipStream, ZipArchiveMode.Create, true, Encoding.UTF8)) {
for (int i = 0; i < doneContents.Count; i++) {
int spot = i;
ZipArchiveEntry entry = newArchive.CreateEntry(doneNames[spot]);
using(var entryStream = entry.Open())
using(var sw = new StreamWriter(entryStream)) {
sw.Write(doneContents[spot]);
}
}
}
using(var fileStream = new FileStream(path, FileMode.Create)) {
zipStream.Seek(0, SeekOrigin.Begin);
zipStream.CopyTo(fileStream);
}
}
}
I've used Microsoft's DocumentFormat.OpenXML and Excel Interop, however, they are both lacking in a few main components that I need.
Update:
using(var fileStream = new FileStream(path, FileMode.Create)) {
var wrapper = new StreamWriter(fileStream);
wrapper.AutoFlush = true;
zipStream.Seek(0, SeekOrigin.Begin);
zipStream.CopyTo(wrapper.BaseStream);
wrapper.Flush();
wrapper.Close();
}
Try the process without changing the string and see if the file size is the same. If so then it would seem that your copy is working correctly, however as Marc B suggested, with compression, even a small change can result in a larger change in the overall size.
I have a function that creates a zip file a string array of files passed. The function does succeed in creating the zip file and the zip entry files inside it, but these zip entry files are empty. I've tried a couple of different methods - the function code below is the closest I've gotten to something working:
public static bool ZipFile(string[] arrFiles, string sZipToDirectory, string sZipFileName)
{
if (Directory.Exists(sZipToDirectory))
{
FileStream fNewZipFileStream;
ZipOutputStream zos;
try {
fNewZipFileStream = File.Create(sZipToDirectory + sZipFileName);
zos = new ZipOutputStream(fNewZipFileStream);
for (int i = 0; i < arrFiles.Length; i++) {
ZipEntry entry = new ZipEntry(arrFiles[i].Substring(arrFiles[i].LastIndexOf("/") + 1));
zos.PutNextEntry(entry);
FileStream fStream = File.OpenRead(arrFiles[i]);
BufferedStream bfStrm = new BufferedStream(fStream);
byte[] buffer = new byte[bfStrm.Length];
int count;
while ((count = bfStrm.Read(buffer, 0, 1024)) != -1) {
zos.Write(buffer);
}
bfStrm.Close();
fStream.Close();
zos.CloseEntry();
}
zos.Close();
fNewZipFileStream.Close();
return true;
}
catch (Exception ex)
{
string sErr = ex.Message;
return false;
}
finally
{
fNewZipFileStream = null;
zos = null;
}
}
else
{
return false;
}
}
I think it's got to do with the byte stream handling. I've tried this bit of code that handles the stream but it goes into an infinite loop:
while ((count = fStream.Read(buffer, 0, 1024)) != -1) {
zos.Write(buffer, 0, count);
}
fStream.Close();
I found a solution that is quite simple - I used the ReadAllBytes method of the static File class.
ZipEntry entry = new ZipEntry(arrFiles[i].Substring(arrFiles[i].LastIndexOf("/") + 1));
zos.PutNextEntry(entry);
byte[] fileContents = File.ReadAllBytes(arrFiles[i]);
zos.Write(fileContents);
zos.CloseEntry();
Using Read() on a FileStream returns the amount of bytes read into the stream or 0 if the end of the stream has been reached. It will never return a value of -1.
From MSDN:
The total number of bytes read into the buffer. This might be less than the number of bytes requested if that number of bytes are not currently available, orzero if the end of the stream is reached.
I'd modify your code to the following:
System.IO.FileStream fos = new System.IO.FileStream(sZipToDirectory + sZipFileName, FileMode.Create);
Java.Util.Zip.ZipOutputStream zos = new Java.Util.Zip.ZipOutputStream(fos);
byte[] buffer = new byte[1024];
for (int i = 0; i < arrFiles.Length; i++) {
FileInfo fi = new FileInfo (arrFiles[i]);
Java.IO.FileInputStream fis = new Java.IO.FileInputStream(fi.FullName);
ZipEntry entry = new ZipEntry(arrFiles[i].Substring(arrFiles[i].LastIndexOf("/") + 1));
zos.PutNextEntry(entry);
int count = 0;
while ((count = fis.Read(buffer)) > 0) {
zos.Write(buffer, 0, count);
}
fis.Close();
zos.CloseEntry();
}
This is nearly identical to the code I've used for creating zip archives on Android in the past.
Are you allowed to use SharpZip? It's really easy to use.
Here is a blog post I wrote to extract zip files
private static void upzip(string url)
{
WebClient wc = new WebClient();
wc.DownloadFile(url, "temp.zip");
//unzip
ZipFile zf = null;
try
{
zf = new ZipFile(File.OpenRead("temp.zip"));
foreach (ZipEntry zipEntry in zf)
{
string fileName = zipEntry.Name;
byte[] buffer = new byte[4096];
Stream zipStream = zf.GetInputStream(zipEntry);
using (FileStream streamWriter = File.Create( fileName))
{
StreamUtils.Copy(zipStream, streamWriter, buffer);
}
}
}
finally
{
if (zf != null)
{
zf.IsStreamOwner = true;
zf.Close();
}
}
}
private void ZipFolder(string[] _files, string zipFileName)
{
using var memoryStream = new MemoryStream();
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
foreach (var item in _files)
{
var demoFile = archive.CreateEntry(Path.GetFileName(item));
using var readStreamW = File.OpenRead(item);
using (var entryStream = demoFile.Open())
{
using (var streamWriter = new StreamWriter(entryStream))
{
readStreamW.Seek(0, SeekOrigin.Begin);
readStreamW.CopyTo(streamWriter.BaseStream);
}
}
}
}
using var fileStream = new FileStream(zipFileName, FileMode.Create);
memoryStream.Seek(0, SeekOrigin.Begin);
memoryStream.CopyTo(fileStream);
}
I am reading a file using streamreader opened in ReadWrite mode. The requirement I have is to check for a file for specific text, and if it is found, replace that line with a new line.
Currently I have initialized a StreamWriter for writing.
It is writing text to a file but it's appending that to a new line.
So what should I do to replace the particular line text?
System.IO.FileStream oStream = new System.IO.FileStream(sFilePath, System.IO.FileMode.Append, System.IO.FileAccess.Write, System.IO.FileShare.Read);
System.IO.FileStream iStream = new System.IO.FileStream(sFilePath, System.IO.FileMode.Open, System.IO.FileAccess.Read, System.IO.FileShare.ReadWrite);
System.IO.StreamWriter sw = new System.IO.StreamWriter(oStream);
System.IO.StreamReader sr = new System.IO.StreamReader(iStream);
string line;
int counter = 0;
while ((line = sr.ReadLine()) != null)
{
if (line.Contains("line_found"))
{
sw.WriteLine("line_found false");
break;
}
counter++;
}
sw.Close();
sr.Close();
Hi Try the below Code.... it will help you....
//Replace all the HI in the Text file...
var fileContents = System.IO.File.ReadAllText(#"C:\Sample.txt");
fileContents = fileContents.Replace("Hi","BYE");
System.IO.File.WriteAllText(#"C:\Sample.txt", fileContents);
//Replace the HI in a particular line....
string[] lines = System.IO.File.ReadAllLines("Sample.txt");
for (int i = 0; i < lines.Length; i++)
{
if(lines[i].Contains("hi"))
{
MessageBox.Show("Found");
lines[i] = lines[i].Replace("hi", "BYE");
break;
}
}
System.IO.File.WriteAllLines("Sample.txt", lines);
im having a tough time getting a small application to work faster. im not a developer and it took me some time to get this working as is. Can anyone offer any suggestions or alternate code to speed this process up, its taking about 1 Hour to process 10m of the input file.
the code is listed below and here is an example of the input file.
4401,imei:0000000000,2012-09-01 12:12:12.9999
using System;
using System.Globalization;
using System.IO;
class Sample
{
public static void Main(string[] args)
{
if (args.Length == 0)
{
return;
}
using (FileStream stream = File.Open(args[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
using (StreamReader streamReader = new StreamReader(stream))
{
System.Text.StringBuilder builder = new System.Text.StringBuilder();
while (!streamReader.EndOfStream)
{
var line = streamReader.ReadLine();
var values = line.Split(',');
DateTime dt = new DateTime();
DateTime.TryParse(values[2], out dt);
values[2] = Convert.ToString(dt.Ticks);
string[] output = new string[values.Length];
bool firstColumn = true;
for (int index = 0; index < values.Length; index++)
{
if (!firstColumn)
builder.Append(',');
builder.Append(values[index]);
firstColumn = false;
}
File.WriteAllText(args[1], builder.AppendLine().ToString());
}
}
}
}
}
The biggest performance hit is that every time a line is read the entire file (processed so far) is written back to disk. For a quick win, try moving your StringBuilder out of the loop:
System.Text.StringBuilder builder = new System.Text.StringBuilder();
using (FileStream stream = File.Open(args[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
using (StreamReader streamReader = new StreamReader(stream))
{
while (!streamReader.EndOfStream)
{
var line = streamReader.ReadLine();
var values = line.Split(',');
DateTime dt = new DateTime();
DateTime.TryParse(values[2], out dt);
values[2] = Convert.ToString(dt.Ticks);
string[] output = new string[values.Length];
bool firstColumn = true;
for (int index = 0; index < values.Length; index++)
{
if (!firstColumn)
builder.Append(',');
builder.Append(values[index]);
firstColumn = false;
}
builder.AppendLine();
}
}
}
File.WriteAllText(args[1], builder.ToString());
If you want to refactor further change the comma separating logic:
System.Text.StringBuilder builder = new System.Text.StringBuilder();
using (FileStream stream = File.Open(args[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
using (StreamReader streamReader = new StreamReader(stream))
{
while (!streamReader.EndOfStream)
{
var line = streamReader.ReadLine();
var values = line.Split(',');
DateTime dt = new DateTime();
DateTime.TryParse(values[2], out dt);
values[2] = Convert.ToString(dt.Ticks);
builder.AppendLine(string.Join(",", values));
}
}
}
File.WriteAllText(args[1], builder.ToString());
Edit: To avoid the memory usage, remove the Stringbuilder and use another FileStream to write to disk. Your proposed solution (using a List) will still use a substantial amount of memory and likely break on larger files:
using (FileStream input = File.Open(args[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
using (FileStream output = File.Create(args[1]))
{
using (StreamReader streamReader = new StreamReader(input))
using (StreamWriter streamWriter = new StreamWriter(output))
{
while (!streamReader.EndOfStream)
{
var line = streamReader.ReadLine();
var values = line.Split(',');
DateTime dt = new DateTime();
DateTime.TryParse(values[2], out dt);
values[2] = Convert.ToString(dt.Ticks);
streamWriter.WriteLine(string.Join(",", values));
}
}
}
here is what i found can fix this and handle the large files.
thanks to #Muzz and #Vache for the assistance.
string line = "";
System.IO.StreamReader file = new System.IO.StreamReader("c:/test.txt");
List<string> convertedLines = new List<string>();
while ((line = file.ReadLine()) != null)
{
string[] lineSplit = line.Split(',');
DateTime dt = new DateTime();
DateTime.TryParse(lineSplit[2], out dt);
lineSplit[2] = Convert.ToString(dt.Ticks);
string convertedline = lineSplit[0] + "," + lineSplit[1] + "," + lineSplit[2];
convertedLines.Add(convertedline);
}
file.Close();
File.WriteAllLines("c:/newTest.txt", convertedLines);