Develop Reference

How to Compress Large Files C#

I am using this method to compress files and it works great until I get to a file that is 2.4 GB then it gives me an overflow error:
void CompressThis (string inFile, string compressedFileName)
{
FileStream sourceFile = File.OpenRead(inFile);
FileStream destinationFile = File.Create(compressedFileName);
byte[] buffer = new byte[sourceFile.Length];
sourceFile.Read(buffer, 0, buffer.Length);
using (GZipStream output = new GZipStream(destinationFile,
CompressionMode.Compress))
{
output.Write(buffer, 0, buffer.Length);
}
// Close the files.
sourceFile.Close();
destinationFile.Close();
}
What can I do to compress huge files?

You should not to write the whole file to into the memory. Use Stream.CopyTo instead. This method reads the bytes from the current stream and writes them to another stream using a specified buffer size (81920 bytes by default).
Also you don't need to close Stream objects if use using keyword.
void CompressThis (string inFile, string compressedFileName)
{
using (FileStream sourceFile = File.OpenRead(inFile))
using (FileStream destinationFile = File.Create(compressedFileName))
using (GZipStream output = new GZipStream(destinationFile, CompressionMode.Compress))
{
sourceFile.CopyTo(output);
}
}
You can find a more complete example on Microsoft Docs (formerly MSDN).

You're trying to allocate all of this into memory. That just isn't necessary, you can feed the input stream directly into the output stream.

Alternative solution for zip format without allocating memory -
using (var sourceFileStream = new FileStream(this.GetFilePath(sourceFileName), FileMode.Open))
{
using (var destinationStream =
new FileStream(this.GetFilePath(zipFileName), FileMode.Create, FileAccess.ReadWrite))
{
using (var archive = new ZipArchive(destinationStream, ZipArchiveMode.Create, true))
{
var file = archive.CreateEntry(sourceFileName, CompressionLevel.Optimal);
using (var entryStream = file.Open())
{
var fileStream = sourceFileStream;
await fileStream.CopyTo(entryStream);
}
}
}
}
The solution will write directly from input stream to output stream

FileStream string method for Read/Write?

I am absolute beginner and I am using VS Community 2017 C# and trying to open file on my android phone (7.0 API 24) and write and read some text in that file. This is my methods that suppose to do that:
public string WriteFile(string fileName, byte content)
{
FileStream fileStream = new FileStream(#fileName,
FileMode.OpenOrCreate,
FileAccess.ReadWrite,
FileShare.None);
fileStream.WriteByte(content);
fileStream.Close();
return "WriteOpen";
}
public string ReadFile(string fileName)
{
FileStream fileStream = new FileStream(#fileName,
FileMode.OpenOrCreate,
FileAccess.ReadWrite,
FileShare.None);
int tekts = fileStream.ReadByte();
fileStream.Close();
return "" + tekts;
}
but I cannot find FileStream method with string arguments, so I use WriteByte and ReadByte just to test writing and reading file.
Any suggestions?

Compress large file using SharpZipLib causing Out Of Memory Exception

I have a 453MB XML file which I'm trying to compress to a ZIP using SharpZipLib.
Below is the code I'm using to create the zip, but it's causing an OutOfMemoryException. This code successfully compresses a file of 428MB.
Any idea why the exception is happening, as I can't see why, as my system has plenty of memory available.
public void CompressFiles(List<string> pathnames, string zipPathname)
{
try
{
using (FileStream stream = new FileStream(zipPathname, FileMode.Create, FileAccess.Write, FileShare.None))
{
using (ZipOutputStream stream2 = new ZipOutputStream(stream))
{
foreach (string str in pathnames)
{
FileStream stream3 = new FileStream(str, FileMode.Open, FileAccess.Read, FileShare.Read);
byte[] buffer = new byte[stream3.Length];
try
{
if (stream3.Read(buffer, 0, buffer.Length) != buffer.Length)
{
throw new Exception(string.Format("Error reading '{0}'.", str));
}
}
finally
{
stream3.Close();
}
ZipEntry entry = new ZipEntry(Path.GetFileName(str));
stream2.PutNextEntry(entry);
stream2.Write(buffer, 0, buffer.Length);
}
stream2.Finish();
}
}
}
catch (Exception)
{
File.Delete(zipPathname);
throw;
}
}

You're trying to create a buffer as big as the file. Instead, make the buffer a fixed size, read some bytes into it, and write the number of read bytes into the zip file.
Here's your code with a buffer of 4096 bytes (and some cleanup):
public static void CompressFiles(List<string> pathnames, string zipPathname)
{
const int BufferSize = 4096;
byte[] buffer = new byte[BufferSize];
try
{
using (FileStream stream = new FileStream(zipPathname,
FileMode.Create, FileAccess.Write, FileShare.None))
using (ZipOutputStream stream2 = new ZipOutputStream(stream))
{
foreach (string str in pathnames)
{
using (FileStream stream3 = new FileStream(str,
FileMode.Open, FileAccess.Read, FileShare.Read))
{
ZipEntry entry = new ZipEntry(Path.GetFileName(str));
stream2.PutNextEntry(entry);
int read;
while ((read = stream3.Read(buffer, 0, buffer.Length)) > 0)
{
stream2.Write(buffer, 0, read);
}
}
}
stream2.Finish();
}
}
catch (Exception)
{
File.Delete(zipPathname);
throw;
}
}
Especially note this block:
const int BufferSize = 4096;
byte[] buffer = new byte[BufferSize];
// ...
int read;
while ((read = stream3.Read(buffer, 0, buffer.Length)) > 0)
{
stream2.Write(buffer, 0, read);
}
This reads bytes into buffer. When there are no more bytes, the Read() method returns 0, so that's when we stop. When Read() succeeds, we can be sure there is some data in the buffer but we don't know how many bytes. The whole buffer might be filled, or just a small portion of it. Therefore, we use the number of read bytes read to determine how many bytes to write to the ZipOutputStream.
That block of code, by the way, can be replaced by a simple statement that was added to .Net 4.0, which does exactly the same:
stream3.CopyTo(stream2);
So, your code could become:
public static void CompressFiles(List<string> pathnames, string zipPathname)
{
try
{
using (FileStream stream = new FileStream(zipPathname,
FileMode.Create, FileAccess.Write, FileShare.None))
using (ZipOutputStream stream2 = new ZipOutputStream(stream))
{
foreach (string str in pathnames)
{
using (FileStream stream3 = new FileStream(str,
FileMode.Open, FileAccess.Read, FileShare.Read))
{
ZipEntry entry = new ZipEntry(Path.GetFileName(str));
stream2.PutNextEntry(entry);
stream3.CopyTo(stream2);
}
}
stream2.Finish();
}
}
catch (Exception)
{
File.Delete(zipPathname);
throw;
}
}
And now you know why you got the error, and how to use buffers.

You're allocating a lot of memory for no good reason, and I bet you have a 32-bit process. 32-bit processes can only allocate up to 2GB of virtual memory in normal conditions, and the library surely allocates memory too.
Anyway, several things are wrong here:
byte[] buffer = new byte[stream3.Length];
Why? You don't need to store the whole thing in memory to process it.
if (stream3.Read(buffer, 0, buffer.Length) != buffer.Length)
This one is nasty. Stream.Read is explicitly allowed to return less bytes than what you asked for, and this is still a valid result. When reading a stream into a buffer you have to call Read repeatedly until the buffer is filled or the end of the stream is reached.
Your variables should have more meaningful names. You can easily get lost with these stream2, stream3 etc.
A simple solution would be:
using (var zipFileStream = new FileStream(zipPathname, FileMode.Create, FileAccess.Write, FileShare.None))
using (ZipOutputStream zipStream = new ZipOutputStream(zipFileStream))
{
foreach (string str in pathnames)
{
using(var itemStream = new FileStream(str, FileMode.Open, FileAccess.Read, FileShare.Read))
{
var entry = new ZipEntry(Path.GetFileName(str));
zipStream.PutNextEntry(entry);
itemStream.CopyTo(zipStream);
}
}
zipStream.Finish();
}

How to Overwrite Existing Text Using FileStream

NOTE: I can't use FileMode.Create or FileMode.Truncate because they would cause some unwanted problem
byte[] data = new UTF8Encoding().GetBytes(this.Box.Text);
FileStream f = File.Open(path, FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite);
f.Write(data, 0, data.Length);
f.Close();
This will append new text to the top of the old one. How can I overwrite everything?

your code
byte[] data = new UTF8Encoding().GetBytes(this.Box.Text);
FileStream f = File.Open(path, FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite);
f.Write(data, 0, data.Length);
f.Close();
how come you can't do something like this..? please explain why you can't use FileMode.Create
byte[] data = new UTF8Encoding().GetBytes(this.Box.Text);
using(Stream f = File.Open(path, FileMode.Write))
{
f.Write(data, 0, data.Length);
}
you could also do something like the following
1. Let the users write to the same file
2. capture the users Machine Name or User Id then
2. write a line in your file like this
strSeprate = new string('*',25);
//will write to the file "*************************";
f.Write(strSeprate);
f.Write(Machine Name or UserId);
f.Write(data);
f.Write(DateTime.Now.ToString());
f.Write(strSeprate);
just an idea..

Corrupted file while compressing stream using dotnetzip library

The file is created and the size seems to be ok, but when I double click it is says its format is wrong or the file is damaged.
This is the code I'm using
public MemoryStream CompressFiles(Dictionary<string, MemoryStream> filesToBeCompressed)
{
var output = new MemoryStream();
using (var zip = new ZipFile())
{
foreach (var entry in filesToBeCompressed)
{
entry.Value.Seek(0, SeekOrigin.Begin); // <-- must do this after writing the stream (I've read this in a blog
zip.AddEntry(entry.Key.Substring(entry.Key.LastIndexOf('/') + 1, entry.Key.Length - entry.Key.LastIndexOf('/') - 1), entry.Value);
zip.Save(output);
}
}
return output;
}
Then in the calling method
SaveStreamToFile(documentCompressedName,getDocument());
getDocument() calls Compress internally
And that method finally
private static void SaveStreamToFile(string fileFullPath, Stream stream)
{
if (stream.Length == 0) return;
// Create a FileStream object to write a stream to a file
using (FileStream fileStream = System.IO.File.Create(fileFullPath, (int)stream.Length))
{
// Fill the bytes[] array with the stream data
var bytesInStream = new byte[stream.Length];
stream.Read(bytesInStream, 0, (int)bytesInStream.Length);
// Use FileStream object to write to the specified file
fileStream.Write(bytesInStream, 0, bytesInStream.Length);
}
}
Any ideas?
Thanks in advance! Guillermo.

I think the problem is in your function SaveStreamToFile. You have to set the position of the stream to the beginning before you write the archive to disk:
private static void SaveStreamToFile(string fileFullPath, Stream stream)
{
if (stream.Length == 0) return;
// Set the position within the stream to the beginning of the stream
stream.Seek(0, SeekOrigin.Begin);
// Create a FileStream object to write a stream to a file
using (FileStream fileStream = System.IO.File.Create(fileFullPath, (int)stream.Length))
{
// Fill the bytes[] array with the stream data
var bytesInStream = new byte[stream.Length];
stream.Read(bytesInStream, 0, (int)bytesInStream.Length);
// Use FileStream object to write to the specified file
fileStream.Write(bytesInStream, 0, bytesInStream.Length);
}
}
Hope, this helps.

From your code snippets, my guess here is that the MemoryStream's Position is at the end of the stream when you pass it to SaveStreamToFile, and as you never set the position back to the start of the stream, your stream.Read is actually reading no bytes at all. If you open your output zip file with a hex editor, you'll probably see that it's full of zeros.
You have a number of options here, but my suggestion would be to try:
private static void SaveStreamToFile(string fileFullPath, Stream stream)
{
if (stream.Length == 0) return;
// Create a FileStream object to write a stream to a file
using (FileStream fileStream = System.IO.File.Create(fileFullPath, (int)stream.Length))
{
// Use FileStream object to write to the specified file
fileStream.Write(stream.GetBuffer(), 0, stream.Length);
}
}
This approach avoids taking a copy of the internal memory buffer of the MemoryStream. Whilst I don't know how large your zip files are so it may not be an issue in terms of memory use, but storing the zip file in memory twice - once in the MemoryStream, and again in your original bytesInStream array seems unnecessary.