Develop Reference

Decompress Stream to String using SevenZipSharp

I'd like to compress a string using SevenZipSharp and have cobbled together a C# console application (I'm new to C#) using the following code, (bits and pieces of which came from similar questions here on SO).
The compress part seems to work (albeit I'm passing in a file instead of a string), output of the compressed string to the console looks like gibberish but I'm stuck on the decompress...
I'm trying to do the same thing as here (I think):
https://stackoverflow.com/a/4305399/3451115
https://stackoverflow.com/a/45861659/3451115
https://stackoverflow.com/a/36331690/3451115
Appreciate any help, ideally the console will display the compressed string followed by the decompressed string.
Thanks :)
using System;
using System.IO;
using SevenZip;
namespace _7ZipWrapper
{
public class Program
{
public static void Main()
{
SevenZipCompressor.SetLibraryPath(#"C:\Temp\7za64.dll");
SevenZipCompressor compressor = new SevenZipCompressor();
compressor.CompressionMethod = CompressionMethod.Ppmd;
compressor.CompressionLevel = SevenZip.CompressionLevel.Ultra;
compressor.ScanOnlyWritable = true;
var compStream = new MemoryStream();
var decompStream = new MemoryStream();
compressor.CompressFiles(compStream, #"C:\Temp\a.txt");
StreamReader readerC = new StreamReader(compStream);
Console.WriteLine(readerC.ReadToEnd());
Console.ReadKey();
// works up to here... below here output to consol is: ""
SevenZipExtractor extractor = new SevenZip.SevenZipExtractor(compStream);
extractor.ExtractFile(0, decompStream);
StreamReader readerD = new StreamReader(decompStream);
Console.WriteLine(readerD.ReadToEnd());
Console.ReadKey();
}
}
}

The result of compression is binary data - it isn't a string. If you try to read it as a string, you'll just see garbage. That's to be expected - you shouldn't be treating it as a string.
The next problem is that you're trying to read from compStream twice, without "rewinding" it first. You're starting from the end of the stream, which means there's no data for it to decompress. If you just add:
compStream.Position = 0;
before you create the extractor, you may well find it works immediately. You may also need to rewind the decompStream before reading from it. So you'd have code like this:
// Rewind to the start of the stream before decompressing
compStream.Position = 0;
SevenZipExtractor extractor = new SevenZip.SevenZipExtractor(compStream);
extractor.ExtractFile(0, decompStream);
// Rewind to the start of the decompressed stream before reading
decompStream.Position = 0;

Can I get a GZipStream for a file without writing to intermediate temporary storage?

Can I get a GZipStream for a file on disk without writing the entire compressed content to temporary storage? I'm currently using a temporary file on disk in order to avoid possible memory exhaustion using MemoryStream on very large files (this is working fine).
public void UploadFile(string filename)
{
using (var temporaryFileStream = File.Open("tempfile.tmp", FileMode.CreateNew, FileAccess.ReadWrite))
{
using (var fileStream = File.OpenRead(filename))
using (var compressedStream = new GZipStream(temporaryFileStream, CompressionMode.Compress, true))
{
fileStream.CopyTo(compressedStream);
}
temporaryFileStream.Position = 0;
Uploader.Upload(temporaryFileStream);
}
}
What I'd like to do is eliminate the temporary storage by creating GZipStream, and have it read from the original file only as the Uploader class requests bytes from it. Is such a thing possible? How might such an implementation be structured?
Note that Upload is a static method with signature static void Upload(Stream stream).
Edit: The full code is here if it's useful. I hope I've included all the relevant context in my sample above however.

Yes, this is possible, but not easily with any of the standard .NET stream classes. When I needed to do something like this, I created a new type of stream.
It's basically a circular buffer that allows one producer (writer) and one consumer (reader). It's pretty easy to use. Let me whip up an example. In the meantime, you can adapt the example in the article.
Later: Here's an example that should come close to what you're asking for.
using (var pcStream = new ProducerConsumerStream(BufferSize))
{
// start upload in a thread
var uploadThread = new Thread(UploadThreadProc(pcStream));
uploadThread.Start();
// Open the input file and attach the gzip stream to the pcStream
using (var inputFile = File.OpenRead("inputFilename"))
{
// create gzip stream
using (var gz = new GZipStream(pcStream, CompressionMode.Compress, true))
{
var bytesRead = 0;
var buff = new byte[65536]; // 64K buffer
while ((bytesRead = inputFile.Read(buff, 0, buff.Length)) != 0)
{
gz.Write(buff, 0, bytesRead);
}
}
}
// The entire file has been compressed and copied to the buffer.
// Mark the stream as "input complete".
pcStream.CompleteAdding();
// wait for the upload thread to complete.
uploadThread.Join();
// It's very important that you don't close the pcStream before
// the uploader is done!
}
The upload thread should be pretty simple:
void UploadThreadProc(object state)
{
var pcStream = (ProducerConsumerStream)state;
Uploader.Upload(pcStream);
}
You could, of course, put the producer on a background thread and have the upload be done on the main thread. Or have them both on background threads. I'm not familiar with the semantics of your uploader, so I'll leave that decision to you.

Path at MemoryStream for use in CMD

Is there a way to get the "Path" to a memorystream?
For example if i want to use CMD and point to a filepath, like "C:..." but instead the file is in a memorystream, is it possible to point it there?
I have tried searching on it but i can´t find any clear information on this.
EDIT:
If it helps, the thing i am wanting to access is an image file, a print screen like this:
using (Bitmap b = new Bitmap(Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height))
{
using (Graphics g = Graphics.FromImage(b))
{
g.CopyFromScreen(0, 0, 0, 0, Screen.PrimaryScreen.Bounds.Size, CopyPixelOperation.SourceCopy);
}
using (MemoryStream ms = new MemoryStream())
{
b.Save(ms, ImageFormat.Bmp);
StreamReader read = new StreamReader(ms);
ms.Position = 0;
var cwebp = new Process
{
StartInfo =
{
WindowStyle = ProcessWindowStyle.Normal,
FileName = "cwebp.exe",
Arguments = string.Format(
"-q 100 -lossless -m 6 -alpha_q 100 \"{0}\" -o \"{1}\"", ms, "C:\test.webp")
},
};
cwebp.Start();
}
}
and then some random testing to get it to work....
And the thing i want to pass it to is cwebp, a Webp encoder.
Which is why i must use CMD, as i can´t work with it at the C# level, else i wouldn´t have this problem.

Yeah that is usually protected. If you know where it is, you might be able to grab it with an unsafe pointer. It might be easier to write it to a text file that cmd could read, or push it to Console to read.

If using .NET 4.0 or greater you can use a MemoryMappedFile. I haven't toyed with this class since 4.0 beta. However, my understanding is its useful for writing memory to disk in cases where you are dealing with large amounts of data or want some level of application memory sharing.
Usage per MSDN:
static void Main(string[] args)
{
long offset = 0x10000000; // 256 megabytes
long length = 0x20000000; // 512 megabytes
// Create the memory-mapped file.
using (var mmf = MemoryMappedFile.CreateFromFile(#"c:\ExtremelyLargeImage.data", FileMode.Open,"ImgA"))
{
// Create a random access view, from the 256th megabyte (the offset)
// to the 768th megabyte (the offset plus length).
using (var accessor = mmf.CreateViewAccessor(offset, length))
{
int colorSize = Marshal.SizeOf(typeof(MyColor));
MyColor color;
// Make changes to the view.
for (long i = 0; i < length; i += colorSize)
{
accessor.Read(i, out color);
color.Brighten(10);
accessor.Write(i, ref color);
}
}
}
}

If cwebp.exe is expecting a filename, there is nothing you can put on the command line that satisfies your criteria. Anything enough like a file that the external program can open it won't be able to get its data from your program's memory. There are a few possibilities, but they probably all require changes to cwebp.exe:
You can write to the new process's standard in
You can create a named pipe from which the process can read your data
You can create a named shared memory object from which the other process can read
You haven't said why you're avoiding writing to a file, so it's hard to say which is best.

BinaryFormatter.Serialize( Image ) - ExternalException - A generic error occurred in GDI+

When I try to Serialize some images using the BinaryFormatter, I'll get a ExternalException - A generic error occurred in GDI+." After scratching my head for awhile, I decided to create a simple test project to narrow down the problem:
static void Main(string[] args)
{
string file = #"C:\temp\delme.jpg";
//Image i = new Bitmap(file);
//using(FileStream fs = new FileStream(file, FileMode.Open, FileAccess.Read))
byte[] data = File.ReadAllBytes(file);
using(MemoryStream originalms = new MemoryStream(data))
{
using (Image i = Image.FromStream(originalms))
{
BinaryFormatter bf = new BinaryFormatter();
using (MemoryStream ms = new MemoryStream())
{
// Throws ExternalException on Windows 7, not Windows XP
bf.Serialize(ms, i);
}
}
}
}
For specific images, I've tried all sorts of ways of loading the image and I could not get it to work under Windows 7, even when running the program as Administrator.
I've copied the exact same executable and image into my Windows XP VMWare instance and I have no problems.
Anyone have any idea of why for some images it doesn't work under Windows 7, but works under XP?
Here's one of the images:
http://www.2shared.com/file/7wAXL88i/SO_testimage.html
delme.jpg md5: 3d7e832db108de35400edc28142a8281

As the OP pointed out, the code provided throws an exception that seems to be occurring only with the image he provided but works fine with other images on my machine.
Option 1
static void Main(string[] args)
{
string file = #"C:\Users\Public\Pictures\delme.jpg";
byte[] data = File.ReadAllBytes(file);
using (MemoryStream originalms = new MemoryStream(data))
{
using (Image i = Image.FromStream(originalms))
{
BinaryFormatter bf = new BinaryFormatter();
using (MemoryStream ms = new MemoryStream())
{
// Throws ExternalException on Windows 7, not Windows XP
//bf.Serialize(ms, i);
i.Save(ms, System.Drawing.Imaging.ImageFormat.Bmp); // Works
i.Save(ms, System.Drawing.Imaging.ImageFormat.Png); // Works
i.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg); // Fails
}
}
}
}
It could be that the image in question was created with a tool that added some additional information that is interfering with the JPEG serialization.
P.S. The image can be saved to memory stream using BMP or PNG format. If changing the format is an option, then you can try out either of these or any other format defined in ImageFormat.
Option 2
If your goal is just to get the contents of the image file into a memory stream, then doing just the following would help
static void Main(string[] args)
{
string file = #"C:\Users\Public\Pictures\delme.jpg";
using (FileStream fileStream = File.OpenRead(file))
{
MemoryStream memStream = new MemoryStream();
memStream.SetLength(fileStream.Length);
fileStream.Read(memStream.GetBuffer(), 0, (int)fileStream.Length);
}
}

Although the Bitmap class is marked as [Serializable], it does not actually support serialisation. The best you can do is serialise the byte[] containing the raw image data and then re-create it using a MemoryStream and the Image.FromStream() method.
I can't explain the inconsistent behaviour you're experiencing; for me, it fails unconditionally (although I first discovered this when trying to marshal images between different app domains, rather than manually serialising them).

I don't know for sure but I would lean towards different security models for XP vs Windows 7. Image inherits from System.MarshalByRefObject. There is probably proxying going on between application domains when serialization is performed. This proxying might be forbidden in Windows 7.
http://msdn.microsoft.com/en-us/library/system.marshalbyrefobject%28v=vs.71%29.aspx

Decompress tar files using C#

I'm searching a way to add embedded resource to my solution. This resources will be folders with a lot of files in them. On user demand they need to be decompressed.
I'm searching for a way do store such folders in executable without involving third-party libraries (Looks rather stupid, but this is the task).
I have found, that I can GZip and UnGZip them using standard libraries. But GZip handles single file only. In such cases TAR should come to the scene. But I haven't found TAR implementation among standard classes.
Maybe it possible decompress TAR with bare C#?

While looking for a quick answer to the same question, I came across this thread, and was not entirely satisfied with the current answers, as they all point to using third-party dependencies to much larger libraries, all just to achieve simple extraction of a tar.gz file to disk.
While the gz format could be considered rather complicated, tar on the other hand is quite simple. At its core, it just takes a bunch of files, prepends a 500 byte header (but takes 512 bytes) to each describing the file, and writes them all to single archive on a 512 byte alignment. There is no compression, that is typically handled by compressing the created file to a gz archive, which .NET conveniently has built-in, which takes care of all the hard part.
Having looked at the spec for the tar format, there are only really 2 values (especially on Windows) we need to pick out from the header in order to extract the file from a stream. The first is the name, and the second is size. Using those two values, we need only seek to the appropriate position in the stream and copy the bytes to a file.
I made a very rudimentary, down-and-dirty method to extract a tar archive to a directory, and added some helper functions for opening from a stream or filename, and decompressing the gz file first using built-in functions.
The primary method is this:
public static void ExtractTar(Stream stream, string outputDir)
{
var buffer = new byte[100];
while (true)
{
stream.Read(buffer, 0, 100);
var name = Encoding.ASCII.GetString(buffer).Trim('\0');
if (String.IsNullOrWhiteSpace(name))
break;
stream.Seek(24, SeekOrigin.Current);
stream.Read(buffer, 0, 12);
var size = Convert.ToInt64(Encoding.ASCII.GetString(buffer, 0, 12).Trim(), 8);
stream.Seek(376L, SeekOrigin.Current);
var output = Path.Combine(outputDir, name);
if (!Directory.Exists(Path.GetDirectoryName(output)))
Directory.CreateDirectory(Path.GetDirectoryName(output));
using (var str = File.Open(output, FileMode.OpenOrCreate, FileAccess.Write))
{
var buf = new byte[size];
stream.Read(buf, 0, buf.Length);
str.Write(buf, 0, buf.Length);
}
var pos = stream.Position;
var offset = 512 - (pos % 512);
if (offset == 512)
offset = 0;
stream.Seek(offset, SeekOrigin.Current);
}
}
And here is a few helper functions for opening from a file, and automating first decompressing a tar.gz file/stream before extracting.
public static void ExtractTarGz(string filename, string outputDir)
{
using (var stream = File.OpenRead(filename))
ExtractTarGz(stream, outputDir);
}
public static void ExtractTarGz(Stream stream, string outputDir)
{
// A GZipStream is not seekable, so copy it first to a MemoryStream
using (var gzip = new GZipStream(stream, CompressionMode.Decompress))
{
const int chunk = 4096;
using (var memStr = new MemoryStream())
{
int read;
var buffer = new byte[chunk];
do
{
read = gzip.Read(buffer, 0, chunk);
memStr.Write(buffer, 0, read);
} while (read == chunk);
memStr.Seek(0, SeekOrigin.Begin);
ExtractTar(memStr, outputDir);
}
}
}
public static void ExtractTar(string filename, string outputDir)
{
using (var stream = File.OpenRead(filename))
ExtractTar(stream, outputDir);
}
Here is a gist of the full file with some comments.

Tar-cs will do the job, but it is quite slow. I would recommend using SharpCompress which is significantly quicker. It also supports other compression types and it has been updated recently.
using System;
using System.IO;
using SharpCompress.Common;
using SharpCompress.Reader;
private static String directoryPath = #"C:\Temp";
public static void unTAR(String tarFilePath)
{
using (Stream stream = File.OpenRead(tarFilePath))
{
var reader = ReaderFactory.Open(stream);
while (reader.MoveToNextEntry())
{
if (!reader.Entry.IsDirectory)
{
ExtractionOptions opt = new ExtractionOptions {
ExtractFullPath = true,
Overwrite = true
};
reader.WriteEntryToDirectory(directoryPath, opt);
}
}
}
}

See tar-cs
using (FileStream unarchFile = File.OpenRead(tarfile))
{
TarReader reader = new TarReader(unarchFile);
reader.ReadToEnd("out_dir");
}

Since you are not allowed to use outside libraries, you are not restricted to a specific format of the tar file either. In fact, they don't even need it to be all in the same file.
You can write your own tar-like utility in C# that walks a directory tree, and produces two files: a "header" file that consists of a serialized dictionary mapping System.IO.Path instances to an offset/length pairs, and a big file containing the content of individual files concatenated into one giant blob. This is not a trivial task, but it's not overly complicated either.

there are 2 ways to compress/decompress in .NET first you can use Gzipstream class and DeflatStream both can actually do compress your files in .gz format so if you compressed any file in Gzipstream it can be opened with any popular compression applications such as winzip/ winrar, 7zip but you can't open compressed file with DeflatStream. these two classes are from .NET 2.
and there is another way which is Package class it's actually same as Gzipstream and DeflatStream the only different is you can compress multiple files which then can be opened with winzip/ winrar, 7zip.so that's all .NET has. but it's not even generic .zip file,
it something Microsoft uses to compress their *x extension office files. if you decompress any docx file with package class you can see everything stored in it. so don't use .NET libraries for compressing or even decompressing cause you can't even make a generic compress file or even decompress a generic zip file. you have to consider for a third party library such as
http://www.icsharpcode.net/OpenSource/SharpZipLib/
or implement everything from the ground floor.