Develop Reference

How best to encrypt/decrypt very large files in .Net Core

My .Net Core 2 application generates very large files (80 MB+), using a third party library method. This method writes to a Stream. The end user can download the generated files via an MVC web page, which returns the file as a stream to the browser, using a FileStreamResult.
Normally, I would use a FileStream to write the file to disk. And another FileStream in my MVC action method to read the file.
However, the files now have to be encrypted while stored on disk on the server (they contain very sensitive information).
Because the files are very large, I want to use a Stream based approach to encrypting/decryping them, rather than loading them into memory.
I would like to use the new .Net Core data protection system, using IDataProtectionProvider and IDataProtector, because it is very easy to use, manages keys for me, etc. However, its IDataProtector.Protect and Unprotect methods simply encrypt/decrypt byte arrays. So they are not stream based.
The older CryptoStream class does support Streams, so I could use that. But I would have to do my own key management. And because other parts of my application already use the .Net Core data protection system, I would wind up with two ways of doing encryption/decryption.
How can I encrypt/decrypt 80MB+ files using the new .Net Core data protection system?
Alternatively, is there a way to use the .Net Core data protection system with CryptoStreamd?
Or is there some other option that I missed?

As usual, Microsoft documentation on these classes is literally as unhelpful as it could possibly be, but I'd say it is a fair assumption that these classes use an AEAD cipher mode underneath. This makes them inappropriate for use with files of the size you are dealing with and also explains why no streaming interface is available.
I suggest using CryptoStream as you outline in your answer, but consider the difficulties you will encounter around integrity - you'll need to pass through the stream once to produce and confirm the MAC and then pass through again to actually decrypt - otherwise you'll end up streaming decrypted data to the consumer before verifying if the MAC is actually correct.

EXI decoder using OpenEXI in C#/.NET

I cannot find out how to create an EXI decoder using C#/.NET which accepts a MemoryStream containing EXI valid code and simply outputs another MemoryStream containing XML code. I will parse XML code later with custom methods; I'm using EXI only to achieve best compression performances and low memory footprint. Until now, I have found some Java implementations as examples, but none for C#/.NET counterpart; hints of any kind are really appreciated.

zlib.h versus zlib.net

I have made a compression in c/c++ (NO UNDER THE CLR) using the library zlib.h, and it works great. the functions that I use deflate() and inflate().
Now the file compressed by the c application, I want to decompress it with the zlib.net application, using c#, but I do not manage to get it working. When trying to decompress it, I get the error of the magic number, a number used by an specific application in the header. Does anyone know how to get through this problem, or if someone can give me an example of the inflate()/deflate() functionality in .net
for more info on how I have done my compression, is similar to the one in the link http://www.zlib.net/zlib_how.html
Also, can any one advice me of a good lib to perform compression in both c++ and .net,
Many thanks in advance...

There's some discussion on this here: Zlib-compatible compression streams?
I think Boost may work with zlib to add the header information: http://www.boost.org/doc/libs/1_36_0/libs/iostreams/doc/classes/gzip.html

The best way to Compress XML

I need to compress a very large xml file to the smallest possible size.
I work in C#, and I prefer it to be some open source or application that I can access thru my code, but I can handle an algorithm as well.
Thank you!

It may not be the "smallest size possible", but you could use use System.IO.Compression to compress it. Zipping tends to provide very good compression for text.
using (var fileStream = File.OpenWrite(...))
using (var zipStream = new GZipStream(fileStream, CompressionMode.Compress))
{
zipStream.Write(...);
}

As stated above, Efficient XML Interchange (EXI) achieves the best available XML compression pretty consistently. Even without schemas, it is not uncommon for EXI to be 2-5 times smaller than zip. With schemas, you'll do even better.
If you're not opposed to a commercial implementation, you can use the .NET version of Efficient XML and call it directly from your C# code using standard .NET APIs. You can download a free trial copy from http://www.agiledelta.com/efx_download.html.

have a look at XML Compression Tools you can also compress it using SharpZipLib

If you have a schema available for the XML file, you could try EXIficient. It is an implementation of the Efficient XML Interchange (EXI) format that is pretty much the best available general-purpose XML compression method. If you don't have a schema, EXI is still better than regular zip (the deflate algorithm, that is), but not very much, especially for large files.
EXIficient is only Java but you can probably make it into an application that you can call. I'm not aware of any open-source implementations of EXI in C#.

File size is not the only advantage of EXI (or any binary scheme). The processing time and memory overhead are also greatly reduced when reading/writing it. Imagine a program that copies floating point numbers to disk by simply copying the bytes. Now imagine another program converts the floating point numbers to formatted text, and pastes them into a text stream, and then feeds that stream through an expensive compression algorithm. Because of this ridiculous overhead, XML is basically unusable for very large files that could have been effortlessly processed with a binary representation.
Binary XML promises to address this longstanding weakness of XML. It would be very easy to make a utility that converts between binary/text representations (without knowing the XML schema), which means you can still edit the files easily when you want to.

XML is highly compressible. You can use DotNetZip to produce compressed zip files from you XML.

if you require maximum compression level i would recommend LZMA. There is a SDK (including C#) that is part of the open source 7-Zip project, available here.

If you are looking for the smallest possible size then try Fast Infoset as binary XML encoding and then compress using BZIP2 or LZMA. You will probably get better results than compressing text XML or using EXI. FastInfoset.NET includes implementations of the Fast Infoset standard and several compression formats to choose from but it's commercial.

DeflateStream 4GB Limit in .NET

From MSDN: DeflateStream Class
DeflateStream cannot be used to compress files larger than 4 GB.
Are there any other implementations for .NET without the 4 GB limit?
NOTE: I really need to decompress a file in GZ format with content larger than 4 GB. Can any code do that?

FYI, we have removed the 4 GB limit from DeflateStream in .NET 4.

There is sample code at CodeProject using the 7-Zip library.
The license is open, so you should be able to use this in your project.
7-Zip also supports GZ files.

Take a look at SharpZipLib. Not sure if it's subject to the same limitation, but worth a look.

Look for libraries that support DEFLATE64 (not Zip64, that's an extension to the ZIP file format). Xceed Zip for .NET does support Deflate64, and I'm sure others do too.

Having a look around, it seems a lot of people have encountered this problem.
System.IO.Compressio.DeflateStream clarifications please seems to be the most comprehensive.
The only implementation I was able to find that seems to overcome this problem by using Zip64 is
Xceed Zip for .NET.
However, it is very expensive and I am not sure if it would suite your needs.
Edit:
There does seem to be quite a number of implementations of Zip64 for .NET, but I can't find any that are free.

DotNetZip does ZIP64 for .NET, and it is free. But Zip64 is not the same as Deflate64.

Although that documentation says the 4GB limitation is for both the DeflateStream and GZipStream, only GZipStream is limited because of the CRC32 checksum. If you do not need CRC32 then use DeflateStream.