Develop Reference

How to get the length of the compressed data from DeflateStream?

The following is a simple compression method I wrote using DeflateStream:
public static int Compress(
byte[] inputData,
int inputStartIndex,
int inputLength,
byte[] outputData,
int outputStartIndex,
int outputLength)
{
if (inputData == null)
throw new ArgumentNullException("inputData must be non-null");
MemoryStream memStream = new MemoryStream(outputData, outputStartIndex, outputLength);
using (DeflateStream dstream = new DeflateStream(memStream, CompressionLevel.Optimal))
{
dstream.Write(inputData, inputStartIndex, inputLength);
return (int)(memStream.Position - outputStartIndex);
}
}
What is special in this method is that I didn't use the parameter-less constructor of MemoryStream. This is because it is a high-throughput server. Array outputData is rented from ArrayPool, to be used to hold the compressed bytes, so that after I make use of it I can return it to ArrayPool.
The compression happened properly, and the compressed data is properly placed into outputData, but memStream.Position was zero, so I can't find out how many bytes have been written into the MemoryStream.
Only part of outputData is occupied by the compressed data. How do I find out the length of the compressed data?

MemoryStream.Position is 0 because data was not actually written there yet at the point you read Position. Instead, tell DeflateStream to leave underlying stream (MemoryStream) open, then dispose DeflateStream. At this point you can be sure it's done writing whatever it needs. Now you can read MemoryStream.Position to check how many bytes were written:
public static int Compress(
byte[] inputData,
int inputStartIndex,
int inputLength,
byte[] outputData,
int outputStartIndex,
int outputLength)
{
if (inputData == null)
throw new ArgumentNullException("inputData must be non-null");
using (var memStream = new MemoryStream(outputData, outputStartIndex, outputLength)) {
// leave open
using (DeflateStream dstream = new DeflateStream(memStream, CompressionLevel.Optimal, leaveOpen: true)) {
dstream.Write(inputData, inputStartIndex, inputLength);
}
return (int) memStream.Position; // now it's not 0
}
}
You also don't need to substract outputStartIndex, because Position is already relative to that index you passed to constructor.

Decryption providing a padding error

I'm trying to save a serialized object to an encrypted file. This isn't production quality and I am aware of the security risks with the way that I am doing this, but ignoring those I will have a key in a resource (data.Settings.key) that wont change and I have a salt that is also a constant.
My encryption seems to work, but decryption returns me an Exception saying that padding is invalid and cannot be closed when I try to close my CryptoStream.
private static byte[] decrypt(byte[] bytes)
{
var decryptor = algorithm.CreateDecryptor();
using (var sMemoryStream = new MemoryStream())
using (var sCryptoStream = new CryptoStream(sMemoryStream, decryptor, CryptoStreamMode.Write))
{
sCryptoStream.Write(bytes, 0, bytes.Length);
sCryptoStream.Close();
return sMemoryStream.ToArray();
}
}
The algorithm variable is the same one that the encrypt method uses and is built by this method which is called in the classes constructor:
private static SymmetricAlgorithm GetAlgorithm()
{
var algorithm = Rijndael.Create();
// Create key from salt and password in config
var rdb = new Rfc2898DeriveBytes(data.Settings.key, new byte[] {
0x44,0x61,0x79,0x6e,0x65,0x44,0x6f,0x75,0x67,0x61,0x6e
});
algorithm.Padding = PaddingMode.PKCS7;
// Set key and IV from rdb
algorithm.Key = rdb.GetBytes(32);
algorithm.IV = rdb.GetBytes(16);
return algorithm;
}
I've tried changing the padding mode in the algorithm but I can't understand why it's fine with this padding when encrypting, but now when decrypting.
If it helps here is the method that calls the decrypt method:
private static User OpenFile(String sUserName)
{
Console.WriteLine("Opening file...");
using (Stream sFileStream = new FileStream(data.Settings.dir + "data\\accounts\\" + sUserName + ".dat",
FileMode.Open, FileAccess.Read, FileShare.None))
using (Stream sMemoryStream = new MemoryStream())
{
// Read from File to memory stream
sFileStream.CopyTo(sMemoryStream);
// Decrypt data and store in new memory stream
byte[] bytes = new byte[sMemoryStream.Length];
Console.WriteLine("\tb:" + bytes.Length);
bytes = decrypt(bytes);
Console.WriteLine("\ta:" + bytes.Length);
Stream stream = new MemoryStream(bytes);
Console.WriteLine("\ts:" + bytes.Length);
// Deserialise memory stream and return as User object
User user = (User)bfFormatter.Deserialize(stream);
stream.Close();
return user;
}
}

Add IV to beginning of CryptoStream

I'm implementing local encryption within an existing file management program.
Much of the example code I can find, such as Microsoft's, demonstrates how to write directly to a file, but what I need to do is provide a stream that is consumed elsewhere in the program:
CryptoStream GetEncryptStream(string filename)
{
var rjndl = new RijndaelManaged();
rjndl.KeySize = 256;
rjndl.BlockSize = 256;
rjndl.Mode = CipherMode.CBC;
rjndl.Padding = PaddingMode.PKCS7;
// Open read stream of unencrypted source fileStream:
var fileStream = new FileStream(filename, FileMode.Open);
/* Get key and iv */
var transform = rjndl.CreateEncryptor(key, iv);
// CryptoStream in *read* mode:
var cryptoStream = new CryptoStream(fileStream, transform, CryptoStreamMode.Read);
/* What can I do here to insert the unencrypted IV at the start of the
stream so that the first X bytes returned by cryptoStream.Read are
the IV, before the bytes of the encrypted file are returned? */
return cryptoStream; // Return CryptoStream to be consumed elsewhere
}
My issue is outlined in the comment on the last line but one: how can I add the IV to the start of the CryptoStream such that it will be the first X bytes returned when the CryptoStream is read, given that control of when to actually start reading the stream and writing to a file is outside the scope of my code?

Ok... now that your problem is clear, it is "quite" easy... Sadly .NET doesn't include a class to merge two Stream, but we can easily create it. The MergedStream is a read-only, forward-only multi-Stream merger.
You use like:
var mergedStream = new MergedStream(new Stream[]
{
new MemoryStream(iv),
cryptoStream,
});
Now... When someone tries to read from the MergedStream, first the MemoryStream containing the IV will be consumed, then the cryptoStream will be consumed.
public class MergedStream : Stream
{
private Stream[] streams;
private int position = 0;
private int currentStream = 0;
public MergedStream(Stream[] streams) => this.streams = streams;
public override bool CanRead => true;
public override bool CanSeek => false;
public override bool CanWrite => false;
public override long Length => streams.Sum(s => s.Length);
public override long Position
{
get => position;
set => throw new NotSupportedException();
}
public override void Flush()
{
}
public override int Read(byte[] buffer, int offset, int count)
{
if (streams == null)
{
throw new ObjectDisposedException(nameof(MergedStream));
}
if (currentStream >= streams.Length)
{
return 0;
}
int read;
while (true)
{
read = streams[currentStream].Read(buffer, offset, count);
position += read;
if (read != 0)
{
break;
}
currentStream++;
if (currentStream == streams.Length)
{
break;
}
}
return read;
}
public override long Seek(long offset, SeekOrigin origin)
=> throw new NotSupportedException();
public override void SetLength(long value)
=> throw new NotSupportedException();
public override void Write(byte[] buffer, int offset, int count)
=> throw new NotSupportedException();
protected override void Dispose(bool disposing)
{
try
{
if (disposing && streams != null)
{
for (int i = 0; i < streams.Length; i++)
{
streams[i].Close();
}
}
}
finally
{
streams = null;
}
}
}

It is not a good design to use a CryptoStream to communicate between two local parties. You should use a generic InputStream or pipe (for inter-process communication) instead. Then you can combine a MemoryStream for the IV and a CryptoStream and return the combination. See the answer of xanatos on how to do this (you may still need to fill in the Seek functionality if that's required).
A CryptoStream will only ever be able to handle ciphertext. As you need to change the code at the receiver anyway if you'd want to decrypt you might as well refactor to InputStream.
If you're required to keep the current design then there is a hack available. First "decrypt" the IV using ECB mode without padding. As a single block cipher call always succeeds the result will be a block of data that - when encrypted using the CipherStream - turns into the IV again.
Steps:
generate 16 random bytes in an array, this will be the real IV;
decrypt the 16 byte IV using ECB without padding and the key used for the CipherStream;
initialize the CipherStream using the key and an all zero, 16 byte IV;
encrypt the "decrypted" IV using the CipherStream;
input the rest of the plaintext.
You will need to create an InputStream that first receives the decrypted IV (as MemoryStream) and then the plaintext (as FileStream) for this to be feasible. Again, also see the answer of xanatos on how to do this. Or see for instance this combiner and this HugeStream on good ol' StackOverflow. Then use the combined stream as source for the CipherInputStream.
But needless to say hacks like these should be well documented and removed at the earliest convenience.
Notes:
This trick won't work on any mode; it works for CBC mode, but other modes may use the IV differently;
Note that an OutputStream would generally make more sense for encryption, there may be other things wrong with the design.

Thanks for those who've taken the time to answer. In the end I realized I have to have knowledge of the IV length in the buffering code, there's no way around it, so elected to keep it simple:
Encryption method (Pseudo-code):
/* Get key and IV */
outFileStream.Write(IV); // Write IV to output stream
var transform = rijndaelManaged.CreateEncryptor(key, iv);
// CryptoStream in read mode:
var cryptoStream = new CryptoStream(inFileStream, transform, CryptoStreamMode.Read);
do
{
cryptoStream.Read(chunk, 0, blockSize); // Get and encrypt chunk
outFileStream.Write(chunk); // Write chunk
}
while (chunk.Length > 0)
/* Cleanup */
Decryption method (Pseudo-code):
/* Get key */
var iv = inFileStream.Read(ivLength); // Get IV from input stream
var transform = rijndaelManaged.CreateDecryptor(key, iv);
// CryptoStream in write mode:
var cryptoStream = new CryptoStream(outFileStream, transform, CryptoStreamMode.Write);
do
{
inFileStream.Read(chunk, 0, blockSize); // Get chunk
cryptoStream.Write(chunk); // Decrypt and write chunk
}
while (chunk.Length > 0)
/* Cleanup */

FileStream.Read() Throws ArgumentException

I would like to make some secure container for my application, and here's the map :
I finished opening/saving code now, and tested it, however, ArgumentException was thrown.
The code will run like this.
Create byte[] type variable for containing not crypted user data.
FileStream Writes Magic Number to first 5 bytes.
RijndaelManaged accepts key, and generates Initialization Vector.
FileStream Writes Initialization Vector to next 16 bytes. <- Exception thrown!
CryptoStream transform the variable from 1.
FileStream Writes the crypted data from 22th bytes.
Debugging, and I found the reason that FileStream.Read() has been thrown the Exception. and the message is:
Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection.
I tried to set the length of stream as (user data) + 21. but it doesn't work. I attach entire code for saving file, and I hope this problem will be solved.
Thank you!
private bool SaveFile(string FilePath, bool IsCrypt)
{
byte[] Data = Encoding.UTF8.GetBytes(WorkspaceList[CurrentIndex]._textbox.Text);
using (var Stream = new FileStream(FilePath, FileMode.OpenOrCreate, FileAccess.ReadWrite))
{
if (IsCrypt)
{
Stream.SetLength(Data.Length + 21); // Tried when I got Exception
Stream.Write(MagicNumber, 0, 5); //Magic Number
using (var CryptoHandler = new RijndaelManaged()) // AES256 Encryption
{
CryptoHandler.BlockSize = 128;
CryptoHandler.KeySize = 256;
CryptoHandler.Padding = PaddingMode.PKCS7;
CryptoHandler.Mode = CipherMode.CBC;
var tempKey = WorkspaceList[CurrentIndex]._cryptkey;
if(tempKey.Length < 32)
{
tempKey.PadRight(32);
}
else if (tempKey.Length > 32)
{
tempKey.Remove(33);
}
CryptoHandler.Key = Encoding.UTF8.GetBytes(WorkspaceList[CurrentIndex]._cryptkey.PadRight(32));
CryptoHandler.GenerateIV();
Stream.Write(CryptoHandler.IV, 5, 16); //IV Insertion *** ArgumentException ***
var CryptoInstance = CryptoHandler.CreateEncryptor(CryptoHandler.Key, CryptoHandler.IV);
using (var MemoryHandler = new MemoryStream())
{
using (var Crypto = new CryptoStream(MemoryHandler, CryptoInstance, CryptoStreamMode.Write))
{
byte[] _Buffer = Data;
Crypto.Read(Data, 0, Data.Length);
_Buffer = MemoryHandler.ToArray();
Stream.Write(_Buffer, 21, _Buffer.Length); // Insert Crypted Data
Stream.Close();
return true;
}
}
}
}
else
{
Stream.Write(Data, 0, Data.Length);
Stream.Close();
return true;
}
}
}

Replace Stream.Write(CryptoHandler.IV, 5, 16); //IV Insertion
With Stream.Write(CryptoHandler.IV, 0, CryptoHandler.IV.Length); //IV Insertion
array = CryptoHandler.IV (the data you want to write)
offset = 0 (you write from the first byte of array)
count = CryptoHandler.IV.Length (you write all bytes from CryptoHandler.IV)
Note that offset is intrinsic to array, not to the Stream. After a successful Write operation, the stream cursor waits at the last written position. I suppose you specified an offset of 5 to take into account the MagicNumber?
You would have add the same problem with Stream.Write(_Buffer, 21, _Buffer.Length);

Writing to ZipArchive using the HttpContext OutputStream

I've been trying to get the "new" ZipArchive included in .NET 4.5 (System.IO.Compression.ZipArchive) to work in a ASP.NET site. But it seems like it doesn't like writing to the stream of HttpContext.Response.OutputStream.
My following code example will throw
System.NotSupportedException: Specified method is not supported
as soon as a write is attempted on the stream.
The CanWrite property on the stream returns true.
If I exchange the OutputStream with a filestream, pointing to a local directory, it works. What gives?
ZipArchive archive = new ZipArchive(HttpContext.Response.OutputStream, ZipArchiveMode.Create, false);
ZipArchiveEntry entry = archive.CreateEntry("filename");
using (StreamWriter writer = new StreamWriter(entry.Open()))
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
}
Stacktrace:
[NotSupportedException: Specified method is not supported.]
System.Web.HttpResponseStream.get_Position() +29
System.IO.Compression.ZipArchiveEntry.WriteLocalFileHeader(Boolean isEmptyFile) +389
System.IO.Compression.DirectToArchiveWriterStream.Write(Byte[] buffer, Int32 offset, Int32 count) +94
System.IO.Compression.WrappedStream.Write(Byte[] buffer, Int32 offset, Int32 count) +41

Note: This has been fixed in .Net Core 2.0. I'm not sure what is the status of the fix for .Net Framework.
Calbertoferreira's answer has some useful information, but the conclusion is mostly wrong. To create an archive, you don't need seek, but you do need to be able to read the Position.
According to the documentation, reading Position should be supported only for seekable streams, but ZipArchive seems to require this even from non-seekable streams, which is a bug.
So, all you need to do to support writing ZIP files directly to OutputStream is to wrap it in a custom Stream that supports getting Position. Something like:
class PositionWrapperStream : Stream
{
private readonly Stream wrapped;
private long pos = 0;
public PositionWrapperStream(Stream wrapped)
{
this.wrapped = wrapped;
}
public override bool CanSeek { get { return false; } }
public override bool CanWrite { get { return true; } }
public override long Position
{
get { return pos; }
set { throw new NotSupportedException(); }
}
public override void Write(byte[] buffer, int offset, int count)
{
pos += count;
wrapped.Write(buffer, offset, count);
}
public override void Flush()
{
wrapped.Flush();
}
protected override void Dispose(bool disposing)
{
wrapped.Dispose();
base.Dispose(disposing);
}
// all the other required methods can throw NotSupportedException
}
Using this, the following code will write a ZIP archive into OutputStream:
using (var outputStream = new PositionWrapperStream(Response.OutputStream))
using (var archive = new ZipArchive(outputStream, ZipArchiveMode.Create, false))
{
var entry = archive.CreateEntry("filename");
using (var writer = new StreamWriter(entry.Open()))
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
}
}

If you compare your code adaptation with the version presented in MSDN page you'll see that the ZipArchiveMode.Create is never used, what is used is ZipArchiveMode.Update.
Despite that, the main problem is the OutputStream that doesn't support Read and Seek which is need by the ZipArchive in Update Mode:
When you set the mode to Update, the underlying file or stream must
support reading, writing, and seeking. The content of the entire
archive is held in memory, and no data is written to the underlying
file or stream until the archive is disposed.
Source: MSDN
You weren't getting any exceptions with the create mode because it only needs to write:
When you set the mode to Create, the underlying file or stream must support writing, but does not have to support seeking. Each entry in the archive can be opened only once for writing. If you create a single entry, the data is written to the underlying stream or file as soon as it is available. If you create multiple entries, such as by calling the CreateFromDirectory method, the data is written to the underlying stream or file after all the entries are created.
Source: MSDN
I believe you can't create a zip file directly in the OutputStream since it's a network stream and seek is not supported:
Streams can support seeking. Seeking refers to querying and modifying the current position within a stream. Seek capability depends on the kind of backing store a stream has. For example, network streams have no unified concept of a current position, and therefore typically do not support seeking.
An alternative could be writing to a memory stream, then use the OutputStream.Write method to send the zip file.
MemoryStream ZipInMemory = new MemoryStream();
using (ZipArchive UpdateArchive = new ZipArchive(ZipInMemory, ZipArchiveMode.Update))
{
ZipArchiveEntry Zipentry = UpdateArchive.CreateEntry("filename.txt");
foreach (ZipArchiveEntry entry in UpdateArchive.Entries)
{
using (StreamWriter writer = new StreamWriter(entry.Open()))
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
}
}
}
byte[] buffer = ZipInMemory.GetBuffer();
Response.AppendHeader("content-disposition", "attachment; filename=Zip_" + DateTime.Now.ToString() + ".zip");
Response.AppendHeader("content-length", buffer.Length.ToString());
Response.ContentType = "application/x-compressed";
Response.OutputStream.Write(buffer, 0, buffer.Length);
EDIT: With feedback from comments and further reading, you could be creating large Zip files, so the memory stream could cause you problems.
In this case i suggest you create the zip file on the web server then output the file using Response.WriteFile .

A refinement to svick's answer of 2nd February 2014. I found that it was necessary to implement some more methods and properties of the Stream abstract class and to declare the pos member as long. After that it worked like a charm. I haven't extensively tested this class, but it works for the purposes of returning a ZipArchive in the HttpResponse. I assume I've implemented Seek and Read correctly, but they may need some tweaking.
class PositionWrapperStream : Stream
{
private readonly Stream wrapped;
private long pos = 0;
public PositionWrapperStream(Stream wrapped)
{
this.wrapped = wrapped;
}
public override bool CanSeek
{
get { return false; }
}
public override bool CanWrite
{
get { return true; }
}
public override long Position
{
get { return pos; }
set { throw new NotSupportedException(); }
}
public override bool CanRead
{
get { return wrapped.CanRead; }
}
public override long Length
{
get { return wrapped.Length; }
}
public override void Write(byte[] buffer, int offset, int count)
{
pos += count;
wrapped.Write(buffer, offset, count);
}
public override void Flush()
{
wrapped.Flush();
}
protected override void Dispose(bool disposing)
{
wrapped.Dispose();
base.Dispose(disposing);
}
public override long Seek(long offset, SeekOrigin origin)
{
switch (origin)
{
case SeekOrigin.Begin:
pos = 0;
break;
case SeekOrigin.End:
pos = Length - 1;
break;
}
pos += offset;
return wrapped.Seek(offset, origin);
}
public override void SetLength(long value)
{
wrapped.SetLength(value);
}
public override int Read(byte[] buffer, int offset, int count)
{
pos += offset;
int result = wrapped.Read(buffer, offset, count);
pos += count;
return result;
}
}

An simplified version of svick's answer for zipping a server-side file and sending it via the OutputStream:
using (var outputStream = new PositionWrapperStream(Response.OutputStream))
using (var archive = new ZipArchive(outputStream, ZipArchiveMode.Create, false))
{
var entry = archive.CreateEntryFromFile(fullPathOfFileOnDisk, fileNameAppearingInZipArchive);
}
(In case this seems obvious, it wasn't to me!)

Presumably this is not an MVC app, where you could easily just use the FileStreamResult class.
I'm using this currently with ZipArchive created using a MemoryStream, so I know it works.
With that in mind, have a look at the FileStreamResult.WriteFile() method:
protected override void WriteFile(HttpResponseBase response)
{
// grab chunks of data and write to the output stream
Stream outputStream = response.OutputStream;
using (FileStream)
{
byte[] buffer = newbyte[_bufferSize];
while (true)
{
int bytesRead = FileStream.Read(buffer, 0, _bufferSize);
if (bytesRead == 0)
{
// no more data
break;
}
outputStream.Write(buffer, 0, bytesRead);
}
}
}
(Entire FileStreamResult on CodePlex)
Here is how I'm generating and returning the ZipArchive.
You should have no issues replacing the FSR with the guts of the WriteFile method from above, where FileStream becomes resultStream from the code below:
var resultStream = new MemoryStream();
using (var zipArchive = new ZipArchive(resultStream, ZipArchiveMode.Create, true))
{
foreach (var doc in req)
{
var fileName = string.Format("Install.Rollback.{0}.v{1}.docx", doc.AppName, doc.Version);
var xmlData = doc.GetXDocument();
var fileStream = WriteWord.BuildFile(templatePath, xmlData);
var docZipEntry = zipArchive.CreateEntry(fileName, CompressionLevel.Optimal);
using (var entryStream = docZipEntry.Open())
{
fileStream.CopyTo(entryStream);
}
}
}
resultStream.Position = 0;
// add the Response Header for downloading the file
var cd = new ContentDisposition
{
FileName = string.Format(
"{0}.{1}.{2}.{3}.Install.Rollback.Documents.zip",
DateTime.Now.Year, DateTime.Now.Month, DateTime.Now.Day, (long)DateTime.Now.TimeOfDay.TotalSeconds),
// always prompt the user for downloading, set to true if you want
// the browser to try to show the file inline
Inline = false,
};
Response.AppendHeader("Content-Disposition", cd.ToString());
// stuff the zip package into a FileStreamResult
var fsr = new FileStreamResult(resultStream, MediaTypeNames.Application.Zip);
return fsr;
Finally, if you will be writing large streams (or a larger number of them at any given time), then you may want to consider using anonymous pipes to write the data to the output stream immediately after you write it to the underlying stream in the zip file. Because you will be holding all the file contents in memory on the server. The end of this answer to a similar question has a nice explanation of how to do that.