I want to compress and then encrypt my data, and for improved speed (by not having to write to byte arrays and back) decided to chain the streams used for compression and encryption together.
It works perfectly when I write (compress and encrypt) the data, but when I try to read the data (decompress and decrypt), the Read operation breaks - simply calling Read once reads exactly 0 bytes, because the first Read always returns 0. Looping as in the below code almost works, except that at a certain point, Read stops returning anything > 0 even though there's still data to be read.
Everything before those last few bytes are decompressed and decrypted perfectly.
The number of bytes left when that happens remains the same for the same plaintext; for example, it's always 9 bytes for a certain string, but always 1 byte for another.
The following is the relevant encryption and decryption code; any ideas as to what could be going wrong?
Encryption:
// Create the streams used for encryption.
using (MemoryStream msEncrypt = new MemoryStream())
{
using (ICryptoTransform encryptor = aes.CreateEncryptor(aes.Key, aes.IV))
using (CryptoStream csEncrypt = new CryptoStream(msEncrypt, encryptor, CryptoStreamMode.Write))
using (DeflateStream zip = new DeflateStream(csEncrypt, CompressionMode.Compress, true))
{
zip.Write(stringBytes, 0, stringBytes.Length);
csEncrypt.FlushFinalBlock();
Decryption:
// Create the streams used for decryption.
using (MemoryStream msDecrypt = new MemoryStream())
{
// Writes the actual data (sans prepended headers) to the stream
msDecrypt.Write(stringBytes, prependLength, stringBytes.Length - prependLength);
// Reset position to prepare for read
msDecrypt.Position = 0;
// init buffer to read to
byte[] buffer = new byte[originalSize];
using (ICryptoTransform decryptor = aes.CreateDecryptor(aes.Key, aes.IV))
using (CryptoStream csDecrypt = new CryptoStream(msDecrypt, decryptor, CryptoStreamMode.Read))
using (DeflateStream zip = new DeflateStream(csDecrypt, CompressionMode.Decompress))
{
// Hangs with "offset" at a small, deterministic number away from originalSize (I've gotten 9 less and 1 less for different strings)
// Loop fixed as per advice
int offset = 0;
while (offset < originalSize)
{
int read = zip.Read(buffer, offset, originalSize - offset);
if (read > 0)
offset += read;
else if (read < 0)
Console.WriteLine(read); // Catch it if it happens.
}
// Hangs with "left" at a small, deterministic number (I've gotten 9 and 1 for different strings)
/*
for (int left = buffer.Length; left > 0; )
left -= zip.Read(buffer, 0, left);
*/
Solution (Modification to Encryption):
// Create the streams used for encryption.
using (MemoryStream msEncrypt = new MemoryStream())
{
using (ICryptoTransform encryptor = aes.CreateEncryptor(aes.Key, aes.IV))
using (CryptoStream csEncrypt = new CryptoStream(msEncrypt, encryptor, CryptoStreamMode.Write))
{
using (DeflateStream zip = new DeflateStream(csEncrypt, CompressionMode.Compress, true))
zip.Write(stringBytes, 0, stringBytes.Length);
//Flush after DeflateStream is disposed.
csEncrypt.FlushFinalBlock();
The problem lies in the following line:
csEncrypt.FlushFinalBlock();
If you remove that, the code will work.
The reason is that when you write to DeflateStream, not all data is written to the underlying stream. That happens only when you call Close() or Dispose() explicitly or implicitly by leaving the using block.
So in your code, this happens:
You Write() all of the data to the DeflateStream, which in turn writes most of the data to the underlying CryptoStream.
You call csEncrypt.FlushFinalBlock(), which closes the CryptoStream.
You leave the using block of the DeflateStream, which tries to write the rest of the data to the already closed CryptoStream.
You leave the using block of the CryptoStream, which would call FlushFinalBlock(), if it wasn't called already.
The correct sequence of events is:
You Write() all of the data to the DeflateStream, which in turn writes most of the data to to the underlying CryptoStream.
You leave the using block of the DeflateStream, which writes the rest of the data to the already closed CryptoStream.
You leave the using block of the CryptoStream, which calls FlushFinalBlock().
Although I would expect that writing to a closed stream would fail with an exception. I'm not sure why that doesn't happen.
I had similar issue recently while reading from remote stream.
For me, solution was to change line which read entire stream in single call into while loop:
zip.Write(stringBytes, 0, stringBytes.Length);
Remote streams not always can return requested amount of data, so try read stream while you read enough bytes.
Related
My application requires the need to crypt parts of a stream to other streams, as some files have some parts encrypted with one key and others with other keys. To support this, I tried to make a method that crypts a part of a stream using an ICryptoTransform (yes, crypt, since ICryptoTransforms can be decryptors or encryptors) with a given offset and size, and it should be buffered.
This was my idea:
Open a buffer stream, for data chunks
Open a CryptoStream, and pass the buffer stream into it
Read and hopefully crypt it like this:
Read a chunk (the size of bufferSize) from the stream
Write that chunk into the crypto stream (which should write it to the
buffer stream)
Call .Flush() on that, to make sure the data
has been crypted
Write the contents of the buffer stream to the output stream
Seek the buffer stream back to the beginning, so that a new chunk can be written and crypted in it
Repeat until offset + size has been reached on the input stream.
This is my current code:
public static void CryptStreamPartBuffered(Stream input, Stream output, ICryptoTransform transform, long offset, long size, int bufferSize = 4000000)
{
using (MemoryStream ms = new MemoryStream()) //opening a memory stream, which will be the "buffer" for the crypted chunks
{
using (CryptoStream cs = new CryptoStream(ms, transform, CryptoStreamMode.Write)) //opening a cryptostream on the "buffer" stream, to crypt it's contents
{
input.Seek(offset, SeekOrigin.Begin); //seeking input stream to given start offset
byte[] buffer = new byte[bufferSize];
while (input.Position < offset + size)
{
int remaining = bufferSize, bytesRead;
while (remaining > 0 && (bytesRead = input.Read(buffer, 0, Math.Min(remaining, bufferSize))) > 0)
{
remaining -= bytesRead;
cs.Write(buffer); //writing current chunk of data to crypto stream
cs.Flush(); //making sure that the crypto stream has done its work on the current chunk (although I'm not really sure whether this is the right thing to do)
output.Write(ms.ToArray()); //writing the (hopefully) crypted data to the output stream
ms.Seek(0, SeekOrigin.Begin); //re-seeking the chunk stream to it's beginning, so that when the next chunk gets crypted, it can be written there
}
}
}
}
}
This straight-up doesn't work for files smaller than the default buffer size. While I do know that this can be fixed by just setting a smaller buffer size manually when calling the method, but having the support would be great.
However, this doesn't seem to do its job very well. I seem to get garbled data and I feel like I'm doing something wrong here, probably some very obvious mistake I just can't figure out.
I'm trying to encrypt/decrypt a string with AES, using streams. I'm using the following code for encryption:
var provider = Aes.Create();
provider.Mode = CipherMode.CBC;
provider.Padding = PaddingMode.PKCS7;
using var encryptor = provider.CreateEncryptor();
using var memoryStream = new MemoryStream();
using var cryptoStream = new CryptoStream(memoryStream, encryptor, CryptoStreamMode.Write);
using var streamWriter = new StreamWriter(cryptoStream, Encoding.UTF8);
streamWriter.Write(plainText);
cryptoStream.FlushFinalBlock();
var cipher = memoryStream.ToArray();
This successfully produces a byte array, though no matter the plaintext length, the cipher is always 16 bytes. From my understanding, with a block size of 16, a plaintext string with a length of 16 or more should result in a cipher that is larger than 16 bytes. Also, even for plaintext that is less than 16 bytes, decryption always results in an empty string.
var provider = Aes.Create();
provider.Mode = CipherMode.CBC;
provider.Padding = PaddingMode.PKCS7;
using var decryptor = _provider.CreateDecryptor(key, iv);
using var memoryStream = new MemoryStream(cipher);
using var cryptoStream = new CryptoStream(memoryStream, decryptor, CryptoStreamMode.Read);
using var streamReader = new StreamReader(cryptoStream, Encoding.UTF8);
var plainText = streamReader.ReadToEnd();
My code is based on this sample in the Microsoft docs, though I'm calling cryptoStream.FlushFinalBlock(), after writing to the stream, although this isn't working as desired.
Calling FlushFinalBlock is not really necessary if you correctly close the stream. It might be useful if you want to write the last block (including padding) without closing it.
However, using the generic streaming API, preferably with the using statement and closing the stream should write any bytes left in the buffer + any padding that could be required.
Of course you should include any stream that writes to the CryptoStream in that using statement, otherwise they may have leftover data. Of course the receiving stream should only be closed after the data has been retrieved, for instance in case a MemoryStream is used.
Let e = 'password' and I am transforming it to 'as9kio0736' in a CryptoStream.
Let d = 'as9kio0736' and I am transforming it to 'password in a CryptoStream.
When I am transforming d back to 'password' why is it not considered writing in a CryptoStream?
using (MemoryStream msEncrypt = new MemoryStream()) {
using (CryptoStream csEncrypt = new CryptoStream(msEncrypt, encryptor, CryptoStreamMode.Write)) {
using (StreamWriter swEncrypt = new StreamWriter(csEncrypt)) {
swEncrypt.Write(plainText);
}
}
}
using (MemoryStream msDecrypt = new MemoryStream(cipherText)) {
using (CryptoStream csDecrypt = new CryptoStream(msDecrypt, decryptor, CryptoStreamMode.Read)) {
using (StreamReader srDecrypt = new StreamReader(csDecrypt)) {
plaintext = srDecrypt.ReadToEnd();
}
}
}
CryptoStream is designed to perform transformation from a stream to another stream only and allows transformations chaining. For instance you can encrypt a data stream then Base 64 encode the encryption output.
Chose the mode depending on whether you want to write to the transformation input stream or read from the transformation output stream.
CryptoStream documentation is misleading.
The first CrytoStream constructor argument is described as:
"The stream on which to perform the cryptographic transformation"
This description is ok if constructor third argument value is CryptoStreamMode.Read.
But if third argument value is CryptoStreamMode.Write the first constructor argument description should be:
"The stream on which the result of cryptographic transformation is written to"
Also, documentation does not mention clearly that if you use CryptoStreamMode.Write, you MUST call FlushFinalBlock on your CryptoStream object after you finish writing.
To summarize this:
Writing to the transformation input stream:
CryptoStream constructor arguments:
argument 1: destination stream
argument 3: CryptoStreamMode.Write
CryptoStream object use:
Write data to the CryptoStream object
Call FlushFinalBlock on the CryptoStream object
Reading from the transformation output stream:
CryptoStream constructor arguments:
argument 1: source stream
argument 3: CryptoStreamMode.Read
CryptoStream object use:
Read data from the CryptoStream object until you reach the stream end
You can use CryptoStream in either direction for either operation; it's just where your data is and what you want to do with it.
If the data to process is already in a Stream (and you're okay with the stream getting drained and disposed), use CryptoStream in read mode and read the data out (including by using cryptoStream.CopyTo(someOtherStream)). If the data is in a byte[] and you want to write it to a Stream, use CryptoStream.Write.
In the .NET Core tests you can find examples both ways.
Using Read for both encrypt and decrypt: https://github.com/dotnet/corefx/blob/61fb32299a276f1aa4103c85dcec215dfddc252d/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/AES/AesCipherTests.cs#L466-L480
Using Write for both encrypt and decrypt: https://github.com/dotnet/corefx/blob/61fb32299a276f1aa4103c85dcec215dfddc252d/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/AES/AesCipherTests.cs#L609-L631
Heck, those are even in the same files. It's all just a matter of preference.
CryptoStream gives the flexibility to read/write for both Encrypt and Decrypt. For the case where the direction is fixed it can be changed to move in either direction regardless of the implementation.
Here's a way to do it:
https://github.com/djpai/StreamConduit
I am in the process of writing a small encoding class which takes a byte[] array, and cyphers it. As the IV can be made public, I wish to prepend it to the ciphertext to use for decryption.
Following the MSDN Documentation and some SO Post source (Cant find link), I started the stream with the raw IV;
//Start encryption process
using (var msEncrypt = new MemoryStream())
{
using (var csEncrypt = new CryptoStream(msEncrypt, encryptor.CreateEncryptor(), CryptoStreamMode.Write))
{
using (var swWriter = new StreamWriter(csEncrypt))
{
//Write all data to the stream.
swWriter.Write(encryptor.IV);
swWriter.Write(plainTextRaw);
}
}
Console.WriteLine("Encrypted Value: {0}", BitConverter.ToString(msEncrypt.ToArray()).Replace("-", String.Empty));
return msEncrypt.ToArray();
}
However, upon writing a simple unit test extracting the first 16 bytes, they do not seem to match.
I am certain the IV is being encrypted within the stream, so where is the best place to inject it?
Just write the IV tot the initial memory stream msEncrypt, not to the stream that is being encrypted swWriter.
Following feedback from Alexei, a simplification of the question:
How do I use a buffered Stream approach to convert the contents of a CryptoStream (using ToBase64Transform) into a StreamWriter (Unicode encoding) without using Convert.ToBase64String()?
Note: Calling Convert.ToBase64String() throws OutOfMemoryException, hence the need for a buffered/Stream approach to the conversion.
You probably should implement custom Stream, not a TextWriter. It is much easier to compose streams than writers (like pass your stream to compressed stream).
To create custom stream - derive from Stream and implement at least Write and Flush (and Read if you need R/W stream). The rest is more or less optional and depends on you additional needs, regular copy to other stream does not need anything else.
In constructor get inner stream passed to you for writing to. Base64 is always producing ASCII characters, so it should be easy to write output as UTF-8 with or without BOM directly to a stream, but if you want to specify encoding you can wrap inner stream with StreamWriter internally.
In your Write implementation buffer data till you get enough bytes to have block of multiple of 3 bytes (i.e. 300) and call Convert.ToBase64String on that portion. Make sure not to loose not-yet-converted portion. Since Base64 converts 3 bytes to 4 characters converting in blocks of multiple of 3 size will never have =/== padding at the end and can be concatenated with next block. So write that converted portion into inner stream/writer. Note that you want to limit block size to something relatively small like 3*10000 to avoid allocation of your blocks on large objects heap.
In Flush make sure to convert the last unwritten bytes (this will be the only one with = padding at the end) and write it to the stream too.
For reading you may need to be more careful as in Base64 white spaces are allowed, so you can't read fixed number of characters and convert to bytes. The easiest approach would be to read by character from StreamReader and convert each 4 non-space ones to bytes.
Note: you can consider writing/reading Base64 by hand directly from bytes. It will give you some performance benefits, but may be hard if you are not good with bit shifting.
Please try using following to encrypt. I am using fileName/filePath as input. You can adjust it as per your requirement. Using this I have encrypted over 1 gb file successfully without any out of memory exception.
public bool EncryptUsingStream(string inputFileName, string outputFileName)
{
bool success = false;
// here assuming that you already have key
byte[] key = new byte[128];
SymmetricAlgorithm algorithm = SymmetricAlgorithm.Create();
algorithm.Key = key;
using (ICryptoTransform transform = algorithm.CreateEncryptor())
{
CryptoStream cs = null;
FileStream fsEncrypted = null;
try
{
using (FileStream fsInput = new FileStream(inputFileName, FileMode.Open, FileAccess.Read))
{
//First write IV
fsEncrypted = new FileStream(outputFileName, FileMode.Create, FileAccess.Write);
fsEncrypted.Write(algorithm.IV, 0, algorithm.IV.Length);
//then write using stream
cs = new CryptoStream(fsEncrypted, transform, CryptoStreamMode.Write);
int bytesRead;
int _bufferSize = 1048576; //buggersize = 1mb;
byte[] buffer = new byte[_bufferSize];
do
{
bytesRead = fsInput.Read(buffer, 0, _bufferSize);
cs.Write(buffer, 0, bytesRead);
} while (bytesRead > 0);
success = true;
}
}
catch (Exception ex)
{
//handle exception or throw.
}
finally
{
if (cs != null)
{
cs.Close();
((IDisposable)cs).Dispose();
if (fsEncrypted != null)
{
fsEncrypted.Close();
}
}
}
}
return success;
}