I'm attempting to read an email attachment and I'm getting a "Memory Stream is not expandable" error. I researched this some and most of the solutions seemed related to determining the size of the buffer dynamically, but I'm already doing that. I'm not very experienced with memory streams, so I'd like to know WHY this is a problem. Thanks.
foreach (MailMessage m in messages)
{
byte[] myBuffer = null;
if (m.Attachments.Count > 0)
{
//myBuffer = new byte[25 * 1024]; old way
myBuffer = new byte[m.Attachments[0].ContentStream.Length];
int read;
while ((read = m.Attachments[0].ContentStream.Read(myBuffer, 0, myBuffer.Length)) > 0)
{
// error occurs on executing next statement
m.Attachments[0].ContentStream.Write(myBuffer, 0, read);
}
... more unrelated code ...
If you create a MemoryStream over a pre-allocated byte array, it can't expand (ie. get longer than the size you specified when you started). Instead, why not just use:
using (var ms = new MemoryStream())
{
// Do your thing, for example:
m.Attachments[0].ContentStream.CopyTo(ms);
return ms.ToArray(); // This gives you the byte array you want.
}
You need to replace the line
m.Attachments[0].ContentStream.Write(myBuffer, 0, read);
with a line that writes to a previously created MemoryStream, e.g.
foreach (MailMessage m in messages)
{
byte[] myBuffer = null;
if (m.Attachments.Count > 0)
{
//myBuffer = new byte[25 * 1024]; old way
myBuffer = new byte[m.Attachments[0].ContentStream.Length];
int read;
MemoryStream ms = new MemoryStream();
while ((read = m.Attachments[0].ContentStream.Read(myBuffer, 0, myBuffer.Length)) > 0)
{
ms.Write(myBuffer, 0, read);
}
Related
I'm trying to capture an audio stream in CScore and save it in various encodings and to various locations. One of my intended output encodings is MP3 via the MediaFoundationEncoder APIs.
I am able to successfully encode to MP3 when saving to a local file path. However, if I try to write to a memory stream the memory stream completes writing with a 0 length.
What is wrong with this implementation?
Working Local Storage
var fileName = "c:\audio.mp3";
using (var encoder = MediaFoundationEncoder.CreateMP3Encoder(waveFormat, fileName, waveFormat.BytesPerSecond))
{
byte[] buffer = new byte[waveFormat.BytesPerSecond];
int read;
while ((read = inputStream.Read(buffer, 0, buffer.Length)) > 0)
{
encoder.Write(buffer, 0, read);
}
}
Non-working Memory Stream
using (var outputStream = new MemoryStream())
{
var encoder = MediaFoundationEncoder.CreateMP3Encoder(waveFormat, outputStream, waveFormat.BytesPerSecond);
var buffer = new byte[waveFormat.BytesPerSecond];
int read;
while ((read = inputStream.Read(buffer, 0, buffer.Length)) > 0)
{
encoder.Write(buffer, 0, read);
}
log.Debug("MP3 File Size: " + outputStream.Length); // <-- Returns as 0
}
You have to dispose the encoder after write, this completes the output stream.
public static byte[] EncodeBytes(byte[] bytes, WaveFormat waveFormat)
{
var outputStream = new MemoryStream();
var encoder = MediaFoundationEncoder.CreateMP3Encoder(waveFormat, outputStream, waveFormat.BytesPerSecond);
var buffer = new byte[waveFormat.BytesPerSecond];
var inputStream = new MemoryStream(bytes);
int read;
while ((read = inputStream.Read(buffer, 0, buffer.Length)) > 0)
{
encoder.Write(buffer, 0, read);
}
encoder.Dispose();
return outputStream.ToArray();
}
I'm working with GZipStream at the moment using .net 3.5.
I have two methods listed below. As input file I use text file which consists of chars 's'. Size of the file is 2MB. This code works fine if I use .net 4.5 but with .net 3.5 after compress and decompress I get file of size 435KB which of course isn't the same with source file.
If I try to decompress file via WinRAR it is also looks good (the same with source file).
If I try decompress file using GZipStream from .net4.5 (file compressed via GZipStream from .net 3.5) the result is bad.
UPD:
In general I really need to read the file as several separate gzip chunks, in this case all the bytes of copressed files are read at one call of the Read() method so I still don't understand why decompressing doesn't works.
public void CompressFile()
{
string fileIn = #"D:\sin2.txt";
string fileOut = #"D:\sin2.txt.pgz";
using (var fout = File.Create(fileOut))
{
using (var fin = File.OpenRead(fileIn))
{
using (var zip = new GZipStream(fout, CompressionMode.Compress))
{
var buffer = new byte[1024 * 1024 * 10];
int n = fin.Read(buffer, 0, buffer.Length);
zip.Write(buffer, 0, n);
}
}
}
}
public void DecompressFile()
{
string fileIn = #"D:\sin2.txt.pgz";
string fileOut = #"D:\sin2.1.txt";
using (var fsout = File.Create(fileOut))
{
using (var fsIn = File.OpenRead(fileIn))
{
var buffer = new byte[1024 * 1024 * 10];
int n;
while ((n = fsIn.Read(buffer, 0, buffer.Length)) > 0)
{
using (var ms = new MemoryStream(buffer, 0, n))
{
using (var zip = new GZipStream(ms, CompressionMode.Decompress))
{
int nRead = zip.Read(buffer, 0, buffer.Length);
fsout.Write(buffer, 0, nRead);
}
}
}
}
}
}
You're trying to decompress each "chunk" as if it's a separate gzip file. Don't do that - just read from the GZipStream in a loop:
using (var fsout = File.Create(fileOut))
{
using (var fsIn = File.OpenRead(fileIn))
{
using (var zip = new GZipStream(fsIn, CompressionMode.Decompress))
{
var buffer = new byte[1024 * 32];
int bytesRead;
while ((bytesRead = zip.Read(buffer, 0, buffer.Length)) > 0)
{
fsout.Write(buffer, 0, bytesRead);
}
}
}
}
Note that your compression code should look similar, reading in a loop rather than assuming a single call to Read will read all the data.
(Personally I'd skip fsIn, and just use new GZipStream(File.OpenRead(fileIn)) but that's just a personal preference.)
First, as #Jon Skeet mentioned, you are not using Stream.Read method correctly. It doesn't matter if your buffer is big enough or not, the stream is allowed to return less bytes than requested, with zero indicating no more, so reading from stream should always be performed in a loop.
However the main problem in your decompress code is the way you share the buffer. Your read the input into a buffer, than wrap it in a MemoryStream (note that the constructor used does not make a copy of the passed array, but actually sets it as it's internal buffer), and then you try to read and write to that buffer at the same time. Taking into account that decompressing writes data "faster" than reading, it's surprising that your code works at all.
The correct implementation is quite simple
static void CompressFile()
{
string fileIn = #"D:\sin2.txt";
string fileOut = #"D:\sin2.txt.pgz";
using (var input = File.OpenRead(fileIn))
using (var output = new GZipStream(File.Create(fileOut), CompressionMode.Compress))
Write(input, output);
}
static void DecompressFile()
{
string fileIn = #"D:\sin2.txt.pgz";
string fileOut = #"D:\sin2.1.txt";
using (var input = new GZipStream(File.OpenRead(fileIn), CompressionMode.Decompress))
using (var output = File.Create(fileOut))
Write(input, output);
}
static void Write(Stream input, Stream output, int bufferSize = 10 * 1024 * 1024)
{
var buffer = new byte[bufferSize];
for (int readCount; (readCount = input.Read(buffer, 0, buffer.Length)) > 0;)
output.Write(buffer, 0, readCount);
}
I'm having issues with an "out of memory" exception that I can't reproduce, but a build server that runs unit tests hits every time. Running the unit tests on my machine doesn't cause the exception. The changes made were because the original code was having an odd issue with large PDF's in the passed in stream. If you have an idea of why the original code had issues with the large PDF's or why the new code would cause an "out of memory" exception then let me know.
Original Code:
// stream is a valid Stream and parentKey is a valid int
// Reset the stream position
stream.Position = 0;
int sequenceNumber = 0;
int StreamReadSize = short.MaxValue;
byte[] buffer = new byte[StreamReadSize];
MemoryStream outStream = null;
try
{
long previousStreamPosition = 0;
long DataBlockSize = 52428800;
int read;
while ((read = stream.Read(buffer, 0, buffer.Length)) > 0)
{
if (outStream == null)
outStream = new MemoryStream(new byte[System.Math.Min(stream.Length - previousStreamPosition, DataBlockSize)]);
previousStreamPosition = stream.Position;
outStream.Write(buffer, 0, read);
if (outStream.Position <= (DataBlockSize - StreamReadSize) && stream.Position < stream.Length)
continue;
var dataRow = dataSet.Tables["table_name"].NewRow();
dataRow["parent_key"] = parentKey;
dataRow["key"] = FuncThatReturnsNextAvailableKey();
dataRow["sequence_number"] = ++sequenceNumber;
// Reset the position and Zip up the data
outStream.Position = 0;
dataRow["data_segment"] = FuncThatZipsAStreamToByteArray(outStream);
dataSet.Tables["table_name"].Rows.Add(dataRow);
outStream.Flush();
outStream.Dispose();
outStream = null;
}
}
finally
{
if (outStream != null)
outStream.Dispose();
}
New Code:
// stream is a valid Stream and parentKey is a valid int
// Reset the stream position and create the variables needed for saving the file data
stream.Position = 0;
int sequenceNumber = 0;
int bytesRead;
int DataBlockSize = 52428800;
byte[] buffer = new byte[DataBlockSize];
while ((bytesRead = stream.Read(buffer, 0, DataBlockSize)) > 0)
{
sequenceNumber++;
// Create and initialize the row
var dataRow = dataSet.Tables["table_name"].NewRow();
dataRow["parent_key"] = parentKey;
dataRow["key"] = FuncThatReturnsNextAvailableKey(); ;
dataRow["sequence_number"] = sequenceNumber;
// If the stream reads in less data than the size of the buffer then create an appropriately sized version of the buffer
// that will only hold the data that was read in
if (bytesRead != DataBlockSize)
{
var shrunkBuffer = new byte[bytesRead];
Array.Copy(buffer, shrunkBuffer, bytesRead);
using (var memoryStream = new MemoryStream(shrunkBuffer))
dataRow["data_segment"] = FuncThatZipsAStreamToByteArray(memoryStream);
}
else
{
using (var memoryStream = new MemoryStream(buffer))
dataRow["data_segment"] = FuncThatZipsAStreamToByteArray(memoryStream);
}
// Add the finished row
dataSet.Tables["table_name"].Rows.Add(dataRow);
}
It makes sense that two different environments might generate a different result. it could be that your build server has less memory than your personal coding environment.
It could be that you are keeping your byte arrays in memory via:
dataRow["data_segment"] = FuncThatZipsAStreamToByteArray(memoryStream);
Your are disposing the output stream, but i am assuming your data row stays in memory, hence your are keeping a reference to that byte array. it could be so that multiple PDFs reach the maximum amount of allocation your process can allocate for itself.
Use a class memorytributary
From sources at https://gist.github.com/bittercoder/3588074
using (System.IO.FileStream stream = new System.IO.FileStream(fileName, System.IO.FileMode.Open, System.IO.FileAccess.Read))
{
using (MemoryTributary memT = new MemoryTributary())
{
memT.ReadFrom(stream, stream.Length);
return memT.ToArray();
}
}
I recently wanted to track progress of a HTTPWebRequest Upload progress. So I started small and started with buffered read of a simple text file. I then discovered that a simple task like
File.ReadAllText("text.txt");
becomes something like below, with all the streams, readers, writers etc. Or can somethings be removed? Also the code below is not working. Maybe I did something wrong, whats the way to read (i guess write will be similar) into buffer so that I can track progress, assuming the stream are not local eg. WebRequest
byte[] buffer = new byte[2560]; // 20KB Buffer, btw, how should I decide the buffer size?
int bytesRead = 0, read = 0;
FileStream inStream = new FileStream("./text.txt", FileMode.Open, FileAccess.Read);
MemoryStream outStream = new MemoryStream();
BinaryWriter outWriter = new BinaryWriter(outStream);
// I am getting "Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection."
// inStream.Length = Length = 9335092
// bytesRead = 2560
// buffer.Length = 2560
while ((read = inStream.Read(buffer, bytesRead, buffer.Length)) > 0)
{
outWriter.Write(buffer);
//outStream.Write(buffer, bytesRead, buffer.Length);
bytesRead += read;
Debug.WriteLine("Progress: " + bytesRead / inStream.Length * 100 + "%");
}
outWriter.Flush();
txtLog.Text = outStream.ToString();
Update: Solution
byte[] buffer = new byte[2560];
int bytesRead = 0, read = 0;
FileStream inStream = File.OpenRead("text.txt");
MemoryStream outStream = new MemoryStream();
while ((read = inStream.Read(buffer, 0, buffer.Length)) > 0)
{
outStream.Write(buffer, 0, buffer.Length);
bytesRead += read;
Debug.WriteLine((double)bytesRead / inStream.Length * 100);
}
inStream.Close();
outStream.Close();
should probably be
outWriter.Write(buffer,0,read);
Since you seem to be reading text (although I could be wrong), it seems that your program could be a lot simpler if you read character by character instead of calling the standard Read():
BinaryReader reader = new BinaryReader(File.Open("./text.txt", FileMode.Open));
MemoryStream outStream = new MemoryStream();
StreamWriter outWriter = new StreamWriter(outStream);
while (Reader.BaseStream.Position < Reader.BaseStream.Length)
{
outWriter.Write(reader.ReadChar());
Debug.WriteLine("Progress: " + ((double)reader.BaseStream.Position) / (double)(reader.BaseStream.Length) + "%");
}
outWriter.Close();
txtLog.Text = outStream.ToString();
Since you only need to check the progress of the upload operation you can just check the size of the file using a fileinfo object.
In the FileInfo class theres a property called length that returns the file size in bytes. Not sure if it gives the current size when the file being written. But I think it'll be worth giving a try as it is more simple and efficient than the method that you are using
I am trying to check if a file is an image before I upload it to the image server.
I am doing it with the following function, which works exceptionally well:
static bool IsValidImage(Stream imageStream)
{
bool isValid = false;
try
{
// Read the image without validating image data
using (Image img = Image.FromStream(imageStream, false, false))
{
isValid = true;
}
}
catch
{
;
}
return isValid;
}
The problem is that when the below is called immediately afterwards, The line:
while ((bytesRead = request.FileByteStream.Read(buffer, 0, bufferSize)) > 0)
evalueates to zero and no bytes are read. I notice that when I remove the
IsValidImage function, bytes are read and the file is written. It seems
that bytes can only be read once? Any idea how to fix this?
using (FileStream outfile = new FileStream(filePath, FileMode.Create))
{
const int bufferSize = 65536; // 64K
int bytesRead = 0;
Byte[] buffer = new Byte[bufferSize];
while ((bytesRead = request.FileByteStream.Read(buffer, 0, bufferSize)) > 0)
{
outfile.Write(buffer, 0, bytesRead);
}
outfile.Close(); //necessary?
}
UPDATE: Thanks for your help Marc. I am new to stream manipulation and could use a little
more help here. I took a shot but may be mixing up the use of filestream and memorystream.
Would you mind taking a look? Thanks again.
using (FileStream outfile = new FileStream(filePath, FileMode.Create))
using (MemoryStream ms = new MemoryStream())
{
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = request.FileByteStream.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, bytesRead);
}
// ms now has a seekable/rewindable copy of the data
// TODO: read ms the first time
// I replaced request.FileByteStream with ms but am unsure about
// the using statement in the IsValidImage function.
if (!IsValidImage(ms) == true)
{
ms.Close();
request.FileByteStream.Close();
return;
}
ms.Position = 0;
// TODO: read ms the second time
byte[] m_buffer = new byte[ms.Length];
while ((bytesRead = ms.Read(m_buffer, 0, (int)ms.Length)) > 0)
{
outfile.Write(m_buffer, 0, bytesRead);
}
}
static bool IsValidImage(MemoryStream imageStream)
{
bool isValid = false;
try
{
// Read the image without validating image data
using (Image img = Image.FromStream(imageStream, false, false))
{
isValid = true;
}
}
catch
{
;
}
return isValid;
}
As you read from any stream, the position increases. If you read a stream to the end (as is typical), and then try to read again, then it will return EOF.
For some streams, you can seek - set the Position to 0, for example. However, you should try to avoid relying on this as it is not available for many streams (especially when network IO is involved). You can query this ability via CanSeek, but it would be simpler to avoid this - partly as if you are branching based on this, you suddenly have twice as much code to maintain.
If you need the data twice, then the options depends on the size of the data. For small streams, buffer it in-memory, as either a byte[] or a MemoryStream. For larger streams (or if you don't know the size) then writing to a scratch file (and deleting afterwards) is a reasonable approach. You can open and read the file as many times (in series, not in parallel) as you like.
If you are happy the stream isn't too large (although maybe add a cap to prevent people uploading swap-files, etc):
using (MemoryStream ms = new MemoryStream()) {
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = inputStream.Read(buffer, 0, buffer.Length)) > 0) {
ms.Write(buffer, 0, bytesRead);
}
// ms now has a seekable/rewindable copy of the data
// TODO: read ms the first time
ms.Position = 0;
// TODO: read ms the second time
}
Indeed Stream instances remember where the current "cursor" is. Some streams support "rewinding". The "CanSeek" property will then return true. In the case of a HTTP request Ithis won't work (CanSeek = false).
Isn't a MIME-type sent from the browser as well?
If you really want to keep your way of checking you'll have to go with Marc's proposition
In your update, you have a problem reading the stream a second time.
byte[] m_buffer = new byte[ms.Length];
while ((bytesRead = ms.Read(m_buffer, 0, (int)ms.Length)) > 0)
{
outfile.Write(m_buffer, 0, bytesRead);
}
The solution is simple:
byte[] m_buffer = ms.ToArray();
outfile.Write(m_buffer, 0, m_buffer.Length);
See also MemoryStream.ToArray
public static bool IsImagen(System.IO.Stream stream, String fileName)
{
try
{
using (Image img = Image.FromStream(stream, false, false))
{
if (fileName.ToLower().IndexOf(".jpg") > 0)
return true;
if (fileName.ToLower().IndexOf(".gif") > 0)
return true;
if (fileName.ToLower().IndexOf(".png") > 0)
return true;
}
}
catch (ArgumentException){}
return false;
}