Develop Reference

C# - Having problems with processing a Stream more than once

I have a android mobile application that has functionality to set a profile picture.
I send a variable containing the path of the image to a method that does the following:
string imagePath = _ProfilePicture.GetTag (Resource.String.profile_picture_path).ToString ();
byte[] imageBytes = System.IO.File.ReadAllBytes(imagePath);
Stream imageStream = new MemoryStream(imageBytes);
After this block of code I send the imageStream variable to UploadUserProfilePicture(imageStream); which is located on the WCF Service
Currently it only sends the stream, but because we cannot send another parameter containing the extension. We save all images as png. I have however found a library that requires the stream to be parsed to bytes and then based on the bytes the file type can retrieved.
However when I then try to use the same stream to Write the file to the location on the server, the position is at the end so the file created is always 0 bytes.
I have tried:
Doing the conversion to Bytes in another method and only returning the fileType, however the originals position was still at the end.
The CopyTo function gave me the same results.
I tried using the Seek function and setting it's position back to zero however the I get a NotSupportedException.
I tried this as well:
string content;
var reader = new StreamReader(image);
content = reader.ReadToEnd();
image.Dispose();
image = new MemoryStream(Encoding.UTF8.GetBytes(content));
^ this seems to corrupt the stream as I cannot get the FileType nor write it to the above location.
I have also had a look at: How to read a Stream and reset its position to zero even if stream.CanSeek == false
This is the method on the WCF Service:
public Result UploadUserProfilePicture(Stream image)
{
try
{
FileType fileType = CommonMethods.ReadToEnd(image).GetFileType();
Guid guid = Guid.NewGuid();
string imageName = guid.ToString() + "." + fileType.Extension;
var buf = new byte[1024];
var path = Path.Combine(#"C:\" + imageName);
int len = 0;
using (var fs = File.Create(path))
{
while ((len = image.Read(buf, 0, buf.Length)) > 0)
{
fs.Write(buf, 0, len);
}
}
return new Result
{
Success = true,
Message = imageName
};
}
catch(Exception ex)
{
return new Result
{
Success = false,
Message = ex.ToString()
};
}
Link to Library Used: https://github.com/Muraad/Mime-Detective
The CommonMethods.ReadToEnd(image) method can be found here: How to convert an Stream into a byte[] in C#? as the questions answer
I hope this is enough information on my problem.

On the server side, you receive a stream from WCF that does not support seek operations. You can, however, read the stream to memory as the GetFileType method requires an array of bytes as input parameter. Instead of accessing the original stream again, you can write the bytes of the array to disk in a very easy way using the File.WriteAllBytes method:
public Result UploadUserProfilePicture(Stream image)
{
try
{
// Store bytes in a variable
var bytes = CommonMethods.ReadToEnd(image);
FileType fileType = bytes.GetFileType();
Guid guid = Guid.NewGuid();
string imageName = guid.ToString() + "." + fileType.Extension;
var path = Path.Combine(#"C:\" + imageName);
File.WriteAllBytes(path, bytes);
return new Result
{
Success = true,
Message = imageName
};
}
catch(Exception ex)
{
return new Result
{
Success = false,
Message = ex.ToString()
};
}
}
Please note that this means that you store a possibly large amount of bytes in memory, in the same way you already did before. It would be better if you could use the stream without reading all bytes into memory, so looking for an alternative for the GetFileType method that can handle a stream is well worth the time. You could then first save the image to a temporary file and then open a new FileStream to discover the correct file type so that you can rename the file.

C# Out of Memory Exception Store File as Byte Array in DataSet

I'm having issues with an "out of memory" exception that I can't reproduce, but a build server that runs unit tests hits every time. Running the unit tests on my machine doesn't cause the exception. The changes made were because the original code was having an odd issue with large PDF's in the passed in stream. If you have an idea of why the original code had issues with the large PDF's or why the new code would cause an "out of memory" exception then let me know.
Original Code:
// stream is a valid Stream and parentKey is a valid int
// Reset the stream position
stream.Position = 0;
int sequenceNumber = 0;
int StreamReadSize = short.MaxValue;
byte[] buffer = new byte[StreamReadSize];
MemoryStream outStream = null;
try
{
long previousStreamPosition = 0;
long DataBlockSize = 52428800;
int read;
while ((read = stream.Read(buffer, 0, buffer.Length)) > 0)
{
if (outStream == null)
outStream = new MemoryStream(new byte[System.Math.Min(stream.Length - previousStreamPosition, DataBlockSize)]);
previousStreamPosition = stream.Position;
outStream.Write(buffer, 0, read);
if (outStream.Position <= (DataBlockSize - StreamReadSize) && stream.Position < stream.Length)
continue;
var dataRow = dataSet.Tables["table_name"].NewRow();
dataRow["parent_key"] = parentKey;
dataRow["key"] = FuncThatReturnsNextAvailableKey();
dataRow["sequence_number"] = ++sequenceNumber;
// Reset the position and Zip up the data
outStream.Position = 0;
dataRow["data_segment"] = FuncThatZipsAStreamToByteArray(outStream);
dataSet.Tables["table_name"].Rows.Add(dataRow);
outStream.Flush();
outStream.Dispose();
outStream = null;
}
}
finally
{
if (outStream != null)
outStream.Dispose();
}
New Code:
// stream is a valid Stream and parentKey is a valid int
// Reset the stream position and create the variables needed for saving the file data
stream.Position = 0;
int sequenceNumber = 0;
int bytesRead;
int DataBlockSize = 52428800;
byte[] buffer = new byte[DataBlockSize];
while ((bytesRead = stream.Read(buffer, 0, DataBlockSize)) > 0)
{
sequenceNumber++;
// Create and initialize the row
var dataRow = dataSet.Tables["table_name"].NewRow();
dataRow["parent_key"] = parentKey;
dataRow["key"] = FuncThatReturnsNextAvailableKey(); ;
dataRow["sequence_number"] = sequenceNumber;
// If the stream reads in less data than the size of the buffer then create an appropriately sized version of the buffer
// that will only hold the data that was read in
if (bytesRead != DataBlockSize)
{
var shrunkBuffer = new byte[bytesRead];
Array.Copy(buffer, shrunkBuffer, bytesRead);
using (var memoryStream = new MemoryStream(shrunkBuffer))
dataRow["data_segment"] = FuncThatZipsAStreamToByteArray(memoryStream);
}
else
{
using (var memoryStream = new MemoryStream(buffer))
dataRow["data_segment"] = FuncThatZipsAStreamToByteArray(memoryStream);
}
// Add the finished row
dataSet.Tables["table_name"].Rows.Add(dataRow);
}

It makes sense that two different environments might generate a different result. it could be that your build server has less memory than your personal coding environment.
It could be that you are keeping your byte arrays in memory via:
dataRow["data_segment"] = FuncThatZipsAStreamToByteArray(memoryStream);
Your are disposing the output stream, but i am assuming your data row stays in memory, hence your are keeping a reference to that byte array. it could be so that multiple PDFs reach the maximum amount of allocation your process can allocate for itself.

Use a class memorytributary
From sources at https://gist.github.com/bittercoder/3588074
using (System.IO.FileStream stream = new System.IO.FileStream(fileName, System.IO.FileMode.Open, System.IO.FileAccess.Read))
{
using (MemoryTributary memT = new MemoryTributary())
{
memT.ReadFrom(stream, stream.Length);
return memT.ToArray();
}
}

RavenDB Attachments - Functionality how to do?

I have a file input control.
<input type="file" name="file" id="SaveFileToDB"/>
Lets say I browse to C:/Instruction.pdf document and click on submit. On Submit, I want to save the document in RavenDB and also later retrieve it for download purposes. I saw this link http://ravendb.net/docs/client-api/attachments
that says.. do this..
Stream data = new MemoryStream(new byte[] { 1, 2, 3 });
documentStore.DatabaseCommands.PutAttachment("videos/2", null, data,
new RavenJObject {{"Description", "Kids play in the garden"}});
I am not following what 1,2,3 mean here and what it means to say videos/2 in the command... how I can use these two lines to use it in my case.. to save word/pdfs in ravendb.. if any one has done such thing before, please advise.
I am not clear on one thing.. how the attachment is stored. If I want to store the attachment itself (say pdf) it is stored independently in ravendb.. and I just store the key of the attachment in the main document that it is associated with? If that is so, where is the pdf stored physically in ravendb? can I see it?

The 1,2,3 is just example data. What it is trying to get across is that you create a memory stream of whatever you want then use that memory stream in the PutAttachment method. Below is ad-hoc and not tested but should work:
using (var mem = new MemoryStream(file.InputStream)
{
_documentStore.DatabaseCommands.PutAttachment("upload/" + YourUID, null, mem,
new RavenJObject
{
{ "OtherData", "Can Go here" },
{ "MoreData", "Here" }
});
}
Edited for the rest of the questions
How is attachment stored? I believe it is a json document with one property holding the byte array of the attachment
Is the "document" stored independently? Yes. An attachment is a special document that is not indexed but it is part of the database so that tasks like replication work.
"Should I" store the key of the attachment in the main document that it is associated with? Yes you would reference the Key and anytime you want to get that you would just ask Raven for the attachment with that id.
Is the pdf stored physically in ravendb? Yes.
Can you see it? No. It does even show up in the studio (at least as far as I know)
Edit Corrected and Updated Sample
[AcceptVerbs(HttpVerbs.Post)]
public ActionResult Upload(HttpPostedFileBase file)
{
byte[] bytes = ReadToEnd(file.InputStream);
var id = "upload/" + DateTime.Now.Second.ToString(CultureInfo.InvariantCulture);
using (var mem = new MemoryStream(bytes))
{
DocumentStore.DatabaseCommands.PutAttachment(id, null, mem,
new RavenJObject
{
{"OtherData", "Can Go here"},
{"MoreData", "Here"},
{"ContentType", file.ContentType}
});
}
return Content(id);
}
public FileContentResult GetFile(string id)
{
var attachment = DocumentStore.DatabaseCommands.GetAttachment("upload/" + id);
return new FileContentResult(ReadFully(attachment.Data()), attachment.Metadata["ContentType"].ToString());
}
public static byte[] ReadToEnd(Stream stream)
{
long originalPosition = 0;
if (stream.CanSeek)
{
originalPosition = stream.Position;
stream.Position = 0;
}
try
{
var readBuffer = new byte[4096];
int totalBytesRead = 0;
int bytesRead;
while ((bytesRead = stream.Read(readBuffer, totalBytesRead, readBuffer.Length - totalBytesRead)) > 0)
{
totalBytesRead += bytesRead;
if (totalBytesRead == readBuffer.Length)
{
int nextByte = stream.ReadByte();
if (nextByte != -1)
{
var temp = new byte[readBuffer.Length*2];
Buffer.BlockCopy(readBuffer, 0, temp, 0, readBuffer.Length);
Buffer.SetByte(temp, totalBytesRead, (byte) nextByte);
readBuffer = temp;
totalBytesRead++;
}
}
}
byte[] buffer = readBuffer;
if (readBuffer.Length != totalBytesRead)
{
buffer = new byte[totalBytesRead];
Buffer.BlockCopy(readBuffer, 0, buffer, 0, totalBytesRead);
}
return buffer;
}
finally
{
if (stream.CanSeek)
{
stream.Position = originalPosition;
}
}
}
public static byte[] ReadFully(Stream input)
{
byte[] buffer = new byte[16 * 1024];
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
return ms.ToArray();
}
}

How is attachment stored?
It is stored as binary data inside RavenDB. It is NOT stored as json.
Is the "document" stored independently?
There isn't a document here, you have some metadata that is associated with the attachment, it isn't a seaprate document.
"Should I" store the key of the attachment in the main document that it is associated with?
Yes, there is no way to query for that.
Is the pdf stored physically in ravendb?
Yes
Can you see it?
Only if you go to the attachment directly, such as http://localhost:8080/static/ATTACHMENT_KEY
It won't show in the UI

SharpZipLib Deflater creates bad data

Original compressed data can be correctly inflated back. However, if I inflate data, deflate, and again inflate, resulting data are incorrect (e.g. simple data extraction, its modification and again compression - only now when testing no modification occurs, so I can test it).
Resulting data are somehow "damaged". The starting (about) 40 bytes are OK, and then "block" of incorrect data follows (remnants of original data are still there, but many bytes are missing).
Changing compression level doesn't help (except setting NO_COMPRESSION creates somehow incomplete stream).
Question is simple: why is that happening?
using ICSharpCode.SharpZipLib.Zip.Compression;
public byte[] Inflate(byte[] inputData)
{
Inflater inflater = new Inflater(false);
using (var inputStream = new MemoryStream(inputData))
using (var ms = new MemoryStream())
{
var inputBuffer = new byte[4096];
var outputBuffer = new byte[4096];
while (inputStream.Position < inputData.Length)
{
var read = inputStream.Read(inputBuffer, 0, inputBuffer.Length);
inflater.SetInput(inputBuffer, 0, read);
while (inflater.IsNeedingInput == false)
{
var written = inflater.Inflate(outputBuffer, 0, outputBuffer.Length);
if (written == 0)
break;
ms.Write(outputBuffer, 0, written);
}
if (inflater.IsFinished == true)
break;
}
inflater.Reset();
return ms.ToArray();
}
}
public byte[] Deflate(byte[] inputData)
{
Deflater deflater = new Deflater(Deflater.BEST_SPEED, false);
deflater.SetInput(inputData);
deflater.Finish();
using (var ms = new MemoryStream())
{
var outputBuffer = new byte[65536 * 4];
while (deflater.IsNeedingInput == false)
{
var read = deflater.Deflate(outputBuffer);
ms.Write(outputBuffer, 0, read);
if (deflater.IsFinished == true)
break;
}
deflater.Reset();
return ms.ToArray();
}
}
Edit: My bad, by mistake I rewrote first several bytes of the original compressed data. This isn't SharpZipLib fault, but mine.

I know this is a tangential answer, but the exact same thing happened to me, I abandoned SharpZipLib and went to DotNetZip :
http://dotnetzip.codeplex.com/
Easier API, no corrupt or strange byte order files.

If I check stream for valid image I can't write bytes to server

I am trying to check if a file is an image before I upload it to the image server.
I am doing it with the following function, which works exceptionally well:
static bool IsValidImage(Stream imageStream)
{
bool isValid = false;
try
{
// Read the image without validating image data
using (Image img = Image.FromStream(imageStream, false, false))
{
isValid = true;
}
}
catch
{
;
}
return isValid;
}
The problem is that when the below is called immediately afterwards, The line:
while ((bytesRead = request.FileByteStream.Read(buffer, 0, bufferSize)) > 0)
evalueates to zero and no bytes are read. I notice that when I remove the
IsValidImage function, bytes are read and the file is written. It seems
that bytes can only be read once? Any idea how to fix this?
using (FileStream outfile = new FileStream(filePath, FileMode.Create))
{
const int bufferSize = 65536; // 64K
int bytesRead = 0;
Byte[] buffer = new Byte[bufferSize];
while ((bytesRead = request.FileByteStream.Read(buffer, 0, bufferSize)) > 0)
{
outfile.Write(buffer, 0, bytesRead);
}
outfile.Close(); //necessary?
}
UPDATE: Thanks for your help Marc. I am new to stream manipulation and could use a little
more help here. I took a shot but may be mixing up the use of filestream and memorystream.
Would you mind taking a look? Thanks again.
using (FileStream outfile = new FileStream(filePath, FileMode.Create))
using (MemoryStream ms = new MemoryStream())
{
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = request.FileByteStream.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, bytesRead);
}
// ms now has a seekable/rewindable copy of the data
// TODO: read ms the first time
// I replaced request.FileByteStream with ms but am unsure about
// the using statement in the IsValidImage function.
if (!IsValidImage(ms) == true)
{
ms.Close();
request.FileByteStream.Close();
return;
}
ms.Position = 0;
// TODO: read ms the second time
byte[] m_buffer = new byte[ms.Length];
while ((bytesRead = ms.Read(m_buffer, 0, (int)ms.Length)) > 0)
{
outfile.Write(m_buffer, 0, bytesRead);
}
}
static bool IsValidImage(MemoryStream imageStream)
{
bool isValid = false;
try
{
// Read the image without validating image data
using (Image img = Image.FromStream(imageStream, false, false))
{
isValid = true;
}
}
catch
{
;
}
return isValid;
}

As you read from any stream, the position increases. If you read a stream to the end (as is typical), and then try to read again, then it will return EOF.
For some streams, you can seek - set the Position to 0, for example. However, you should try to avoid relying on this as it is not available for many streams (especially when network IO is involved). You can query this ability via CanSeek, but it would be simpler to avoid this - partly as if you are branching based on this, you suddenly have twice as much code to maintain.
If you need the data twice, then the options depends on the size of the data. For small streams, buffer it in-memory, as either a byte[] or a MemoryStream. For larger streams (or if you don't know the size) then writing to a scratch file (and deleting afterwards) is a reasonable approach. You can open and read the file as many times (in series, not in parallel) as you like.
If you are happy the stream isn't too large (although maybe add a cap to prevent people uploading swap-files, etc):
using (MemoryStream ms = new MemoryStream()) {
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = inputStream.Read(buffer, 0, buffer.Length)) > 0) {
ms.Write(buffer, 0, bytesRead);
}
// ms now has a seekable/rewindable copy of the data
// TODO: read ms the first time
ms.Position = 0;
// TODO: read ms the second time
}

Indeed Stream instances remember where the current "cursor" is. Some streams support "rewinding". The "CanSeek" property will then return true. In the case of a HTTP request Ithis won't work (CanSeek = false).
Isn't a MIME-type sent from the browser as well?
If you really want to keep your way of checking you'll have to go with Marc's proposition

In your update, you have a problem reading the stream a second time.
byte[] m_buffer = new byte[ms.Length];
while ((bytesRead = ms.Read(m_buffer, 0, (int)ms.Length)) > 0)
{
outfile.Write(m_buffer, 0, bytesRead);
}
The solution is simple:
byte[] m_buffer = ms.ToArray();
outfile.Write(m_buffer, 0, m_buffer.Length);
See also MemoryStream.ToArray

public static bool IsImagen(System.IO.Stream stream, String fileName)
{
try
{
using (Image img = Image.FromStream(stream, false, false))
{
if (fileName.ToLower().IndexOf(".jpg") > 0)
return true;
if (fileName.ToLower().IndexOf(".gif") > 0)
return true;
if (fileName.ToLower().IndexOf(".png") > 0)
return true;
}
}
catch (ArgumentException){}
return false;
}