I am trying to send large objects (>30MB) to a MSMQ queue. Due to the large amount of data we are are tring to send the idea was to GZip the objects prior to sending them, then unzipping them on the receiving end.
However, writing the compressed stream to the message.BodyStream property seems to work, but not reading it out from there.
I don't know what's wrong.
Message l_QueueMessage = new Message();
l_QueueMessage.Priority = priority;
using (MessageQueue l_Queue = CreateQueue())
{
GZipStream stream = new GZipStream(l_QueueMessage.BodyStream, CompressionMode.Compress);
Formatter.Serialize(stream, message);
l_Queue.Send(l_QueueMessage);
}
The Formatter is a global property of type BinaryFormatter. This is used to serialize/deserialize to the type of object we want to send/receive, e.g. "ProductItem".
The receving end looks like this:
GZipStream stream = new GZipStream(l_Message.BodyStream, CompressionMode.Decompress);
object decompressedObject = Formatter.Deserialize(stream);
ProductItem l_Item = decompressedObject as ProductItem;
m_ProductReceived(sender, new MessageReceivedEventArgs<ProductItem>(l_Item));
l_ProductQueue.BeginReceive();
I get an EndOfStreamException "{"Unable to read beyond the end of the stream."} trying to deserialize
at System.IO.BinaryReader.ReadByte()
Using the messageBodyStream property I actually circumvent the message.Formatter, which I don't initialize to anything, becasue I'm using my own ser/deser mechanism with the GZipStream. However, I am not sure if that's the correct way of doing this.
What am I missing?
Thanks!
In your original code, the problem is that you need to close the GZipStream in order for a GZip footer to be written correctly, and only then you can send it. If you dont, you end up sending bytes that can not be deserialized. That's also why your new code where sending is done later works.
OK, I made this work. The key was to convert the decompressed stream on the receiver to a byte[] array. Then the deserialization started working.
The sender code (notice the stream is closed before sending the message):
using (MessageQueue l_Queue = CreateQueue())
{
using (GZipStream stream = new GZipStream(l_QueueMessage.BodyStream, CompressionMode.Compress, true))
{
Formatter.Serialize(stream, message);
}
l_Queue.Send(l_QueueMessage);
}
The receiving end (notice how I convert the stream to a byte[] then deserialize):
using (GZipStream stream = new GZipStream(l_QueueMessage.BodyStream, CompressionMode.Decompress))
{
byte[] bytes = ReadFully(stream);
using (MemoryStream ms = new MemoryStream(bytes))
{
decompressedObject = Formatter.Deserialize(ms);
}
}
Still, don't know why this works using the ReadFully() function and not the Stream.CopyTo().
Does anyone?
Btw, ReadFully() is a function that creates a byte[] out of a Stream. I have to credit Jon Skeet for this at http://www.yoda.arachsys.com/csharp/readbinary.html. Thanks!
Try to separate compressing and sending:
byte[] binaryBuffer = null;
using (MemoryStream compressedBody = new MemoryStream())
{
using(GZipStream stream = new GZipStream(compressedBody, CompressionMode.Compress))
{
Formatter.Serialize(compressedBody, message);
binaryBuffer = compressedBody.GetBuffer();
}
}
using (MessageQueue l_Queue = CreateQueue())
{
l_QueueMessage.BodyStream.Write(binaryBuffer, 0, binaryBuffer.Length);
l_QueueMessage.BodyStream.Seek(0, SeekOrigin.Begin);
l_Queue.Send(l_QueueMessage);
}
Related
On my service I am trying to create and return a MemoryStream like this:
// Write data.
MemoryStream memoryStream = new MemoryStream();
using (StreamWriter txtwriter = new StreamWriter(memoryStream, System.Text.Encoding.UTF8))
{
string tempString = "Test";
txtwriter.Write(tempString);
txtwriter.WriteLine();
// End response.
memoryStream.Position = 0L;
response.ReportMemoryStream = memoryStream;
response.Length = memoryStream.Length;
return response;
}
Note: having the End response part outside the using statement causes a Cannot access a closed stream error.
My response is a simple DataContract. When I try to read this data in my client doing something like this:
// Write stream.
if (remoteReport.ReportMemoryStream != null)
{
remoteReport.ReportMemoryStream.WriteTo(Response.OutputStream);
}
I get again the same error about the stream being closed.
How can I fix this issue and why is my stream closing even though I'm not explicitly doing so?
replace:
response.ReportMemoryStream = memoryStream;
with
memoryStream.CopyTo(response.ReportMemoryStream);
now you are using reference type I think, and your using statment is just disposing stream you are using.
You'll need to flush your StreamWriter instance, so it writes to the MemoryStream instance -
txtwriter.Flush();
Then you'll need to remove the using block, so the txtwriter doesn't by default destroyes the memoryStream.
The SendGrid API docs specify you can add attachments from a Stream. The example it gives uses a FileStream object.
I have some blobs in Azure Storage which I'd like to email as attachments. To achieve this I'm trying to use a MemoryStream:
var getBlob = blobContainer.GetBlobReferenceFromServer(fileUploadLink.Name);
if(getBlob != null)
{
// Get file as a stream
MemoryStream memoryStream = new MemoryStream();
getBlob.DownloadToStream(memoryStream);
emailMessage.AddAttachment(memoryStream, fileUploadLink.Name);
}
emailTransport.Deliver(emailMessage);
It sends fine but when the email arrives, the attachment appears to be there but it's actually empty. Looking at the email source, there is no content for the attachment.
Is using a MemoryStream a known limitation when using the SendGrid C# API to send attachments? Or should I be approaching this in some other way?
You probably just need to reset the stream position back to 0 after you call DownloadToStream:
var getBlob = blobContainer.GetBlobReferenceFromServer(fileUploadLink.Name);
if (getBlob != null)
{
var memoryStream = new MemoryStream();
getBlob.DownloadToStream(memoryStream);
memoryStream.Seek(0,SeekOrigin.Begin); // Reset stream back to beginning
emailMessage.AddAttachment(memoryStream, fileUploadLink.Name);
}
emailTransport.Deliver(emailMessage);
You might want to check who cleans up the stream as well and if they don't you should dispose of it after you've called Deliver().
According to their API, they have implemented void AddAttachment(Stream stream, String name).
You are probably using a MemoryStream which you have written to before. I suggest resetting the position inside the stream to the beginning, like:
memoryStream.Seek(0, SeekOrigin.Begin);
I ended up with the following which fixed the issue for me:
fileByteArray = new byte[getBlob.Properties.Length];
getBlob.DownloadToByteArray(fileByteArray, 0);
attachmentFileStream = new MemoryStream(fileByteArray);
emailMessage.AddAttachment(attachmentFileStream, fileUploadLink.Name);
The thread is a bit old, but I use a varient with NReco PDF converter:
private async Task SendGridasyncBid(string from, string to, string displayName, string subject, **byte[] PDFBody**, string TxtBody, string HtmlBody)
{
...
var myStream = new System.IO.MemoryStream(**PDFBody**);
myStream.Seek(0, SeekOrigin.Begin);
myMessage.AddAttachment(myStream, "NewBid.pdf");
...
}
convert the html to pdf and return it instead of writing it for download...
private byte[] getHTML(newBidViewModel model)
{
string strHtml = ...;
HtmlToPdfConverter pdfConverter = new HtmlToPdfConverter();
pdfConverter.CustomWkHtmlArgs = "--page-size Letter";
var pdfBytes = pdfConverter.GeneratePdf(strHtml);
return **pdfBytes**;
}
I am not sure how efficient this is, but it is working for me and I hope it helps someone else get their attachments figured out.
I have been trying get a stream from a byte array in metro style app using the following code.
InMemoryRandomAccessStream memoryStream = new InMemoryRandomAccessStream();
memoryStream.AsStreamForWrite().Write(byteArray, 0, byteArray.Length);
memoryStream.Seek(0);
It executes with no errors but stream size is zero (0). Can anybody tell me why is its size is zero?
You can use the DataWriter and DataReader classes. For example ...
// put bytes into the stream
var ms = new InMemoryRandomAccessStream();
var dw = new Windows.Storage.Streams.DataWriter(ms);
dw.WriteBytes(new byte[] { 0x00, 0x01, 0x02 });
await dw.StoreAsync();
// read them out
ms.Seek(0);
byte[] ob = new byte[ms.Size];
var dr = new Windows.Storage.Streams.DataReader(ms);
await dr.LoadAsync((uint)ms.Size);
dr.ReadBytes(ob);
You can also use the BinaryWriter/BinaryReader to read and write from and to byte[] and Streams.
private Stream ConvertToStream(byte[] raw)
{
Stream streamOutput = new MemoryStream();
using (BinaryWriter writer = new BinaryWriter(streamOutput))
{
writer.Write(raw);
}
return streamOutput;
}
Another option is to use built in extension methods as Marc Gravell already mentioned:
private Stream ConvertToStream(byte[] raw)
{
return raw.AsBuffer().AsStream();
}
The extension methods are commented with [Security Critical] which may indicate a later change. However, after looking around a bit I couldn't find any additional information on the security code comment.
I know this is a very old question, but I was running into this issue myself today and figured it out so I'll leave this here for others.
I realized that the stream wasn't being written if it was too small. To fix this, I explicitly set the length of the stream like this:
ms.AsStreamForWrite(imageBytes.Length).Write(imageBytes, 0, imageBytes.Length);
That should be all you need.
I use HttpListener for my own http server (I do not use IIS). I want to compress my OutputStream by GZip compression:
byte[] refBuffer = Encoding.UTF8.GetBytes(...some data source...);
var varByteStream = new MemoryStream(refBuffer);
System.IO.Compression.GZipStream refGZipStream = new GZipStream(varByteStream, CompressionMode.Compress, false);
refGZipStream.BaseStream.CopyTo(refHttpListenerContext.Response.OutputStream);
refHttpListenerContext.Response.AddHeader("Content-Encoding", "gzip");
But I getting error in Chrome:
ERR_CONTENT_DECODING_FAILED
If I remove AddHeader, then it works, but the size of response is not seems being compressed. What am I doing wrong?
The problem is that your transfer is going in the wrong direction. What you want to do is attach the GZipStream to the Response.OutputStream and then call CopyTo on the MemoryStream, passing in the GZipStream, like so:
refHttpListenerContext.Response.AddHeader("Content-Encoding", "gzip");
byte[] refBuffer = Encoding.UTF8.GetBytes(...some data source...);
var varByteStream = new MemoryStream(refBuffer);
System.IO.Compression.GZipStream refGZipStream = new GZipStream(refHttpListenerContext.Response.OutputStream, CompressionMode.Compress, false);
varByteStream.CopyTo(refGZipStream);
refGZipStream.Flush();
The first problem (as mentioned by Brent M Spell) is the wrong position of the header. The second is that you don't use properly the GZipStream. This stream requires a "top" stream to write to, meaning an empty stream (you fill it with your buffer). Having an empty "top" stream then all you have to do is to write on GZipStream your buffer. As a result the memory stream will be filled by the compressed content. So you need something like:
byte[] buffer = ....;
using (var ms = new MemoryStream())
{
using (var zip = new GZipStream(ms, CompressionMode.Compress, true))
zip.Write(buffer, 0, buffer.Length);
buffer = ms.ToArray();
}
response.AddHeader("Content-Encoding", "gzip");
response.ContentLength64 = buffer.Length;
response.OutputStream.Write(buffer, 0, buffer.Length);
Hopeful this might help, they discuss how to get GZIP working.
Sockets in C#: How to get the response stream?
HI
I have an xml file with 500KB size which i need to send it to webservice, so i want to compress this data and send it to the webservice
i have heard of some base24Encoding something...
Can anyone throw more light on this
Suppose if i use GZipStream how can i send the file to the webservice
Thanks in Advance
Something like below (the first part just writes some random xml for us to work with). Your web-service would ideally take a byte[] argument, and would have (if using WSE3 or MCF over basic-http) MTOM enabled, which reduces the base-64 overhead. You just post it the byte[] and then reverse the compression at the other end.
if (File.Exists("my.xml")) File.Delete("my.xml");
using (XmlWriter xmlFile = XmlWriter.Create("my.xml")) {
Random rand = new Random();
xmlFile.WriteStartElement("xml");
for (int i = 0; i < 1000; i++) {
xmlFile.WriteElementString("add", rand.Next().ToString());
}
xmlFile.WriteEndElement();
xmlFile.Close();
}
// now we have some xml!
using (MemoryStream ms = new MemoryStream()) {
int origBytes = 0;
using (GZipStream zip = new GZipStream(ms, CompressionMode.Compress, true))
using (FileStream file = File.OpenRead("my.xml")) {
byte[] buffer = new byte[2048];
int bytes;
while ((bytes = file.Read(buffer, 0, buffer.Length)) > 0) {
zip.Write(buffer, 0, bytes);
origBytes += bytes;
}
}
byte[] blob = ms.ToArray();
string asBase64 = Convert.ToBase64String(blob);
Console.WriteLine("Original: " + origBytes);
Console.WriteLine("Raw: " + blob.Length);
Console.WriteLine("Base64: " + asBase64.Length);
}
Alternatively, consider a different serialization format; there are dense binary protocols that are much smaller (and as a consequence don't benefit from gzip etc). For example, serializing via protobuf-net would give you a very efficient size. But this only applies to an object-model, not to arbitrary xml data.
There are some good articles about that here:
http://www.ibm.com/developerworks/xml/library/x-tipcomp.html
http://www.xml.com/pub/a/2000/03/22/deviant/
The best way to handle this scenario would be to have your web service accept a byte[] parameter which will represent the compressed XML. The Base64 encoding will be done automatically. To improve compression ratio you could use MTOM encoding. This will avoid the Base64 step which consists of converting your byte array to a string in order to send it over the wire and where you loose might in compression ratio.
You have following to choose from:
BinaryFormatter
ArrayList itemsToSerialize = new ArrayList();
itemsToSerialize.Add ( "john" );
itemsToSerialize.Add ( "smith" );
Stream stream = new FileStream( #"MyApplicationData.dat", System.IO.FileMode.Create );
IFormatter formatter = new BinaryFormatter();
formatter.Serialize( stream, itemsToSerialize );
stream.Close();
You could use WCF netTcpBinding
You could use *HttpBinding for WCF service hosted on IIS and follow this blog which walks you through setting up WCF gzip compression through IIS
You could zip response and request and then decompress it
new StreamReader(new GZipStream(webResponse.GetResponseStream(), CompressionMode.Decompress));