I have (several) WebAPI action(s), which load QuickFix logs from database (via EF) and use this private method to return them as CSV:
private HttpResponseMessage BuildCsvResponse<T>(T[] entries, Func<T, string> row, string fileName)
{
var response = new HttpResponseMessage(HttpStatusCode.OK);
var stream = new MemoryStream();
var writer = new StreamWriter(stream);
var i = entries.Length;
foreach (var entry in entries)
{
i--;
writer.WriteLine(row(entry)); // simply call to overridden ToString() method
}
stream.Seek(0, SeekOrigin.Begin);
stream.Flush();
response.Content = new StreamContent(stream);
response.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment")
{
FileName = fileName,
};
response.Content.Headers.ContentType = new MediaTypeHeaderValue("text/csv");
return response;
}
The problem is that content is never loaded to the end and cut on random symbol not so far from end. Why could it happen?
May be it is important - all log strings contain delimiter 0x01
You need to Flush your streamwriter's internal buffers before you touch the underlying stream.
Best is to tell your StreamWriter to keep the stream open by using another contructor. You can then safely dispose your streamwriter causing it to flush its buffer while your memorystream instance stays open and doesn't get disposed.
Notice that you need to pick an encoding that matches your HTTP content response. I choose UTF8 here, adapt accordingly.
var stream = new MemoryStream();
// notice the true as last parameter, false is the default.
using(var writer = new StreamWriter(stream, Encoding.UTF8, 8192, true))
{
var i = entries.Length;
foreach (var entry in entries)
{
i--;
writer.WriteLine(row(entry)); // simply call to overridden ToString() method
}
}
// your streamwriter has now flushed its buffer and left the stream open
stream.Seek(0, SeekOrigin.Begin);
// calling Flush on the stream was never needed so I removed that.
response.Content = new StreamContent(stream);
Related
I want to write a JToken to a stream asynchronously. And I referred to JToken.WriteToAsync does not write to JsonWriter.
However, the stream output is ?[], while ToString() output is []. Why does the stream contain extra bytes at the beginning?
My code is below:
static async Task Main(string[] args)
{
JArray arr = new JArray();
//var c = JToken.FromObject("abc");
//arr.Add(c);
var stream = new MemoryStream();
await using (var requestWriter = new StreamWriter(stream, System.Text.Encoding.UTF8, leaveOpen: true))
{
var jsonWriter = new JsonTextWriter(requestWriter);
try
{
await arr.WriteToAsync(jsonWriter);
}
finally
{
await jsonWriter.CloseAsync();
}
Console.WriteLine(System.Text.Encoding.UTF8.GetString(stream.GetBuffer(), 0, checked((int)stream.Length)));
Console.WriteLine(arr.ToString());
}
}
Why stream output is not correct?
The Json.net's version is 13.0.1.
Summary
Your problem has nothing to do with asynchronous writing. Your problem is that Encoding.UTF8:
returns a UTF8Encoding object that provides a Unicode byte order mark (BOM).
The extra ? you are seeing is that BOM. To prevent the BOM from being written, use new UTF8Encoding(false) when writing. Or, you could just do new StreamWriter(stream, leaveOpen: true) as the StreamWriter constructors will use a UTF-8 encoding without a Byte-Order Mark (BOM) by default.
Details
Your problem can be reproduced more simply as follows:
JArray arr = new JArray();
var stream = new MemoryStream();
using (var requestWriter = new StreamWriter(stream, System.Text.Encoding.UTF8, leaveOpen: true))
using (var jsonWriter = new JsonTextWriter(requestWriter))
{
arr.WriteTo(jsonWriter);
}
var resultJson = Encoding.UTF8.GetString(stream.GetBuffer(), 0, checked((int)stream.Length));
Console.WriteLine(BitConverter.ToString(stream.GetBuffer(), 0, checked((int)stream.Length)));
Console.WriteLine(resultJson);
Console.WriteLine(arr.ToString());
Assert.AreEqual(arr.ToString(), resultJson);
The assertion fails with the following message:
NUnit.Framework.AssertionException: Expected string length 2 but was 3. Strings differ at index 0.
And with the following output from BitConverter.ToString():
EF-BB-BF-5B-5D
Demo fiddle here.
The 5B-5D are the brackets, but what are the three preamble characters EF-BB-BF? A quick search shows it to be the UTF-8 byte order mark. Since RFC 8259 specifies that Implementations MUST NOT add a byte order mark (U+FEFF) to the beginning of a networked-transmitted JSON text you should omit the BOM by using new UTF8Encoding(false). Thus your code should look like:
JArray arr = new JArray();
var stream = new MemoryStream();
await using (var requestWriter = new StreamWriter(stream, new UTF8Encoding(false), leaveOpen: true))
{
var jsonWriter = new JsonTextWriter(requestWriter);
try
{
await arr.WriteToAsync(jsonWriter);
}
finally
{
await jsonWriter.CloseAsync();
}
}
var resultJson = Encoding.UTF8.GetString(stream.GetBuffer(), 0, checked((int)stream.Length));
Console.WriteLine(BitConverter.ToString(stream.GetBuffer(), 0, checked((int)stream.Length)));
Console.WriteLine(resultJson);
Console.WriteLine(arr.ToString());
Assert.AreEqual(arr.ToString(), resultJson);
Demo fiddle #2 here.
I'm trying to get body from request of an authorization class (AuthorizationHandler), but that body is a Stream and after reading your content, the post request that comes on next can not be executed because Stream content has been disposable.
I'm using this code to get Stream content:
protected override Task HandleRequirementAsync(AuthorizationHandlerContext context, Autorizacao requirement)
{
var routeValues = context.Resource as AuthorizationFilterContext;
if (routeValues != null)
{
var obj = StreamToObject(routeValues.HttpContext.Request.Body);
context.Succeed(requirement);
}
return Task.FromResult(0);
}
private Object StreamToObject(Stream stream)
{
try
{
string content;
using (var reader = new StreamReader(stream))
content = reader.ReadToEnd();
return Newtonsoft.Json.JsonConvert.DeserializeObject(content);
}
catch (Exception e)
{
throw e;
}
}
How i can do to workaround this problem ?
StreamReader has a special constructor that allow you to pass a boolean as last parameter. It will prevent dispose underlying stream
EDIT: Because ReadToEnd do not restore position in stream you should do it by yourself like this:
var position = stream.Position;
using (var reader = new StreamReader(stream, Encoding.UTF8, false, 8192, true))
content = reader.ReadToEnd();
stream.Seek(position, SeekOrigin.Begin);
EDIT 2: From MSDN I see that Body has a setter. So you can replace original Body with memory stream:
if (routeValues != null)
{
var memoryStream = new MemoryStream();
routeValues.HttpContext.Request.Body.CopyTo(memoryStream);
// reset position after CopyTo
memoryStream.Seek(0, SeekOrigin.Begin);
var obj = StreamToObject(memoryStream);
// reset position after ReadToEnd
memoryStream.Seek(0, SeekOrigin.Begin);
routeValues.HttpContext.Request.Body = memoryStream;
context.Succeed(requirement);
}
Maybe not needed any more, but you can set request.EnableRewind() and then do the request.Body.Seek(0, SeekOrigin.Begin);
Work for me
My webapi method for zipping on the fly use this code
var result = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new PushStreamContent((stream, content, arg3) =>
{
using (var zipEntry = new Ionic.Zip.ZipFile())
{
using (var ms = new MemoryStream())
{
_xmlRepository.GetInitialDataInXml(employee, ms);
zipEntry.AddEntry("content.xml", ms);
zipEntry.Save(stream); //process sleep on this line
}
}
})
};
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment")
{
FileName = "FromPC.zip"
};
result.Content.Headers.ContentType =
new MediaTypeHeaderValue("application/octet-stream");
return result;
I want to
1) take data from _xmlRepository.GetInitialDataInXml
2) zip data on the fly via Ionic.Zip
3) return zipped stream as output of my WebApi action
But on this line zipEntry.Save(stream); execution process stops and don't go to next line. And method don't return anything
So why it doesnt' return me file?
When using PushStreamContent, you would need to close the stream to signal that you are done writing to the stream.
Remarks section in the documentation:
http://msdn.microsoft.com/en-us/library/jj127066(v=vs.118).aspx
The accepted answer is not correct. It is not necessary to close the stream if you want to start streaming. The streaming starts automatically (download dialog in browser) when the delegated function ends. In case of big files OutOfMemoryException is thrown, but it is handled and the streaming begins -> HttResponseStream is flushed towards the client.
var result = new HttpResponseMessage(HttpStatusCode.OK);
result.Content = new PushStreamContent(async (outputStream, httpContext, transportContext) =>
{
using (var zipStream = new ZipOutputStream(outputStream))
{
var employeeStream = _xmlRepository.GetEmployeeStream(); // PseudoCode
zipStream.PutNextEntry("content.xml");
await employeeStream.CopyToAsync(zipStream);
outputStream.Flush();
}
});
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment") { FileName = "FromPC.zip" };
result.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
return result;
In my program, I am basically reading in a file, doing some processing to it, and then passing it back to the main program as a memorystream, which will be handled by a streamreader. This will all be handled by a class beside my main.
The problem is, when I return the memory stream from my method in another class, the "canread" variable is set to false, and thus causes the streamreader initialization to fail.
Below is an example of the problem happening (though in here I'm writing to the memorystream in the other class, but it still causes the same error when i pass it back.
In the class named "Otherclass":
public static MemoryStream ImportantStreamManipulator()
{
MemoryStream MemStream = new MemoryStream();
StreamWriter writer = new StreamWriter(MemStream);
using (writer)
{
//Code that writes stuff to the memorystream via streamwriter
return MemStream;
}
}
The function calls in the main program:
MemoryStream MStream = Otherclass.ImportantStreamManipulator();
StreamReader reader = new StreamReader(MStream);
When I put a breakpoint on the "return MemStream", the "CanRead" property is still set to true. Once I step such that it gets back to my main function, and writes the returned value to MStream, the "CanRead" property is set to false. This then causes an exception in StreamReader saying that MStream could not be read (as the property indicated). The data is in the streams buffer as it should be, but I just can't get it out.
How do I set it so that "CanRead" will report true once it is returned to my main? Or am I misunderstanding how MemoryStream works and how would I accomplish what I want to do?
This is the problem:
using (writer)
{
//Code that writes stuff to the memorystream via streamwriter
return MemStream;
}
You're closing the writer, which closes the MemoryStream. In this case you don't want to do that... although you do need to flush the writer, and rewind the MemoryStream. Just change your code to:
public static MemoryStream ImportantStreamManipulator()
{
// Probably add a comment here stating that the lack of using statements
// is deliberate.
MemoryStream stream = new MemoryStream();
StreamWriter writer = new StreamWriter(stream);
// Code that writes stuff to the memorystream via streamwriter
writer.Flush();
stream.Position = 0;
return stream;
}
The StreamWriter takes ownership of the memory stream and when the using statement ends, the MemoryStream is also closed.
See Is there any way to close a StreamWriter without closing its BaseStream?.
As others have stated, the problem is that the Stream is closed when the StreamWriter is closed. One possible way to deal with this is to return a byte array rather than a MemoryStream. This avoids having potentially long running objects that must be disposed by the garbage collector.
public static void Main()
{
OutputData(GetData());
}
public static byte[] GetData()
{
byte[] binaryData = null;
using (MemoryStream ms = new MemoryStream())
using (StreamWriter sw = new StreamWriter(ms))
{
string data = "My test data is really great!";
sw.Write(data);
sw.Flush();
binaryData = ms.ToArray();
}
return binaryData;
}
public static void OutputData(byte[] binaryData)
{
using (MemoryStream ms = new MemoryStream(binaryData))
using (StreamReader sr = new StreamReader(ms))
{
Console.WriteLine(sr.ReadToEnd());
}
}
Another method is to copy the Stream to another stream prior to returning. However, this still has the problem that subsequent access to it with a StreamReader will close that stream.
public static void RunSnippet()
{
OutputData(GetData());
}
public static MemoryStream GetData()
{
MemoryStream outputStream = new MemoryStream();
using (MemoryStream ms = new MemoryStream())
using (StreamWriter sw = new StreamWriter(ms))
{
string data = "My test data is really great!";
sw.Write(data);
sw.Flush();
ms.WriteTo(outputStream);
outputStream.Seek(0, SeekOrigin.Begin);
}
return outputStream;
}
public static void OutputData(MemoryStream inputStream)
{
using (StreamReader sr = new StreamReader(inputStream))
{
Console.WriteLine(sr.ReadToEnd());
}
}
We have a very odd problem, the below code is working fine on all developers machine/ our 2 test servers, both with code and with built version, however when it is running on a virtual machine with windows 2003 server and asp.net v2.0 it throws an error
Cannot access a closed stream.
public String convertResultToXML(CResultObject[] state)
{
MemoryStream stream = null;
TextWriter writer = null;
try
{
stream = new MemoryStream(); // read xml in memory
writer = new StreamWriter(stream, Encoding.Unicode);
// get serialise object
XmlSerializer serializer = new XmlSerializer(typeof(CResultObject[]));
serializer.Serialize(writer, state); // read object
int count = (int)stream.Length; // saves object in memory stream
byte[] arr = new byte[count];
stream.Seek(0, SeekOrigin.Begin);
// copy stream contents in byte array
stream.Read(arr, 0, count);
UnicodeEncoding utf = new UnicodeEncoding(); // convert byte array to string
return utf.GetString(arr).Trim();
}
catch
{
return string.Empty;
}
finally
{
if (stream != null) stream.Close();
if (writer != null) writer.Close();
}
}
Any idea why would it do this?
For your Serialize use using to prevent the stream remain open.
Something like this:
using (StreamWriter streamWriter = new StreamWriter(fullFilePath))
{
xmlSerializer.Serialize(streamWriter, toSerialize);
}
I originally thought that it was because you're closing the stream then closing the writer - you should just close the writer, because it will close the stream also : http://msdn.microsoft.com/en-us/library/system.io.streamwriter.close(v=vs.80).aspx.
However, despite MSDNs protestation, I can't see any evidence that it does actually does this when reflecting the code.
Looking at your code, though, I can't see why you're using the writer in the first place. I'll bet if you change your code thus (I've taken out the bad exception swallowing too) it'll be alright:
public String convertResultToXML(CResultObject[] state)
{
using(var stream = new MemoryStream)
{
// get serialise object
XmlSerializer serializer = new XmlSerializer(typeof(CResultObject[]));
serializer.Serialize(stream, state); // read object
int count = (int)stream.Length; // saves object in memory stream
byte[] arr = new byte[count];
stream.Seek(0, SeekOrigin.Begin);
// copy stream contents in byte array
stream.Read(arr, 0, count);
UnicodeEncoding utf = new UnicodeEncoding(); // convert byte array to string
return utf.GetString(arr).Trim();
}
}
Now you're working with the stream directly, and it'll only get closed once - most definitely getting rid of this strange error - which I'll wager could be something to do with a service pack or something like that.