I'm having some trouble with finding the right incantation that will allow me to write to a response stream and then later read the contents in a test. Currently I have this
var res = new HttpResponseMessage(System.Net.HttpStatusCode.OK);
var ms = new MemoryStream();
res.Content = new StreamContent(ms);
using (var sw = new StreamWriter(ms, System.Text.Encoding.UTF8))
using (var csv = new CsvHelper.CsvWriter(sw))
csv.WriteRecords(allData.ToList());
return res;
In my test I'm trying to read this response
var controller = appContainer().Resolve<MyController>();
var res = (await controller.Get()) as HttpResponseMessage;
res.ShouldNotEqual(null);
var csv = await res.Content.ReadAsStringAsync();
the last line generates an error
Error while copying content to a stream.
----> System.ObjectDisposedException : Cannot access a closed Stream.
So there's a couple things here
Why is this error happening and how can I prevent it properly in the test?
The use of MemoryStream doesn't sit right with me, shouldn't I be able to write directly to the content's stream? Isn't MemoryStream potentially hugely increasing my memory usage?
Just put this out there, though it's not perfect... Using PushStreamContent does a lot of the job but it comes with its own headaches - namely that any exceptions that your anonymous method may produce will get swallowed and be difficult to track down without a full repro of the problem. When the bomb goes off is well passed the point of the pipeline where web api unhandled exception handlers would come into effect, and the xmlhttprequest doesn't seem to recognize the closure.
E.g. something like
HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
response.Content = new PushStreamContent((stream, content, context) =>
{
// write your output here
});
return response;
will get you what you want, provided that internal method never slips up or goes wrong in any way.
PushStreamContent flushes your http headers immediately before the anonymous method is called, so you're chunked and no way to reel it back in later.
You can add a try/catch in your anonymous method to leave yourself a note if something goes wrong, but in my experience XmlHttpRequest doesn't recognize when the remote server forcibly closes the request so it keeps on waiting. Only started to figure out what was going on when I put Fiddler in there, and Fiddler squawked.
Related
When I use Postman to try uploading a large file to my server (written in .NET Core 2.2), Postman immediately shows the HTTP Error 404.13 - Not Found error: The request filtering module is configured to deny a request that exceeds the request content length
But when I use my code to upload that large file, it gets stuck at the line to send the file.
My client code:
public async void TestUpload() {
StreamContent streamContent = new StreamContent(File.OpenRead("D:/Desktop/large.zip"));
streamContent.Headers.Add("Content-Disposition", "form-data; name=\"file\"; filename=\"large.zip\"");
MultipartFormDataContent multipartFormDataContent = new MultipartFormDataContent();
multipartFormDataContent.Add(streamContent);
HttpClient httpClient = new HttpClient();
Uri uri = new Uri("https://localhost:44334/api/user/testupload");
try {
HttpResponseMessage httpResponseMessage = await httpClient.PostAsync(uri, multipartFormDataContent);
bool success = httpResponseMessage.IsSuccessStatusCode;
}
catch (Exception ex) {
}
}
My server code:
[HttpPost, Route("testupload")]
public async Task UploadFile(IFormFileCollection formFileCollection) {
IFormFileCollection formFiles = Request.Form.Files;
foreach (var item in formFiles) {
using (var stream = new FileStream(Path.Combine("D:/Desktop/a", item.FileName), FileMode.Create)) {
await item.CopyToAsync(stream);
}
}
}
My client code gets stuck at the line HttpResponseMessage httpResponseMessage = await httpClient.PostAsync(uri, multipartFormDataContent), while the server doesn't receive any request (I use a breakpoint to ensure that).
It gets stuck longer if the file is bigger. Looking at Task Manager, I can see my client program uses up high CPU and Disk as it is actually uploading the file to the server. After a while, the code moves to the next line which is
bool success = httpResponseMessage.IsSuccessStatusCode
Then by reading the response content, I get exactly the result as of Postman.
Now I want to know how to immediately get the error to be able to notify the user in time, I don't want to wait really long.
Note that when I use Postman to upload large files, my server doesn't receive any request as well. I think I am missing something, maybe there is problem with my client code.
EDIT: Actually I think it is the client-side error. But if it is server-side error, then it still doesn't mean too much for me. Because, let me clear my thought. I want to create this little helper class that I can use across projects, maybe I can share it with my friends too. So I think it should be able, like Postman, to determine the error as soon as possible. If Postman can do, I can too.
EDIT2: It's weird that today I found out Postman does NOT know before hand whether the server accepts big requests, I uploaded a big file and I saw it actually sent the whole file to the server until it got the response. Now I don't believe in myself anymore, why I thought Postman knows ahead of time the error, I must be stupid. But it does mean that I have found a way to do my job even better than Postman, so I think this question might be useful for someone.
Your issue has nothing to do with your server-side C# code. Your request gets stuck because of what is happening between the client and the server (by "server" I mean IIS, Apache, Nginx..., not your server-side code).
In HTTP, most clients don't read response until they send all the request data. So, even if your server discovers that the request is too large and returns an error response, the client will not read that response until the server accepts the whole requests.
When it comes to server-side, you can check this question, but I think it would be more convenient to handle it on the client side, by checking the file size before sending it to the server (this is basically what Postman is doing in your case).
Now I am able to do what I wanted. But first I want to thank you #Marko Papic, your informations do help me in thinking about a way to do what I want.
What I am doing is:
First, create an empty ByteArrayContent request, with the ContentLength of the file I want to upload to the server.
Second, surround HttpResponseMessage = await HttpClient.SendAsync(HttpRequestMessage) in a try-catch block. The catch block catches HttpRequestException because I am sending a request with the length of the file but my actual content length is 0, so it will throw an HttpRequestException with the message Cannot close stream until all bytes are written.
If the code reaches the catch block, it means the server ALLOWS requests with the file size or bigger. If there is no exception and the code moves on to the next line, then if HttpResponseMessage.StatusCode is 404, it means the server DENIES requests bigger than the file size. The case when HttpResponseMessage.StatusCode is NOT 404 will never happen (I'm not sure about this one though).
My final code up to this point:
private async Task<bool> IsBigRequestAllowed() {
FileStream fileStream = File.Open("D:/Desktop/big.zip", FileMode.Open, FileAccess.Read, FileShare.Read);
if(fileStream.Length == 0) {
fileStream.Close();
return true;
}
HttpRequestMessage = new HttpRequestMessage();
HttpMethod = HttpMethod.Post;
HttpRequestMessage.Method = HttpMethod;
HttpRequestMessage.RequestUri = new Uri("https://localhost:55555/api/user/testupload");
HttpRequestMessage.Content = new ByteArrayContent(new byte[] { });
HttpRequestMessage.Content.Headers.ContentLength = fileStream.Length;
fileStream.Close();
try {
HttpResponseMessage = await HttpClient.SendAsync(HttpRequestMessage);
if (HttpResponseMessage.StatusCode == HttpStatusCode.NotFound) {
return false;
}
return true; // The code will never reach this line though
}
catch(HttpRequestException) {
return true;
}
}
NOTE: Note that my approach still has a problem. The problem with my code is the ContentLength property, it shouldn't be exact the length of the file, it should be bigger. For example, if my file is exactly 1000 bytes in length, then if the file is successfully uploaded to the server, the Request that the server gets has greater ContentLength value. Because HttpClient doesn't just only send the content of the file, but it has to send many informations in addition. It has to send the boundaries, content types, hyphens, line breaks, etc... Generally speaking, you should somehow find out before hand the exact bytes that HttpClient will send along with your files to make this approach work perfectly (I still don't know how so far, I'm running out of time. I will find out and update my answer later).
Now I am able to immediately determine ahead of time whether the server can accept requests that are as big as the file my user wants to upload.
I'm currently downloading a file from my Web API using a C# RestClient.
This is my current code for returning a file from the Web API part:
[HttpGet]
public HttpResponseMessage Generate()
{
var stream = new MemoryStream();
// processing the stream.
var result = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new ByteArrayContent(stream.GetBuffer())
};
result.Content.Headers.ContentDisposition =
new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment")
{
FileName = "CertificationCard.pdf"
};
result.Content.Headers.ContentType =
new MediaTypeHeaderValue("application/octet-stream");
return result;
}
Taken from this: How to return a file (FileContentResult) in ASP.NET WebAPI
My question is then, how can i validate that the file is downloaded correctly - can i somehow provide an MD5 checksum on the ByteArray and check this in the RestClient, or is this complete unnecessary?
You would generate a hash of the file, add it as a response header and verify when the download completes within the client.
This would only make sense if you think there is a chance of corruption of the data within your stream or network issues outside the ability of TCP error correction to handle.
How necessary this is is a judgement call, see Why is it good practice to compare checksums when downloading a file? for a discussion. (Considering the hash & data originate from the same place in the same response, the security considerations don't really apply)
I'm trying to connect to a companies web service, but I'm getting a BadRequest value returned. I suppose that isn't a huge issue, and I'm not asking for help on how to fix the bad request. The issue I'm running into, is how to get more details on what is causing it. So I have this HttpResponseMessage Class variable:
HttpResponseMessage response = c.Post(address, content);
//...fill in content
response = cAssessments.Post(addressInfo, content); //return "Bad Response"
I send it off to their servers with the proper content, and I get back 400 response stating "Bad Request". My question is how do I dig deeper into that repsonse variable to see exactly what is causing it. I'm figuring that there is some dependance on the service I'm hitting to supply information so I'm going to assume it's in there somewhere. But what are something things inside that variable I should be looking into? For example:
"response.StatusCode" gives me the the 400 code. What other properties can I look into?
Any help is appreciated.
Filling content of BadRequest response with some useful tip or information what could possibly go wrong is totally up to web service which serves response. You cant do anything from your side.
Try the Content itself, also the URL of the originating request.
Depending on the API you're requesting against, they may include what specifically caused the 400 in the response's Content.
Also the format you are requesting it in (json vs xml vs whatever)
It all depends on what the service is returning. To check that, you can read the response stream:
using (var stream = response.GetResponseStream ()){
using (var reader = new StreamReader(stream, Encoding.UTF8)
{
var text = reader.ReadToEnd();
// text contains your info
}
}
Another useful option for debugging is using something like postman, where you can paste in your request and see the full response immediately
If the response doesn't include any useful info, you're stuck though, there's nothing you can do about that.
I'm trying to extend my REST service (built using WCF/webHttpBinding) so that clients can upload gzipped data. I'm not sure what the best way to accomplish this but I thought it would be fairly easy by adding a HTTP module which will decompress the data if Content-Encoding for the incoming request is set to gzip.
So I created an class deriving from IHttpModule with the following implementation:
private void OnBeginRequest(object sender, EventArgs e)
{
var app = (HttpApplication) sender;
var context = app.Context;
var contentEncoding = context.Request.Headers["Content-Encoding"];
if (contentEncoding == "gzip")
{
// some debug code:
var decompressedStream = new GZipStream(context.Request.InputStream, CompressionMode.Decompress);
var memoryStream = new MemoryStream();
decompressedStream.CopyTo(memoryStream);
memoryStream.Seek(0, SeekOrigin.Begin);
var streamReader = new StreamReader(memoryStream);
string msg = streamReader.ReadToEnd();
context.Request.InputStream.Seek(0, SeekOrigin.Begin);
app.Request.Filter = //new TestFilterStream(app.Request.Filter);
new System.IO.Compression.GZipStream(
app.Request.Filter, CompressionMode.Decompress);
}
}
The issue I'm seeing is that the GZipStream decompression is never actually performed. I've confirmed that the incoming data is in fact gzip'd (the msg-variable contains the proper data). I've also tried creating my own stream class (TestFilterStream) above and assign that to app.Request.Filter and I've confirmed that no members on the stream class is actually called by ASP.NET. So it seems like while it's possible to specify a filter, that filter isn't actually used.
Isn't HttpApplication.Request.Filter actually used?
I tried setting the Request Filter in two ways:
Using a HttpModule
Setting it in the start of Application_BeginRequest() (Global.asax)
Both with the same results (VS2012 web project + IISExpress):
If there is no input data (GET request or similar), the Filter.Read is not invoked
In case of a POST with actual data, the filter is executed and the web service gets the filtered data
Even if I read from the Request.InputStream before the Filter is set, I still get the filter triggered from my service code.
I have no easy way of testing with Gzippet input, so I have not tried if the actual filter works. However, I know it is getting triggered, since I get an error from GZipStream while it attempts to look for the input.
Perhaps you are having other HttpModules or Filters that disrupt your input or control flow?
This post proposes a method similar to yours, but also states the following, which may be causing some side effects (my tests were not using WCF):
"It appears that this approach trigger a problem in WCF, as WCF relies on the original Content-Length and not the value obtained after decompressing."
I've just done a couple of tests, and my Request.Filter stream is called into, as long as there is a request body and the request body gets read by the http handler. I'm guessing you use either a PUT or a POST, and certainly read the request body, so that shouldn't be a problem.
I suspect Knaģis' comment is correct. Have you tried without the debug code? If I dig into the HttpRequest source, I see a variable _rawContent being written to exactly once; at that same time the request filters are applied. After that the _rawContent value just gets cached, and is never updated (nor reset when a filter is added).
So by calling Request.InputStream in your debug code you are definitely preventing your filter from being applied later on. Reading the Request.Headers collection is no problem.
Are you sure, that application itself should bother?
Usually that's handled per configuration of host (IIS). So, basically, you only need to implement custom GZip support, when you host the service yourself.
You can take a look here
I have created a simple REST based WCF service which runs on BasicHttpBinding. In one of my webmethod, I am returning a Stream which points to a JSON response.
The Method looks like :
[OperationContract]
[FaultContract(typeof(ApplicationFault))]
[WebInvoke(Method = "POST", UriTemplate = "GetActiveCalls/{nurseid}")]
Stream GetActiveCalls(string nurseid);
From the body of the GetActiveCalls, I am creating an object of MemoryStream and returning the same as response. The code looks like
// Serialize the results as JSON
string jsonResult = new JavaScriptSerializer().Serialize(baseResponses);
// ContentType json
WebOperationContext.Current.OutgoingResponse.ContentType = "application/json";
WebOperationContext.Current.OutgoingResponse.Headers.Add("Cache-Control", "no-cache");
var bytes = Encoding.UTF8.GetBytes(jsonResult);
//Parse to memorystream
var ms = new MemoryStream(bytes);
ms.Seek(0, SeekOrigin.Begin);
ms.SetLength(bytes.LongLength);
return ms;
When trying this from client, I get result like
{"LastEvents":[{"FormatValues":"Klic 2 3 4","Icon":null,"Color":"Red","Acknowledged":false,"EventID":28566}],"Message":"","Status":true}
But sometimes after invoking the same method for multiple times, I start getting the response as :
{"LastEvents":[{"FormatValues":"Klic 2 3 4","Icon":null,"Color":"Red","Acknowledged":false,"EventID":28566}],"Message":"","Statu{"LastEv
You can see after "Statu on the JSON response, the stream gets reset and starts getting data from the beginning.
It looks strange to me.
*From server side, when I put breakpoint, it seems the MemoryStream has correct response.
Putting aside the question of using a memory stream or not, I encountered a similar issue just recently, where the memory stream response appeared corrupted, seemingly randomly. The solution to this problem was to remove the tracing sections from web.config, which I had turned on in dev mode. This may or may not be your issue, but it might be worth having a look at. Seems as though this problem is still present in .NET 4.5 as well.