I'm trying to POST a MultipartFormDataContent in a real case scenario, a data content object could contain anything from a simple string to a video file I'm using a serialized object down there, just a proof of concept.
Also I would like to note that using JSON objects wont serve my real life scenarios
public class GzipMultipartContent : MultipartFormDataContent
{
public GzipMultipartContent()
{
Headers.ContentEncoding.Add("gzip");
}
protected override Task SerializeToStreamAsync(Stream stream, TransportContext context)
{
return Task.Factory.StartNew(() =>
{
using (var gzip = new GZipStream(stream, CompressionMode.Compress, true))
base.SerializeToStreamAsync(gzip, context);
});
}
}
and here's how I call it
var gzipped = new GzipMultipartContent();
var test = new TestClass();
gzipped.Add(new StringContent(JsonConvert.SerializeObject(test)), "value");
var client = new HttpClient();
var result = client.PostAsync("http://localhost:60001/api/Home/", gzipped).Result;
and here's the post action in the controller
// POST: api/Home
[HttpPost]
public void Post([FromForm] object value)
{
}
I have added a break point at the server side and made sure it doesn't even reach the Post method, also I have tried with a normal POST request to make sure that it's not a server configuration problem or a URL mistyping
Client side
If the code in question is your real code, then there are at least two issues:
Did not wait on base.SerializeToStreamAsync
You created a new task, but you did not wait until the base class completed writing to the compressed stream in the task. So you could send unpredictable content to server.
Did not override Content-Length
MultipartFormDataContent calculates length of content based on data not compressed, since you have compressed data, you must re-compute length for the compressed data.
Frankly, I don't think you need to inherit from MultipartFormDataContent to make it compressed. Instead, you could compress the entire MultipartFormDataContent in a wrapper HttpContent:
public class GzipCompressedContent : HttpContent
{
private readonly HttpContent _content;
public GzipCompressedContent(HttpContent content)
{
// Copy original headers
foreach (KeyValuePair<string, IEnumerable<string>> header in content.Headers)
{
Headers.TryAddWithoutValidation(header.Key, header.Value);
}
Headers.ContentEncoding.Add("gzip");
_content = content;
}
protected override async Task SerializeToStreamAsync(Stream stream, TransportContext context)
{
using (var gzip = new GZipStream(stream, CompressionMode.Compress, true))
{
// Compress the entire original content
await _content.CopyToAsync(gzip);
}
}
protected override bool TryComputeLength(out long length)
{
// Content-Lenght is optional, so set to -1
length = -1;
return false;
}
}
And use it:
var test = new TestClass();
using (var client = new HttpClient())
{
var form = new MultipartFormDataContent();
form.Add(new StringContent(JsonConvert.SerializeObject(test)), "value");
var compressed = new GzipCompressedContent(form);
var result = await client.PostAsync(..., compressed);
}
Server side
Your server needs to support compressed stream.
For example, by default, ASP.NET Core does not support compressed request, if you send GZip compressed request to an ASP.NET Core application, you will see exception:
System.IO.IOException: Unexpected end of Stream, the content may have already been read by another component.
at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
The above exception happens in action invocation pipeline before any controller action takes place. So controller actions in this case could not be reached.
To fix such, you will need to enable server side request decompression support.
If you are using ASP.NET Core, check out this nuget package.
I am not sure if I understood the issue, but if it is that your request isn't getting to the server, while your "normal" POST requests are, then I think that I found your problem.
I think that the issue is that your server doesn't know what Content-Type is coming to it. I literally copy-pasted your code, but added
Headers.ContentType = new MediaTypeHeaderValue("application/x-gzip");
to GzipMultipartContent.cs ctor.
After I added the type, I was hitting my breakpoint in the localhost server.
Source: Content-Type
In requests, (such as POST or PUT), the client tells the server what type of data is actually sent.
Related
I'm using Polly in combination with Microsoft.Extensions.Http.Polly to handle communication with an external API which has rate-limiting (N requests / second).I'm also using .NET 6.
The policy itself works fine for most requests, however it doesn't work properly for sending (stream) data. The API Client requires the usage of MemoryStream. When the Polly policy handles the requests and retries it, the stream data is not sent.
I verified this behavior stems from .NET itself with this minimal example:
using var fileStream = File.OpenRead(#"C:\myfile.pdf");
using var memoryStream = new MemoryStream();
await fileStream.CopyToAsync(memoryStream);
var response = await httpClient.SendAsync(
new HttpRequestMessage
{
// The endpoint will fail the request on the first request
RequestUri = new Uri("https://localhost:7186/api/test"),
Content = new StreamContent(memoryStream),
Method = HttpMethod.Post
}
);
Inspecting the request I see that Request.ContentLength is the length of the file on the first try. On the second try it's 0.
However if I change the example to use the FileStream directly it works:
using var fileStream = File.OpenRead(#"C:\myfile.pdf");
var response = await httpClient.SendAsync(
new HttpRequestMessage
{
// The endpoint will fail the request on the first request
RequestUri = new Uri("https://localhost:7186/api/test"),
Content = new StreamContent(fileStream ),
Method = HttpMethod.Post
}
);
And this is my Polly policy that I add to the chain of AddHttpClient.
public static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
return Policy
.HandleResult<HttpResponseMessage>(response =>
{
return response.StatusCode == System.Net.HttpStatusCode.Forbidden;
})
.WaitAndRetryAsync(4, (retry) => TimeSpan.FromSeconds(1));
}
My question:
How do I properly retry requests where StreamContent with a stream of type MemoryStream is involved, similar to the behavior of FileStream?
Edit for clarification:
I'm using an external API Client library (Egnyte) which accepts an instance of HttpClient
public class EgnyteClient {
public EgnyteClient(string apiKey, string domain, HttpClient? httpClient = null){
...
}
}
I pass an instance which I injected via the HttpContextFactory pattern. This instance uses the retry policy from above.
This is my method for writing a file using EgnyteClient
public async Task UploadFile(string path, MemoryStream stream){
// _egnyteClient is assigned in the constructor
await _egnyteClient.Files.CreateOrUpdateFile(path, stream);
}
This method call works (doesn't throw an exception) even when the API sometimes returns a 403 statucode because the internal HttpClient uses the Polly retry policy. HOWEVER the data isn't always properly transferred since it just works if it was the first attempt.
The root cause of your problem could be the following: once you have sent out a request then the MemoryStream's Position is at the end of the stream. So, any further requests needs to rewind the stream to be able to copy it again into the StreamContent (memoryStream.Position = 0;).
Here is how you can do that with retry:
private StreamContent GetContent(MemoryStream ms)
{
ms.Position = 0;
return new StreamContent(ms);
}
var response = await httpClient.SendAsync(
new HttpRequestMessage
{
RequestUri = new Uri("https://localhost:7186/api/test"),
Content = GetContent(memoryStream),
Method = HttpMethod.Post
}
);
This ensures that the memoryStream has been rewinded for each each retry attempt.
UPDATE #1
After receiving some clarification and digging in the source code of the Egnyte I think I know understand the problem scope better.
A 3rd party library receives an HttpClient instance which is decorated with a retry policy (related source code)
A MemoryStream is passed to a library which is passed forward as a StreamContent as a part of an HttpRequestMessage (related source code)
HRM is passed directly to the HttpClient and the response is wrapped into a ServiceResponse (related source code)
Based on the source code you can receive one of the followings:
An HttpRequestException thrown by the HttpClient
An EgnyteApiException or QPSLimitExceededException or RateLimitExceededException thrown by the ExceptionHelper
An EgnyteApiException thrown by the SendRequestAsync if there was a problem related to the deserialization
A ServiceResponse from SendRequestAsync
As far as I can see you can access the StatusCode only if you receive an HttpRequestException or an EgnyteApiException.
Because you can't rewind the MemoryStream whenever an HttpClient performs a retry I would suggest to decorate the UploadFile with retry. Inside the method you can always set the stream parameter's Position to 0.
public async Task UploadFile(string path, MemoryStream stream){
stream.Position = 0;
await _egnyteClient.Files.CreateOrUpdateFile(path, stream);
}
So rather than decorating the entire HttpClient you should decorate your UploadFile method with retry. Because of this you need to alter the policy definition to something like this:
public static IAsyncPolicy GetRetryPolicy()
=> Policy
.Handle<EgnyteApiException>(ex => ex.StatusCode == HttpStatusCode.Forbidden)
.Or<HttpRequestException>(ex => ex.StatusCode == HttpStatusCode.Forbidden)
.WaitAndRetryAsync(4, _ => TimeSpan.FromSeconds(1));
Maybe the Or builder clause is not needed because I haven't seen any EnsureSuccessStatusCode call anywhere, but for safety I would build the policy like that.
I've an ASP.NET Core server that, upon client request, start fetching a stream from an AXIS camera and returns it to the client for displaying. If the http request to server is made directly by the browser by means of img src attribute, it works fine. But if I use HttpClient (which I need to do since I need CancellationToken), the httpClient.GetStreamAsync instruction get stuck and I cannot parse the returned data.
Server side Controller (take from here):
[ApiController]
[Route("[controller]")]
public class CameraSystemController : ControllerBase
{
private string _contentTypeStreaming = "multipart/x-mixed-replace;boundary=myboundary";
private HttpClient _httpClient = new HttpClient(new HttpClientHandler { Credentials = ...});
[HttpGet("getStream")]
public async Task<IActionResult> GetStream(CancellationToken token)
{
Stream stream = await _httpClient.GetStreamAsync("http://.../axis-cgi/mjpg/video.cgi?resolution=800x600&compression=50", token);
if (stream != null) {
FileStreamResult result = new FileStreamResult(stream, _contentTypeStreaming) {
EnableRangeProcessing = true
};
return result;
} else {
return new StatusCodeResult((int)HttpStatusCode.ServiceUnavailable);
}
}
}
Now, as I said, as long as I make the browser perform the http request by means of
// LiveCamera.razor
<img src="CameraSystem/getStream" onerror="#OnImgLoadingError" alt="">
Data acquired in live mode by the camera is correctly displayed in browser.
If instead I make the request in the client this way:
//LiveCamera.razor.cs
[Inject] public HttpClient Http { get; private set; }
private CancellationTokenSource _cancellationTokenSource = new CancellationTokenSource();
private async void StartStreamingRequest(object sender, ElapsedEventArgs e) {
Console.WriteLine("I'm about to request live stream");
Stream responseStream = await Http.GetStreamAsync("CameraSystem/getStream", _cancellationTokenSource.Token);
Console.WriteLine("Header found!");
string boundary = "myboundary";
for (MultipartReader streamReader = new MultipartReader(boundary, responseStream); ;) {
MultipartSection nextFrame = await streamReader.ReadNextSectionAsync(_cancellationTokenSource.Token);
DisplayOneFrameCallback(nextFrame.Body);
}
}
private void DisplayOneFrameCallback(Stream body)
{
StreamReader reader = new StreamReader(body);
string frameData = reader.ReadToEnd();
_imgSrc = "data:image/jpeg;base64," + frameData;
StateHasChanged();
}
In this seconds case the request is performed correctly (server-side I can see the code executed and from task manager I can see the bandwidth usage increasing) but the client got stuck on await instruction, and subsequent code is never executed. Now, in Microsoft documentation under GetStreamAsync it is possible to read
This operation will not block. The returned Task object will complete after the response headers are read. This method does not read nor buffer the response body.
so I would expect it not to block. Thus I suppose there could be a problem in the header I'm producing server-side, even though browser request works just fine.
Just out of curiosity, I've captured with Wireshark the data between the camera and the server. The header and the initial part of the body are like this:
..-...HTTP/1.0 200 OK..Cache-Control: no-cache..Pragma: no-cache..Expires: Thu, 01 Dec 1994 16:00:00 GMT..Connection: close..Content-Type: multipart/x-mixed-replace;boundary=myboundary....--myboundary..Content-Type: image/jpeg..Content-Length: 30146......
I've double-checked with browser developer tools and I can confirm that sending the request from browser or from httpClient yield the same exact request. Furthermore right-clicking on the request url and issueing "open in new tab" opens a tab where I can see the camera live stream in both cases.
Could you please help me in this?
I have a PUT endpoint which receives a payload and coverts it to an object using [FromBody].
This works fine when running on localhost (using IIS Express).
But when I run it via the production server (IIS), it fails. The validation error is:
{
"": [
"A non-empty request body is required."
]
}
I can recreate this with both my client code and using Postman.
Thinking that maybe the payload was malformed, I used some middleware (posted elsewhere on Stack Overflow) to inspect the body before it reached the endpoint (see code below), but even at that stage the request.body is empty.
I can't see any other places where the request has already been read at this stage (as I appreciate that can clear it).
I've read countless Stack Overflow posts and other web pages, and nothing else seems to come close to this situation (where request.body is empty, even though you can see in the client the payload is sent).
Any help would be greatly appreciated. I'm happy to provide any further details.
For reference, here is the code.
Payload
{"currentlySaved":false,"type":"album"}
End point
// PUT api/<controller>/toggleSaveState
[HttpPut("toggleSaveState/{id}")]
public async Task<IActionResult> Put(string id, [FromBody] ToggleSaveStateRequest requestDetails)
{
...
}
The object representing the payload:
public class ToggleSaveStateRequest
{
public bool CurrentlySaved { get; set; }
public string Type { get; set; }
}
Middleware used to check the request.body
Note: the problem takes place whether I include this middleware or not. I've included it in a bid to diagnose the problem.
public class RequestResponseLoggingMiddleware
{
private readonly RequestDelegate _next;
public RequestResponseLoggingMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task Invoke(HttpContext context)
{
//First, get the incoming request
var request = await FormatRequest(context.Request);
Console.WriteLine($"REQUEST: {request}");
//Copy a pointer to the original response body stream
var originalBodyStream = context.Response.Body;
//Create a new memory stream...
using (var responseBody = new MemoryStream())
{
//...and use that for the temporary response body
context.Response.Body = responseBody;
//Continue down the Middleware pipeline, eventually returning to this class
await _next(context);
//Format the response from the server
var response = await FormatResponse(context.Response);
//TODO: Save log to chosen datastore
Console.WriteLine($"RESPONSE: {response}");
//Copy the contents of the new memory stream (which contains the response) to the original stream, which is then returned to the client.
await responseBody.CopyToAsync(originalBodyStream);
}
}
private async Task<string> FormatRequest(HttpRequest request)
{
var body = request.Body;
//This line allows us to set the reader for the request back at the beginning of its stream.
request.EnableRewind();
//We now need to read the request stream. First, we create a new byte[] with the same length as the request stream...
var buffer = new byte[Convert.ToInt32(request.ContentLength)];
//...Then we copy the entire request stream into the new buffer.
await request.Body.ReadAsync(buffer, 0, buffer.Length);
//We convert the byte[] into a string using UTF8 encoding...
var bodyAsText = Encoding.UTF8.GetString(buffer);
//..and finally, assign the read body back to the request body, which is allowed because of EnableRewind()
request.Body = body;
return $"{request.Scheme} {request.Host}{request.Path} {request.QueryString} {bodyAsText}";
}
private async Task<string> FormatResponse(HttpResponse response)
{
//We need to read the response stream from the beginning...
response.Body.Seek(0, SeekOrigin.Begin);
//...and copy it into a string
string text = await new StreamReader(response.Body).ReadToEndAsync();
//We need to reset the reader for the response so that the client can read it.
response.Body.Seek(0, SeekOrigin.Begin);
//Return the string for the response, including the status code (e.g. 200, 404, 401, etc.)
return $"{response.StatusCode}: {text}";
}
}
Edit:
Version of endpoint without the [FromBody]:
The following is here to try and simplify my explanation.
// PUT api/<controller>/toggleSaveState
[HttpPut("toggleSaveState/{id}")]
public async Task<IActionResult> Put(string id)
{
using (var reader = new StreamReader(Request.Body))
{
var body = reader.ReadToEnd();
Console.WriteLine("body", body);
}
...
return Ok();
}
Given the above endpoint (with no middleware in place), when it's run locally, body has the value of:
{"currentlySaved":false,"type":"album"}
However, when it is run remotely, it is empty.
Problem with your code is here:
//Copy a pointer to the original response body stream
var originalBodyStream = context.Response.Body;
You are assigning a reference to the response stream, and then you are overwriting(that stream) it with an empty memory stream here:
//...and use that for the temporary response body
context.Response.Body = responseBody;
So the reference in originalBodyStream variable, is now pointing to that empty stream as well. Other words - at this point the content of context.Response.Body is lost.
If you want to copy the request body stream to the other stream, use below:
await context.Response.Body.CopyToAsync(originalBodyStream);
Remember that streams are reference type so this:
var bodyVar = request.Body;
assigns only a reference to the stream(not the value!) - other words any modifications on the request.Body will be reflected on the bodyVar as it points to the same place in memory.
I have the following code to set content-md5 in my GET method request using HttpClient
httpClient.DefaultRequestHeaders.TryAddWithoutValidation("content-md5", "value");
I cannot use HttpRequestMessage content to set it because it's not a POST method. When using Postman it works like a charm but fails when using HttpClient.GetAsync.
Client request a hmac to the server as follows
{
"content_to_hash": "my content"
}
The server will give response like this
{
"content_md5": "88af7ceab9fdafb76xxxxx",
"date": "Sat, 02 May 2020 00:13:16 +0700",
"hmac_value": "WfHgFyT792IENmK8Mqz9LysmP8ftOP00qA="
}
Now I have to access a GET request using that hmac where it's the problem because I cannot set in httpClient GET request header.
Here's the image
From reading the HttpClient and related source code, there's no way you can get around this and add the header to the actual request object headers. There is an internal list of invalid headers, which includes any Content-* headers. It has to be on a content object.
Therefore, my suggest solution is to create your own content object:
public class NoContentMd5 : HttpContent
{
protected override Task SerializeToStreamAsync(Stream stream, TransportContext context)
{
return Task.CompletedTask;
}
protected override bool TryComputeLength(out long length)
{
length = 0;
return false;
}
public NoContentMd5(byte[] contentMd5)
{
this.Headers.ContentMD5 = contentMd5;
}
public NoContentMd5(string contentMd5)
{
this.Headers.TryAddWithoutValidation("Content-MD5", contentMd5);
}
}
This will add the Content-MD5 header with a value of your choosing, but the request won't contain a body.
The next problem you'll encounter is that you're trying to make a GET request with content, which isn't supported by the helper client.GetAsync(...) method. You'll have to make your own request object and use client.SendAsync(...) instead:
HttpClient client = new HttpClient();
var request = new HttpRequestMessage(HttpMethod.Get, "https://localhost/my/test/uri");
request.Content = new NoContentMd5("d41d8cd98f00b204e9800998ecf8427e ");
var result = await client.SendAsync(request);
Note that if you have your Content-MD5 hash as bytes, I've also added a constructor to NoContentMd5 for byte[] too.
The only potential issue with this is that it includes a Content-Length: 0 header. Hopefully that's OK with the API you're working with.
There's an alternative solution described in this answer to question with a similar issue. I'd argue against using it since is vulnerable to changes in the implementation details of HttpRequestHeaders (because it uses reflection, so if MS change the code, it might break) .
Aside from the fact that it's not considered good practice to send a body with GET request (see HTTP GET with request body), you can try this:
using (var content = new StringContent(string.Empty))
using (var request = new HttpRequestMessage
{
Method = HttpMethod.Get,
RequestUri = new Uri("http://localhost"),
Content = content
})
{
request.Headers.TryAddWithoutValidation("content-md5", "value");;
using (var response = await httpClient.SendAsync(request))
{
response.EnsureSuccessStatusCode();
}
}
UPDATE:
The proper way would be to set the ContentMD5 property on the HttpContentHeaders, for example:
content.Headers.ContentMD5 = Convert.FromBase64String(hashString);
But as you pointed out in the comments, trying to send content in a GET request causes an error.
I have a HttpResponse object as a result of HttpClient.SendAsync() call. The response has a chunked transfer encoding and results in 1.5 GB of data.
I want to pass this data through OWIN pipeline. To do this I need to convert it to a stream. Simplified code to do this is:
public async Task Invoke(IDictionary<string, object> environment)
{
var httpContent = GetHttpContent();
var responseStream = (Stream)environment["owin.ResponseBody"];
await httpContent.CopyToAsync(responseStream);
}
However, the last line results in copying the entire stream to the memory. And when I use wget to download the data directly from the backend server, it is downloaded successfully and shows a progress bar (although it doesn't know the overall size since it is chunked). But when I use wget to download data from my OWIN-hosted application it sticks on sending the request.
How should I stream this data through an OWIN pipeline to prevent copying it to memory?
EDIT
This is how I get the HttpResponse:
var client = new HttpClient(new HttpClientHandler());
// …and then:
using (var request = new HttpRequestMessage { RequestUri = uri, Method = HttpMethod.Get })
{
return client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).Result;
}
I assume this is in IIS? System.Web also buffers responses: https://msdn.microsoft.com/en-us/library/system.web.httpresponse.bufferoutput(v=vs.110).aspx
See server.DisableResponseBuffering in
https://katanaproject.codeplex.com/wikipage?title=OWIN%20Keys&referringTitle=Documentation