Develop Reference

Polly: Retry request with StreamContent and MemoryStream - no content on second try

I'm using Polly in combination with Microsoft.Extensions.Http.Polly to handle communication with an external API which has rate-limiting (N requests / second).I'm also using .NET 6.
The policy itself works fine for most requests, however it doesn't work properly for sending (stream) data. The API Client requires the usage of MemoryStream. When the Polly policy handles the requests and retries it, the stream data is not sent.
I verified this behavior stems from .NET itself with this minimal example:
using var fileStream = File.OpenRead(#"C:\myfile.pdf");
using var memoryStream = new MemoryStream();
await fileStream.CopyToAsync(memoryStream);
var response = await httpClient.SendAsync(
new HttpRequestMessage
{
// The endpoint will fail the request on the first request
RequestUri = new Uri("https://localhost:7186/api/test"),
Content = new StreamContent(memoryStream),
Method = HttpMethod.Post
}
);
Inspecting the request I see that Request.ContentLength is the length of the file on the first try. On the second try it's 0.
However if I change the example to use the FileStream directly it works:
using var fileStream = File.OpenRead(#"C:\myfile.pdf");
var response = await httpClient.SendAsync(
new HttpRequestMessage
{
// The endpoint will fail the request on the first request
RequestUri = new Uri("https://localhost:7186/api/test"),
Content = new StreamContent(fileStream ),
Method = HttpMethod.Post
}
);
And this is my Polly policy that I add to the chain of AddHttpClient.
public static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
return Policy
.HandleResult<HttpResponseMessage>(response =>
{
return response.StatusCode == System.Net.HttpStatusCode.Forbidden;
})
.WaitAndRetryAsync(4, (retry) => TimeSpan.FromSeconds(1));
}
My question:
How do I properly retry requests where StreamContent with a stream of type MemoryStream is involved, similar to the behavior of FileStream?
Edit for clarification:
I'm using an external API Client library (Egnyte) which accepts an instance of HttpClient
public class EgnyteClient {
public EgnyteClient(string apiKey, string domain, HttpClient? httpClient = null){
...
}
}
I pass an instance which I injected via the HttpContextFactory pattern. This instance uses the retry policy from above.
This is my method for writing a file using EgnyteClient
public async Task UploadFile(string path, MemoryStream stream){
// _egnyteClient is assigned in the constructor
await _egnyteClient.Files.CreateOrUpdateFile(path, stream);
}
This method call works (doesn't throw an exception) even when the API sometimes returns a 403 statucode because the internal HttpClient uses the Polly retry policy. HOWEVER the data isn't always properly transferred since it just works if it was the first attempt.

The root cause of your problem could be the following: once you have sent out a request then the MemoryStream's Position is at the end of the stream. So, any further requests needs to rewind the stream to be able to copy it again into the StreamContent (memoryStream.Position = 0;).
Here is how you can do that with retry:
private StreamContent GetContent(MemoryStream ms)
{
ms.Position = 0;
return new StreamContent(ms);
}
var response = await httpClient.SendAsync(
new HttpRequestMessage
{
RequestUri = new Uri("https://localhost:7186/api/test"),
Content = GetContent(memoryStream),
Method = HttpMethod.Post
}
);
This ensures that the memoryStream has been rewinded for each each retry attempt.
UPDATE #1
After receiving some clarification and digging in the source code of the Egnyte I think I know understand the problem scope better.
A 3rd party library receives an HttpClient instance which is decorated with a retry policy (related source code)
A MemoryStream is passed to a library which is passed forward as a StreamContent as a part of an HttpRequestMessage (related source code)
HRM is passed directly to the HttpClient and the response is wrapped into a ServiceResponse (related source code)
Based on the source code you can receive one of the followings:
An HttpRequestException thrown by the HttpClient
An EgnyteApiException or QPSLimitExceededException or RateLimitExceededException thrown by the ExceptionHelper
An EgnyteApiException thrown by the SendRequestAsync if there was a problem related to the deserialization
A ServiceResponse from SendRequestAsync
As far as I can see you can access the StatusCode only if you receive an HttpRequestException or an EgnyteApiException.
Because you can't rewind the MemoryStream whenever an HttpClient performs a retry I would suggest to decorate the UploadFile with retry. Inside the method you can always set the stream parameter's Position to 0.
public async Task UploadFile(string path, MemoryStream stream){
stream.Position = 0;
await _egnyteClient.Files.CreateOrUpdateFile(path, stream);
}
So rather than decorating the entire HttpClient you should decorate your UploadFile method with retry. Because of this you need to alter the policy definition to something like this:
public static IAsyncPolicy GetRetryPolicy()
=> Policy
.Handle<EgnyteApiException>(ex => ex.StatusCode == HttpStatusCode.Forbidden)
.Or<HttpRequestException>(ex => ex.StatusCode == HttpStatusCode.Forbidden)
.WaitAndRetryAsync(4, _ => TimeSpan.FromSeconds(1));
Maybe the Or builder clause is not needed because I haven't seen any EnsureSuccessStatusCode call anywhere, but for safety I would build the policy like that.

Streaming JSON using the dotnet Core 3 System.Text.Json Api

I'm developing a WPF web client using dotnet Core 3.x, and I'm utilising the System.Text.Json APIs. I am trying to use a Stream to pass the data between objects to minimise peak memory usage, as some large messages are being sent.
The wrapper method I've written thus far looks like the following:
public async Task<TResponse> PutItem<TItem, TResponse>(string path, TItem item)
{
HttpResponseMessage response;
await using (Stream stream = new MemoryStream())
{
await JsonSerializer.SerializeAsync(stream, item);
var requestContent = new StreamContent(stream);
requestContent.Headers.ContentType = new MediaTypeHeaderValue("application/json");
response = await _client.PutAsync(path, requestContent);
}
if (!response.IsSuccessStatusCode || response.Content == null)
{
return default;
}
string content = await response.Content.ReadAsStringAsync();
TResponse decodedResponse = JsonSerializer.Deserialize<TResponse>(content);
return decodedResponse;
}
However it does not appear to be writing any content when PUTting to the server.
I have seen users of earlier APIs utilise the PushStreamContent class, however this doesn't appear to exist in dotnet Core.
Any ideas would be very much appreciated, thanks!

I suppose the reason your current code does not work because you do not reset the stream position to 0 after serializing. However, because you create a MemoryStream you are still serializing to JSON in memory.
PushStreamContent is available by referencing the NuGet package Microsoft.AspNet.WebApi.Client.
Try something like:
var requestContent = new PushStreamContent(async (outputStream, httpContext, transportContext) =>
{
await JsonSerializer.SerializeAsync(outputStream, item);
});
requestContent.Headers.ContentType = new MediaTypeHeaderValue("application/json");
var response = await _client.PutAsync(path, requestContent);

RestSharp "Error getting response stream (ReadAsync): ReceiveFailure Value cannot be null. Parameter name: src"

Hello all am trying to do a login to my xamarin api using RestSharp, the API ought to return status code 200 OK if the authentication works and status code 415 if the authentication fails(wrong password) and other codes depending on what the case scenario, but instead i get a status code 0 on all other case asides when the authentication pass(status code 200 ok), the source code below is how i implement
//payload am sending to the api
RequestPayload res = new RequestPayload();
res.appid = appid;
res.data = data;
res.method = "Login";
//convert to json object
var MySerializedObject = JsonConvert.SerializeObject(res);
string APIUrl = ""http://142.168.20.15:8021/RouteTask";
//create client
RestClient client = new RestClient(APIUrl);
//create request
RestRequest request = new RestRequest(Method.POST);
// set request headeer
request.AddHeader("Content-Type", "application/x-www-form-urlencoded");
//request.AddJsonBody(MySerializedObject); --i have also tried this
request.AddParameter("application/json", MySerializedObject, ParameterType.RequestBody);
request.JsonSerializer.ContentType = "application/json; charset=utf-8";
request.AddParameter("RequestSource", "Web", "application/json", ParameterType.QueryString);
client.Timeout = 2000000;
var response = client.Execute(request); // where the issue appears
//RestResponse response = client.Execute(request); // i have tried this
//IRestResponse response = client.Execute(request); // i have tried this
if (response.IsSuccessful)
{
//use response data
}
on all scenerio it comes back with a StatusCode: 0, Content-Type: , Content-Length: 0) and errorMessage
"Error getting response stream (ReadAsync): ReceiveFailure Value
cannot be null. Parameter name: src"
screenshot below indicate when the api call fails
Response receieved when the authentication is valid

I was finally able to find a workaround for this. Bear with the long-winded response.
The tags mention Xamarin, which is what I am working in as well - specifically with iOS. I think it may actually be a bug with Mono, but I didn't take it that far to confirm.
The problem lies with the default way of copying the response buffer. In the RestSharp code, this is done by an extension method in MiscExtensions.cs called ReadAsBytes. It appears that with certain response buffers, the call to the Stream.Read method is failing. When this happens, the exception causes RestSharp to "shortcut" the rest of the processing on the response, hence the status code never gets filled in since it happens after the call to ReadAsBytes.
The good news is RestSharp does give a way to replace this call to ReadAsBytes with one of your own. This is done via the ResponseWriter property on the IRestRequest object. If it has a function defined, it will bypass the ReadAsBytes call and call the function you gave it instead. The problem is, this is defined as an Action and you don't get a copy of the full response object, so it's somewhat useless. Instead you have to use the AdvancedResponseWriter property. This one includes both the response object and the response stream. But you still have to set the ResponseWriter property or it won't bypass the default handler and you'll still get the error.
Ok, so how do you make this work? I ended up implementing it as a wrapper to RestClient so I wouldn't have to implement the code all over the place. Here's the basic setup:
public class MyRestClient : RestClient
{
public MyRestClient(string baseUrl) : base(baseUrl)
{ }
public override IRestResponse Execute(IRestRequest request)
{
request.ResponseWriter = s => { };
request.AdvancedResponseWriter = (input, response) => response.RawBytes = ReadAsBytes(input);
return base.Execute(request);
}
private static byte[] ReadAsBytes(Stream input)
{
var buffer = new byte[16 * 1024];
using (var ms = new MemoryStream())
{
int read;
try
{
while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
{ ms.Write(buffer, 0, read); }
return ms.ToArray();
}
catch (WebException ex)
{ return Encoding.UTF8.GetBytes(ex.Message); }
};
}
}
The ReadAsBytes method is actually just a copy/paste of the RestSharp ReadAsBytes method with the addition of a try/catch. If it fails, it returns the exception reason in to the response buffer. This may or may not be what you want, so modify as needed. You may also need to override other methods for Execute, but in my case this is the only one we're using so it was enough.
So far this seems to be doing the trick for me. Perhaps if someone got ambitious they could trace it all the way in to Mono to try and see what it doesn't like about the stream, but I don't have the time for it at the moment.
Good luck!

OK so after toying around with RestSharp for a bit, i realize just as #steve_In_Co mentioned earlier there were compatibility issues with MONO (we presume this is a bug) so i did it in a basic way using the .Net HTTP library and it works for me, so in case someone is still looking for a way out, find the working .net http implementation code below.
//payload am sending to the api
RequestPayload res = new RequestPayload();
res.appid = appid;
res.data = data;
res.method = "Login";
//convert to json object
var MySerializedObject = JsonConvert.SerializeObject(res);
string APIUrl = ""http://142.168.20.15:8021/RouteTask";
//create basic .net http client
HttpClient client = new HttpClient();
client.BaseAddress = new Uri(APIUrl);
// this was required in the header of my request,
// you may not need this, or you may need to adjust parameter
//("RequestSource","Web") or you own custom headers
client.DefaultRequestHeaders.Add("RequestSource", "Web");
// this class is custom, you can leave it out
connectionService = new ConnectionService();
//check for internet connection on users device before making the call
if (connectionService.IsConnected)
{
//make the call to the api
HttpResponseMessage response = await
client.PostAsJsonAsync(ApiConstants.APIDefault, res);
if (response.IsSuccessStatusCode)
{
string o = response.Content.ReadAsStringAsync().Result;
dynamic payload = JsonConvert.DeserializeObject(o);
string msg = payload["valMessage"];
resp.a = true;
resp.msg = payload["responseDescription"];
}
else
{
string o = response.Content.ReadAsStringAsync().Result;
dynamic payload = JsonConvert.DeserializeObject(o);
resp.a = false;
resp.msg = payload["response"];
}
}

WebApi Route fails if streamcontentis not empty

I'm trying to send an object using Web-APi 2 and protobuf-net. I keep getting a 404 error so I assume something goes wrong with the routing?
When I comment out the serialize line
ProtoBuf.Serializer.Serialize(memstream, package);
(so the memory stream stays empty) the routing works and RecievePackage is called. And the package parameter is empty. (the package object itsself is not NULL but all it's properties are.)
Whenever the memory stream is not empty I get a 404 error. What am I doing wrong here?
Receive code:
[Route("Receive")]
[HttpPost]
public async Task<IHttpActionResult> RecievePackage(PackageModel package)
{
id = await SavePackage(package);
return Created("", id);
}
Send code:
private async Task SyncUpAsync(string apiUrl, PackageModel package)
{
using (HttpClient client = new HttpClient())
using (var memstream = new MemoryStream())
{
var uri = new Uri(new Uri(ApiUrl), apiUrl);
ProtoBuf.Serializer.Serialize(memstream, package);
memstream.Position = 0;
var content = new StreamContent(memstream);
content.Headers.Add("SyncApiToken", ApiKey);
content.Headers.Add("Content-Type", "application/x-protobuf");
var response = await client.PostAsync(uri, content);
}
}

Reading the response stream of a WebException

I have the following method which is basically authenticating users with their Facebook credentials. For some reason, I am getting a WebException when trying to process the authorization key (code). So I tried to read the response stream in order to know what is going on but I keep getting errors while reading the stream. Here's my code:
private void OnAuthCallback(HttpContextWrapper context, WebServerClient client)
{
try
{
IAuthorizationState authorizationState = client.ProcessUserAuthorization(context.Request);
AccessToken accessToken = AccessTokenSerializer.Deserialize(authorizationState.AccessToken);
String username = accessToken.User;
context.Items[USERNAME] = username;
}
catch (ProtocolException e)
{
if (e.InnerException != null)
{
String message = e.InnerException.Message;
if (e.InnerException is WebException)
{
WebException exception = (WebException)e.InnerException;
var responseStream = exception.Response.GetResponseStream();
responseStream.Seek(0, SeekOrigin.Begin);
using (StreamReader sr = new StreamReader(responseStream))
{
message = sr.ReadToEnd();
}
}
EventLog.WriteEntry("OAuth Client", message);
}
}
}
If I remove the responseStream.Seek(0, SeekOrigin.Begin); line, it gives me an ArgumentException with a message that says the stream was not readable. And with this line in place, it tells me that I cannot manipulate a stream that was already closed. How was this stream closed? And why can't I read from it?

You can read the body of the response. I've found the solution in this answer.
You should cast the stream to MemoryStream and use ToArray method, then use Encoding.UTF8.GetString to get text.
private void OnAuthCallback(HttpContextWrapper context, WebServerClient client)
{
try
{
IAuthorizationState authorizationState = client.ProcessUserAuthorization(context.Request);
AccessToken accessToken = AccessTokenSerializer.Deserialize(authorizationState.AccessToken);
String username = accessToken.User;
context.Items[USERNAME] = username;
}
catch (ProtocolException e)
{
if (e.InnerException != null)
{
String message = e.InnerException.Message;
if (e.InnerException is WebException)
{
WebException exception = (WebException)e.InnerException;
message = ExtractResponseString(webException);
}
EventLog.WriteEntry("OAuth Client", message);
}
}
}
public static string ExtractResponseString(WebException webException)
{
if (webException == null || webException.Response == null)
return null;
var responseStream =
webException.Response.GetResponseStream() as MemoryStream;
if (responseStream == null)
return null;
var responseBytes = responseStream.ToArray();
var responseString = Encoding.UTF8.GetString(responseBytes);
return responseString;
}

It looks like you are using DotNetOpenAuth. Unfortunately I think this library closes the response stream of the WebException before wrapping it in the ProtocolException that your code receives. You can enable logging at the DEBUG level to have the response dumped to a log file, but I don't think you'll find any way to access it from your code.
One of the DotNetOpenAuth devs describes the situation here:
DotNetOpenAuth closes the HTTP response stream while handling and throwing the wrapped exception because if DNOA didn't close the stream, your open streams allotment would fill up and then your app would start hanging. DNOA does write the response stream's content to its log if I recall correctly.
Things may have changed in the latest (unreleased) version of DNOA (5.0), as the part of the code that is causing the problem in the current release version has been removed.

a WebException can have different causes and the server isn't always required to return a body under certain (error-)circumstance thus no response.stream.
Have a look at the returned status-code first
Another tool to help you investigate what's going is Fiddler