I'm working on an application I inherited that makes asynchronous calls to an API. The application sends about 60 asynchronous requests to the API, and then retrieves them as they are ready. The API returns the results in the form of a zip archive object. It was using the following (abbreviated) code to retrieve results from the API, but this kept throwing intermittent System.Threading.Tasks.TaskCanceledException errors
HttpResponseMessage response = client.SendAsync(requestMessage).Result;
Stream responseStream = response.Content.ReadAsStreamAsync().Result;
responseStream.Seek(0, 0);
za = new ZipArchive(responseStream, ZipArchiveMode.Read);
So I attempted to fix this by using await, and implemented the following methods, but I'm still getting the same errors. I can check the status of my API requests through a website, so I know they're not being canceled by the API. The requests that fail, fail in less than 5 minutes, so I know it's also not because the timeout value is too low on the HTTPClient. This is my first crack at asynchronous programming so if anyone can help with this, it would be greatly appreciated. Thanks.
public async Task<ZipArchive> GetExport(SourceTable sourceTable)
{
ZipArchive zipArchive = null;
switch (GetResultStatus(sourceTable))
{
case null:
break;
case "Completed":
{
zipArchive = await RetrieveResult(sourceTable);
}
break;
}
return zipArchive;
}
private async Task<ZipArchive> RetrieveResult(SourceTable sourceTable)
{
Export export = sourceTable.exports[0];
ZipArchive za = await RetrieveResultAsync(export);
return za;
}
private async Task<ZipArchive> RetrieveResultAsync(Export export)
{
ZipArchive za = null;
var credentials = new NetworkCredential(userName, password);
HttpClientHandler handler = new HttpClientHandler { Credentials = credentials };
HttpClient client = new HttpClient(handler);
client.Timeout.Add(new TimeSpan(0, 5, 0)); //5 minutes
HttpResponseMessage response = await client.GetAsync(restURL + "file/" + export.FileId);
response.EnsureSuccessStatusCode();
Stream responseStream = await response.Content.ReadAsStreamAsync();
responseStream.Seek(0, 0);
za = new ZipArchive(responseStream, ZipArchiveMode.Read);
return za;
}
UPDATE: After adding some more logging to this code I found out that it was indeed a timeout issue, and that I wasn't setting the timeout value correctly. When setting the value like below, it resolved the issues (of course with setting a higher timeout value than the default)
var credentials = new NetworkCredential(userName, password);
HttpClientHandler handler = new HttpClientHandler { Credentials = credentials };
HttpClient client = new HttpClient(handler);
client.Timeout = TimeSpan.FromMinutes(httpClientTimeout);
Related
I am looking to optimize my code…
I have number of API with early same structure for client and handler.
But I have some questions about disposing.
I have read the using statement automatically dispose resources (here HttpClient and HttpClientHandler).
Could I rewrite my code here:
public static async Task<IEnumerable<T>> deletePostsAsync<T>(IEnumerable<string> urls) where T : BaseReturnValues
{
var httpClientHandler = new HttpClientHandler
{
Proxy = new WebProxy(proxy, true),
UseProxy = IsProxySelected
};
using (var client = new HttpClient(httpClientHandler))
{
client.BaseAddress = new Uri(URI);
client.DefaultRequestHeaders.Clear();
client.DefaultRequestHeaders.Add(Head.key, Head.apikey);
var tasks = urls.Select(async url => await DeleteAsync<T>(client, url).ConfigureAwait(false));
var result = await Task.WhenAll(tasks).ConfigureAwait(false);
return result!;
}
async Task<U> DeleteAsync<U>(HttpClient client, string url) where U : BaseReturnValues
{
var statusCode = -1;
var json = "_";
var isSuccess = false;
try
{
using (HttpResponseMessage response = await client.DeleteAsync(url).ConfigureAwait(false))
{
statusCode = (Int32)response.StatusCode;
json = await response.Content.ReadAsStringAsync();
isSuccess = response.IsSuccessStatusCode;
}
}
catch (Exception ex)
{
//something to catch
}
:
return record;
}
}
To this piece of code without problem? Disposing resource is always done?
public static async Task<IEnumerable<T>> deletePostsAsync<T>(IEnumerable<string> urls) where T : BaseReturnValues
{
using (var client = SetClientSettings())
{
var tasks = urls.Select(async url => await DeleteAsync<T>(client, url).ConfigureAwait(false));
var result = await Task.WhenAll(tasks).ConfigureAwait(false);
return result!;
}
async Task<U> DeleteAsync<U>(HttpClient client, string url) where U : BaseReturnValues
{
:
:
return record;
}
}
public static Httpclient SetClientSettings()
{
var httpClientHandler = new HttpClientHandler
{
Proxy = new WebProxy(proxy, true),
UseProxy = IsProxySelected
};
var client = new HttpClient(httpClientHandler);
client.BaseAddress = new Uri(URI);
client.DefaultRequestHeaders.Clear();
client.DefaultRequestHeaders.Add(Head.key, Head.apikey);
return client;
}
So I have created a method SetClientSettings and this method create the client, the clienthandler, add some headers to client and return client.
so
var httpClientHandler = new HttpClientHandler
{
Proxy = new WebProxy(proxy, true),
UseProxy = IsProxySelected
};
using (var client = new HttpClient(httpClientHandler))
{
client.BaseAddress = new Uri(URI);
client.DefaultRequestHeaders.Clear();
client.DefaultRequestHeaders.Add(Head.key, Head.apikey);
var tasks = urls.Select(async url => await DeleteAsync<T>(client, url).ConfigureAwait(false));
var result = await Task.WhenAll(tasks).ConfigureAwait(false);
return result!;
}
is really equivalent to??:
using (var client = SetClientSettings())
{
var tasks = urls.Select(async url => await DeleteAsync<T>(client, url).ConfigureAwait(false));
var result = await Task.WhenAll(tasks).ConfigureAwait(false);
return result!;
}
Your two pieces of code are not equivalent. There is a small difference, that with the first snippet you will dispose the client even when the following lines (e.g. client.BaseAddress = new Uri(URI); or client.DefaultRequestHeaders.Clear();) fail. Which is better. To achieve this with SetClientSettings you would need to wrap everything into try except and .Dispose() on exception.
I have read the using statement automatically dispose resources (here HttpClient and HttpClientHandler).
The using statement turns this:
using (var instance = something)
{
// body
}
into this
var instance = something;
try
{
// body
}
finally
{
if (instance != null)
{
instance.Dispose();
}
}
That's all it does. It is a syntactic sugar.
And so in your particular case the using statement will ensure that .Dispose() is called on HttpClient, regardless of whether exception is thrown or not.
Now, do we have to dispose HttpClient? Well, they say we have to, there's no reason not to believe it. In reality the HttpClient holds sockets under the hood, which have to be closed manually when done with. And so, yes, you should always dispose HttpClient when done with.
That being said, the best thing you can do is to have a singleton HttpClient for the duration of your app, and reuse it. You can tweak it to your needs (e.g. configure it to use pooled connections) for maximal efficiency. In such scenario you don't dispose it at all.
Note: you don't have to worry about disposing HttpClientHandler. By default HttpClient will dispose it when it is disposed itself. This behaviour can be modified by using different constructor.
Yes, the code you wrote is equivalent to each other. There's an option in HttpClient contructor to not to dispose message handler - by default, it's set to true. Anyway, as already suggested, you don't have to dispose HTTP client at all. There are reasons for that.
There's alot of nice articles about best practices of using HttpClient.
Try to search for IHttpClientFactory.
I've been trying for a day to make this work synchronously, not async.
Here is the code that works:
HttpClientHandler handler = new HttpClientHandler();
handler.UseDefaultCredentials = false;
handler.Credentials = new NetworkCredential(username, password);
HttpClient clientPageFirst = new HttpClient(handler);
HttpResponseMessage responsePageFirst = await clientPageFirst.GetAsync(fullURL);
HttpContent contentPageFirst = responsePageFirst.Content;
string resultPageFirst = await contentPageFirst.ReadAsStringAsync();
Console.WriteLine(resultPageFirst);
It's for a C# console app, and there is another call there to another platform's API which works synchronously, but it uses tokens in the header to validate, not a network credential like this one (calling a local on premise CRM URL).
Can someone please help me change the
HttpResponseMessage responsePageFirst = await clientPageFirst.GetAsync(fullURL);
line so it is synchronous?
T.I.A.
Try this
HttpResponseMessage responsePageFirst = clientPageFirst.GetAsync("fullURL").Result;
I am working on a Hashicorp Vault management .net-core 3.1.3 console application written in C#. I have been tasked with creating a RabbitMQ user on an MQ server from the console app utilizing the RabbitMQ restful API. I have zero experience with this API. I have been reading the documentation but still don't have a clue as to how to begin.
I have limited experience with APIs in general and have never tried to do anything like this from a console app.
Any guidance or example code would be greatly appreciated.
Cheers.
Matt
You'll need the RabbitMQ Management HTTP API, the docs for which are here. Specifically you'll want to PUT a user on the /api/users/name endpoint.
There are many ways to make an HTTP request in c#, the simplest is probably the WebRequest class as documented here. You'll need to set the method to PUT, write your json payload to the request and set your rabbitmq credentials for the request.
Thanks for the clue-bat Adam. Here is where I ended up, and works well.
try
{
// Set MQ server credentials
NetworkCredential networkCredential = new NetworkCredential("mqUserName", "mqPassword");
// Instantiate HttpClientHandler, passing in the NetworkCredential
HttpClientHandler httpClientHandler = new HttpClientHandler { Credentials = networkCredential };
// Instantiate HttpClient passing in the HttpClientHandler
using HttpClient httpClient = new HttpClient(httpClientHandler);
// Get the response from the API endpoint.
HttpResponseMessage httpResponseMessage = await httpClient.GetAsync("http://YourServer:AndPort/api/users/");
// Get the response content.
HttpContent httpContent = httpResponseMessage.Content;
// Get the stream of the content.
using StreamReader streamReader = new StreamReader(await httpContent.ReadAsStreamAsync());
// Get the output string.
string returnedJsonString = await streamReader.ReadToEndAsync();
// Instantiate a list to loop through.
List<string> mqAccountNames = new List<string>();
if (returnedJsonString != "")
{
// Deserialize into object
dynamic dynamicJson = JsonConvert.DeserializeObject(returnedJsonString);
if (dynamicJson != null)
{
foreach (dynamic item in dynamicJson)
{
mqAccountNames.Add(item.name.ToString());
}
}
}
bool accountExists = false;
foreach (string mqAccountName in mqAccountNames)
{
if (mqAccountName == userName)
{
accountExists = true;
}
}
switch (accountExists)
{
case true:
Console.WriteLine("This user already exists on the MQ server.");
break;
case false:
// Create the new user on the MQ Server
Console.WriteLine("This user will be created on the MQ server.");
string uri = $"http://YourServer:AndPort/api/users/{userName}";
MqUser mqUser = new MqUser
{
password = password,
tags = "administrator"
};
string info = JsonConvert.SerializeObject(mqUser);
StringContent content = new StringContent(info, Encoding.UTF8, "application/json");
httpResponseMessage = await httpClient.PutAsync(uri, content);
if (!httpResponseMessage.IsSuccessStatusCode)
{
Console.WriteLine("There was an error creating the MQ user account.");
Thread.Sleep(2500);
return false;
}
uri = $"http://YourServer:AndPort/api/permissions/%2F/{userName}";
MqPermissions mqPermissions = new MqPermissions
{
configure = ".*",
write = ".*",
read = ".*"
};
info = JsonConvert.SerializeObject(mqPermissions);
content = new StringContent(info, Encoding.UTF8, "application/json");
httpResponseMessage = await httpClient.PutAsync(uri, content);
if (!httpResponseMessage.IsSuccessStatusCode)
{
Console.WriteLine("There was an error creating the permissions on the MQ user account.");
Thread.Sleep(2500);
return false;
}
break;
}
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
I created simple MqUser and MqPermissions classes so I could just JsonConvert.SerializeObject to pass the info.
Another weird thing was that my company chose to name the MQ Virtual Host as "/".
This had not been an issue up to this point as we had never tried to use the API before.
Since the / character is expected in a uri, this was a hangup, but I tried encoding it as %2F and it works just fine.
I'm trying to pass credentials back to a web service using HttpClient.
However, I keep getting an Unauthorized request.
However, when I try using a WebRequest it authenticates?
HttpClient:
var handler = new NativeMessageHandler
{
UseDefaultCredentials = true,
Credentials = credential
};
var client = new HttpClient(handler);
var content = _httpClientHelper.Serialize(data);
var response = await _client.PostAsync($"{_baseurl}/api/foos/List", content);
WebRequest:
HttpResponseMessage response = null;
try
{
var data = JsonConvert.SerializeObject(new
{
ViewTitle = "New",
PageCount = 60
});
var content = _httpClientHelper.Serialize(data);
using (var client = new WebClient { UseDefaultCredentials = true, Credentials = credentials })
{
client.Headers.Add(HttpRequestHeader.ContentType, "application/json; charset=utf-8");
client.UploadData("$"{baseurl}/api/foos/List", "POST", Encoding.UTF8.GetBytes(content));
}
I cannot figure out why one works and the other does not.
Any help or insight on this would be greatly appreciated
As noted here and here this behavior of HttpClient could be because of how HttpClientHandler is implemented.
"[..] the StartRequest method is executed in new thread with the credentials of the asp.net process (not the credentials of the impersonated user) [..]"
You might be seeing the difference in behavior of HttpClient and WebClient because
"HttpClient creates new threads
via the Task Factory. WebClient on the other hand, runs synchronously
on the same thread thereby forwarding its
credentials (i.e. the credentials of the impersonated user) ."
I upload my files in azure data lake. I try to download that file through asp.net mvc application.I have adl path for that file. I can download below 150 MB files. But i can't download the more then 150 MB files. Time out error came.
My Code in the bellow...
public ActionResult Download(string adlpath)
{
String header = adlpath;
Console.WriteLine(header);
string[] splitedStr = header.Split('/');
var path = GenerateDownloadPaths(adlpath);
string filename = path["fileName"];
HttpResponseMessage val = DataDownloadFile(path["fileSrcPath"]);
byte[] filedata = val.Content.ReadAsByteArrayAsync().Result;
string contentType = MimeMapping.GetMimeMapping(filename);
var cd = new System.Net.Mime.ContentDisposition
{
FileName = filename,
Inline = true,
};
Response.AppendHeader("Content-Disposition", cd.ToString());
return File(filedata, contentType);
}
public HttpResponseMessage DataDownloadFile(string srcFilePath)
{
string DownloadUrl = "https://{0}.azuredatalakestore.net/webhdfs/v1/{1}?op=OPEN&read=true";
var fullurl = string.Format(DownloadUrl, _datalakeAccountName, srcFilePath);
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", _accesstoken.access_token);
using (var formData = new MultipartFormDataContent())
{
resp = client.GetAsync(fullurl).Result;
}
}
return resp;
}
Image :
You should modify your code to use async and await. Your implementation blocks while retrieving the file and that is probably what times out:
public async Task<HttpResponseMessage> DataDownloadFile(string srcFilePath)
{
string DownloadUrl = "https://{0}.azuredatalakestore.net/webhdfs/v1/{1}?op=OPEN&read=true";
var fullurl = string.Format(DownloadUrl, _datalakeAccountName, srcFilePath);
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", _accesstoken.access_token);
using (var formData = new MultipartFormDataContent())
{
resp = await client.GetAsync(fullurl);
}
}
return resp;
}
The return value of the method is changed to Task<HttpResponseMessage> and the async modifier is added.
Calling client.GetAsync is changed to use await instead of blocking by retrieving the Result property.
Your code may still timeout. I believe that there is a configurable limit on how long a request can take before it is aborted and if you still get a timeout you should investigate this.
Per my understanding, you could try to increase the HttpClient.Timeout (100 seconds by default) for your HttpClient instance.
HttpClient.Timeout
Gets or sets the timespan to wait before the request times out.
The default value is 100,000 milliseconds (100 seconds).
Moreover, if you host your application via Azure Web App, you may encounter an idle timeout setting of 4 minutes from Azure Load Balancer. You could change the idle timeout setting in Azure VM and Azure Cloud Service. Details you could follow here.