Develop Reference

Multiple HttpClients with proxies, trying to achieve maximum download speed

I need to use proxies to download a forum. The problem with my code is that it takes only 10% of my internet bandwidth. Also I have read that I need to use a single HttpClient instance, but with multiple proxies I don't know how to do it. Changing MaxDegreeOfParallelism doesn't change anything.
public static IAsyncEnumerable<IFetchResult> FetchInParallelAsync(
this IEnumerable<Url> urls, FetchContext context)
{
var fetchBlcock = new TransformBlock<Url, IFetchResult>(
transform: url => url.FetchAsync(context),
dataflowBlockOptions: new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = 128
}
);
foreach(var url in urls)
fetchBlcock.Post(url);
fetchBlcock.Complete();
var result = fetchBlcock.ToAsyncEnumerable();
return result;
}
Every call to FetchAsync will create or reuse a HttpClient with a WebProxy.
public static async Task<IFetchResult> FetchAsync(this Url url, FetchContext context)
{
var httpClient = context.ProxyPool.Rent();
var result = await url.FetchAsync(httpClient, context.Observer, context.Delay,
context.isReloadWithCookie);
context.ProxyPool.Return(httpClient);
return result;
}
public HttpClient Rent()
{
lock(_lockObject)
{
if (_uninitiliazedDatacenterProxiesAddresses.Count != 0)
{
var proxyAddress = _uninitiliazedDatacenterProxiesAddresses.Pop();
return proxyAddress.GetWebProxy(DataCenterProxiesCredentials).GetHttpClient();
}
return _proxiesQueue.Dequeue();
}
}
I am a novice at software developing, but the task of downloading using hundreds or thousands of proxies asynchronously looks like a trivial task that many should have been faced with and found a correct way to do it. So far I was unable to find any solutions to my problem on the internet. Any thoughts of how to achieve maximum download speed?

Let's take a look at what happens here:
var result = await url.FetchAsync(httpClient, context.Observer, context.Delay, context.isReloadWithCookie);
You are actually awaiting before you continue with the next item. That's why it is asynchronous and not parallel programming. async in Microsoft docs
The await keyword is where the magic happens. It yields control to the caller of the method that performed await, and it ultimately allows a UI to be responsive or a service to be elastic.
In essence, it frees the calling thread to do other stuff but the original calling code is suspended from executing, until the IO operation is done.
Now to your problem:
You can either use this excellent solution here: foreach async
You can use the Parallel library to execute your code in different threads.
Something like the following from Parallel for example
Parallel.For(0, urls.Count,
index => fetchBlcock.Post(urls[index])
});

HttpClient and long running Windows Services - What is the potential for thread deadlock and how to circumvent it

Note: I am running on .NET Framework 4.6.2
Background
I have a long running Windows Services that, once a minute, queues up a series of business related tasks that are ran on their own threads that are each awaited on by the main thread. There can only be one set of business related tasks running at the same time, as to disallow for race conditions. At certain points, each business task makes a series of asynchronous calls, in parallel, off to an external API via an HttpClient in a singleton wrapper. This results in anywhere between 20-100 API calls per second being made via HttpClient.
The issue
About twice a week for the past month, a deadlock issue (I believe) has been cropping up. Whenever it does happen, I have been restarting the Windows Service frequently as we can't afford to have the service going down for more than 20 minutes at a time without it causing serious business impact. From what I can see, any one of the business tasks will try sending a set of API calls and further API calls made using the HttpClient will fail to ever return, resulting in the task running up against a fairly generous timeout on the cancellation token that is created for each business task. I can see that the requests are reaching the await HttpClientInstance.SendAsync(request, cts.Token).ConfigureAwait(false) line, but do not advance past it.
For a additional clarification here, once the first business task begins deadlocking with HttpClient, any new threads attempting to send API requests using the HttpClient end up timing out. New business threads are being queued up, but they cannot utilize the instance of HttpClient at all.
Is this a deadlocking situation? If so, how do I avoid it?
Relevant Code
HttpClientWrapper
public static class HttpClientWrapper
{
private static HttpClientHandler _httpClientHandler;
//legacy class that is extension of DelegatingHandler. I don't believe we are using any part of
//it outside of the inner handler. This could probably be cleaned up a little more to be fair
private static TimeoutHandler _timeoutHandler;
private static readonly Lazy<HttpClient> _httpClient =
new Lazy<HttpClient>(() => new HttpClient(_timeoutHandler));
public static HttpClient HttpClientInstance => _httpClient.Value;
public static async Task<Response> CallAPI(string url, HttpMethod httpMethod, CancellationTokenSource cts, string requestObj = "")
{
//class that contains fields for logging purposes
var response = new Response();
string accessToken;
var content = new StringContent(requestObj, Encoding.UTF8, "application/json");
var request = new HttpRequestMessage(httpMethod, new Uri(url));
if (!string.IsNullOrWhiteSpace(requestObj))
{
request.Content = content;
}
HttpResponseMessage resp = null;
try
{
resp = await HttpClientInstance.SendAsync(request, cts.Token).ConfigureAwait(false);
}
catch (Exception ex)
{
if ((ex.InnerException is OperationCanceledException || ex.InnerException is TaskCanceledException) && !cts.IsCancellationRequested)
throw new TimeoutException();
throw;
}
response.ReturnedJson = await resp.Content.ReadAsStringAsync();
// non-relevant post-call variables being set for logging...
return response;
}
//called on start up of the Windows Service
public static void SetProxyUse(bool useProxy)
{
if (useProxy || !ServerEnv.IsOnServer)
{
_httpClientHandler = new HttpClientHandler
{
UseProxy = true,
Proxy = new WebProxy {Address = /* in-house proxy */},
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
}
else
{
_httpClientHandler = new HttpClientHandler
{
UseProxy = false,
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
}
_handler = new TimeoutHandler
{
DefaultTimeout = TimeSpan.FromSeconds(120),
InnerHandler = _httpClientHandler
};
}
}
Generalized function from a business class
For more context.
//Code for generating work parameters in each batch of work
...
foreach (var workBatch in batchesOfWork)
{
var tasks = workBatch.Select(async batch =>
workBatch.Result = await GetData(/* work related parms*/)
);
await Task.WhenAll(tasks);
}
...
GetData() function
//code for formating url
try
{
response = await HttpClientWrapper.CallAPI(formattedUrl, HttpMethod.Get, cts);
}
catch (TimeoutException)
{
//retry logic
}
...
//JSON deserialization, error handling, etc.....
Edit
I forgot to mention that this also set on start-up.
ServicePointManager
.FindServicePoint(/* base uri for the API that we are contacting*/)
.ConnectionLeaseTimeout = 60000; // 1 minute
ServicePointManager.DnsRefreshTimeout = 60000;

The above mentioned code example shows that a common instance of HttpClient is being used by all the running applications.
Microsoft documentation recommends that the HttpClient object be instantiated once per application, rather than per-use.
This recommendation is applicable for the requests within one application.
This is for the purpose of ensuring common connection settings for all requests made to specific destination API.
However, when there are multiple applications, then the recommended approach is to have one instance of HttpClient per application instance, in order to avoid the scenario of one application waiting for the other to finish.
Removing the static keyword for the HttpClientWrapper class and updating the code so that each application can have its own instance of HttpClient will resolve the reported problem.
More information:
https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?view=netcore-3.1

After taking #David Browne - Microsoft's advice in the comment section, I changed the default amount of connections from the default (2) to the API provider's rate limit for my organization (100) and that seems to have done the trick. It has been several days since I've installed the change to production, and it is humming along nicely.
Additionally, I slimmed down the HttpClientWrapper class I had to contain the CallAPI function and a default HttpClientHandler implementation with the proxy/decompression settings I have above. It doesn't override the default timer anymore, as my thought is is that I should just retry the API call if it takes more than the default 100 seconds.
To anyone stumbling upon this thread:
1) One HttpClient being used throughout the entirety of your application will be fine, no matter the amount of threads or API calls being done by it. Just make sure to increase the number of DefaultConnections via the ServicePointManager. You also DO NOT have to use the HttpClient in a using context. It will work just fine in a lazy singleton as I demonstrate above. Don't worry about disposing of the HttpClient in a long running service.
2) Use async-await throughout your application. It is worth the pay-off as it makes the application much more readable and allows your threads to be freed up as you are awaiting a response back from the API. This might seem obvious, but it isn't if you haven't used the async-await architecture in an application before.

Reusable HttpClient instance vs static variable (Heavily used in multiple threads)?

I currently have an instance variable of a HttpClient which is used in an instance method to download an image. The method is always called in a Task.Run().
This method can be called thousands of times in a minute and doesn't crash or produce any errors. But I just wanted to know what benefits, if any, there would be if I switched to a static HttpClient, apart from perhaps being more thread safe.
Here is some code for context:
HttpClient client = new HttpClient(); // Make this static?
// Always called in a Task.Run(() => DownloadImage(Obj));
public async void DownloadImage(Object obj)
{
FormUrlEncodedContent formContent = GetFormContent(Obj);
HttpResponseMessage Result = await client.PostAsync("URL", formContent).ConfigureAwait(false);
byte[] Data = Result.Content.ReadAsByteArrayAsync().Result;
StaticClass.Images[Obj.ID] = ImageSource.FromStream(() => new MemoryStream(Data));
formContent.Dispose();
Result.Dispose();
}

No
Since you are not using async version and you are calling ReadAsByteArrayAsync().Result you will most likely end up in deadlocks.
Recommended
Following is the most recommended way,
static HttpClient client = new HttpClient ();
public async Task DownloadImage(Object obj)
{
using(FormUrlEncodedContent formContent = GetFormContent(Obj)) {
using(HttpResponseMessage Result = await
client.PostAsync("URL", formContent)
.ConfigureAwait(false)){
byte[] Data = await Result.Content.ReadAsByteArrayAsync();
StaticClass.Images[Obj.ID] = ImageSource.FromStream(
() => new MemoryStream(Data));
}
}
}
Also don't call Dispose, use using block, Dispose will not be executed if an exception is thrown, using block will properly Dispose even in case of exception.
As long as, all disposables are wrapped in using and async await are used properly, HttpClient will perform best with single static instance.

HttpClient is intented by microsoft to be used as a static object. Even if the time to instatiate it every time is trivial, because other issues could arise.
his link contains a simple implementation :
https://learn.microsoft.com/en-us/aspnet/web-api/overview/advanced/calling-a-web-api-from-a-net-client .
Since you will be using the client in more more cases than simple getting an image I would suggest a class with static methods (async if you may) for get/post/put/delete where you would inserting method names and objects dynamically.

Make Async Operations prettier in .Net4.0

To preface, I'm 100% stuck using .Net 4.0.
I'm trying to consume a web service, and make the async calls a little less messy for the client to consume. I came up with:
protected void Get<T>(string uri, Action<T> callback)
{
var client = GetNewClient();
var request = new HttpRequestMessage(HttpMethod.Get, uri);
client.SendAsync(request)
.Completed(t =>
{
T resp = t.Result.Content.ReadAsAsync<T>().Result;
callback(resp);
})
.Errored(t =>
{
throw t.Exception;
});
}
Completed and Errored are just extension methods that wrap TaskContinuationOptions.OnlyOnRanToCompletion and OnlyOnFaulted, respectively.
This code works well when everything goes as expected. The problem is, if the task faults (i.e. web service down), the error doesn't make it back to the consumer. Also, the task is marked as Completed even if the response HttpStatusCode indicates an error (i.e. 404). I'd certainly like to handle those situations effectively, with this current implementation I have no way to do so (the response item is simply null).
Is there a way to raise an error back to the consumer here, or should I abandon this approach altogether?

Is there a way to raise an error back to the consumer here, or should I abandon this approach altogether?
Well, take a look at your signature:
protected void Get<T>(string uri, Action<T> callback);
When the client calls Get<T>, it starts the asynchronous operation and then returns. Obviously, if there is an exception later, there's no way to have the thread travel back in time and return from this method again.
Instead, you need to modify your callback. You can easily do this by adding a second callback delegate:
protected void Get<T>(string uri, Action<T> callback, Action<Exception> errorCallback);
Now one or the other of the callbacks will be called when the operation is complete. But what is this doing, really? Just re-introducing callbacks when you already have promises. So a superior solution would be (updated due to Servy's comment):
protected Task<T> Get<T>(string uri)
{
var client = GetNewClient();
var request = new HttpRequestMessage(HttpMethod.Get, uri);
return client.SendAsync(request)
.ContinueWith(t => t.Result.Content.ReadAsAsync<T>())
.Unwrap();
}
That said, I do think the best approach is to use Microsoft.Bcl.Async. Bundling KB2468871 into the installer is not too hard.

Decorate your method with async, call await on your get method:
protected async Task<T> Get<T>(string uri)
{
var client = GetNewClient();
var response = await client.GetAsync(uri);
// Throws an exception to the user if it was not a successful request.
response.EnsureSuccessStatusCode();
return await response.ReadAsAsync<T>();
}

Using async and await when external Input/output API provides its own callback delegates

I have a class library that (amongst other things) acts as a wrapper for an external client library for a web API.
My (simplified) code here takes a query string, generates a ReportUtilities object, uses this to download a report, then returns the report string back to the caller:
public string GetXmlReport(string queryString)
{
ReportUtilities reportUtil = new ReportUtilities(_user,queryString,"XML");
byte[] data = reportUtil.GetResponse().Download();
return Encoding.UTF8.GetString(data);
}
The issue is that this method is downloading data from a webservice using a syncronous method. I would like to make this available asyncronously in my library by adding a GetXmlReportAsync() method to my class.
Now, this would be straighforward if the ReportUtilities class provided a GenerateAndDownloadAsync() method returning Task<byte[]> but unfortunately it does not and it is in the external library so I am stuck with what it provides.
The ReportUtilities class does have a GetResponseAsync() method that returns void and provides a delegate for a a OnReadyCallback method together with an OnReadyCallback OnReady{get;set;} property.
I should add that .GetResponse() returns a ReportResponse object which does have a DownloadAsync() method but, again, this returns void rather than Task<byte[]>. Again ReportResponse comes with OnDownloadSuccessCallback delegate and OnDownloadSuccessCallback OnDownloadSuccess { get; set; } property.
It's almost as if the external library authors are 'rolling thier own' async API rather than using the one built into C#?
My question is: How can I implement a GetXmlReportAsync() method in my class to make the most efficient use of the asyncronous functions in the client library?
Obviously I could just do:
public async Task<string> GetXmlReportAsync(string queryString)
{
ReportUtilities reportUtil = new ReportUtilities(_user,queryString,"XML");
byte[] data = await Task.Run(() => { return reportUtil.GetResponse().Download(); });
return Encoding.UTF8.GetString(data);
}
but then the thread gets tied up with 2 syncronous input/output method calls to the external library: .GetResponse() and .Download() which surely isnt optimal?
Alternatively, I could imagine a situation where I just exposed a similar API to the external library in my own with clients having to provide callbacks for when thier reports are ready but I would much prefer to wrap this up into the more familiar async/await style API.
Am I trying to fit a square peg in a round hole or am I missing a neat way to wrap this into an async/await style API?

It's almost as if the external library authors are 'rolling thier own' async API rather than using the one built into C#?
A not uncommon situation for older libraries, particularly ones that were directly ported from other platforms/languages.
Am I trying to fit a square peg in a round hole or am I missing a neat way to wrap this into an async/await style API?
The pattern they're using is very similar to EAP, and there's a common pattern for converting EAP to TAP. You can do something similar with a few adjustments.
I recommend creating extension methods for the third-party library types that give you nice TAP endpoints, and then building your logic on top of that. That way, the TAP method doesn't mix concerns (translating asynchronous patterns, and doing business logic - i.e., converting to a string).
The ReportUtilities class does have a GetResponseAsync() method that returns void and provides a delegate for a a OnReadyCallback method together with an OnReadyCallback OnReady{get;set;} property.
Something like this, then:
public static Task<ReportResponse> GetResponseTaskAsync(this ReportUtilities #this)
{
var tcs = new TaskCompletionSource<ReportResponse>();
#this.OnReady = response =>
{
// TODO: check for errors, and call tcs.TrySetException if one is found.
tcs.TrySetResult(response);
};
#this.GetResponseAsync();
return tcs.Task;
}
Similarly for the next level:
public static Task<byte[]> DownloadTaskAsync(this ReportResponse #this)
{
var tcs = new TaskCompletionSource<byte[]>();
// TODO: how to get errors? Is there an OnDownloadFailed?
#this.OnDownloadSuccess = result =>
{
tcs.TrySetResult(result);
};
#this.DownloadAsync();
return tcs.Task;
}
Then your business logic can use the clean TAP endpoints:
public async Task<string> GetXmlReportAsync(string queryString)
{
ReportUtilities reportUtil = new ReportUtilities(_user, queryString, "XML");
var response = await reportUtil.GetResponseTaskAsync().ConfigureAwait(false);
var data = await response.DownloadTaskAsync().ConfigureAwait(false);
return Encoding.UTF8.GetString(data);
}

How can I implement a GetXmlReportAsync() method in my class to make
the most efficient use of the asyncronous functions in the client
library?
You could wrap the asynchoronous GetResponseAsync call with a TaskCompletionSource<string>. It would register the delegate once complete and set the completion of the task via SetResult. It would look something of this sort:
public Task<string> GetXmlReportAsync()
{
var tcs = new TaskCompletionSource<string>();
ReportUtilities reportUtil = new ReportUtilities(_user,queryString,"XML");
reportUtil.GetResponseAsync(callBack =>
{
// I persume this would be the callback invoked once the download is done
// Note, I am assuming sort sort of "Result" property provided by the callback,
// Change this to the actual property
byte[] data = callBack.Result;
tcs.SetResult(Encoding.UTF8.GetString(data));
});
return tcs.Task;
}