Develop Reference

Multiple HttpClients with proxies, trying to achieve maximum download speed

I need to use proxies to download a forum. The problem with my code is that it takes only 10% of my internet bandwidth. Also I have read that I need to use a single HttpClient instance, but with multiple proxies I don't know how to do it. Changing MaxDegreeOfParallelism doesn't change anything.
public static IAsyncEnumerable<IFetchResult> FetchInParallelAsync(
this IEnumerable<Url> urls, FetchContext context)
{
var fetchBlcock = new TransformBlock<Url, IFetchResult>(
transform: url => url.FetchAsync(context),
dataflowBlockOptions: new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = 128
}
);
foreach(var url in urls)
fetchBlcock.Post(url);
fetchBlcock.Complete();
var result = fetchBlcock.ToAsyncEnumerable();
return result;
}
Every call to FetchAsync will create or reuse a HttpClient with a WebProxy.
public static async Task<IFetchResult> FetchAsync(this Url url, FetchContext context)
{
var httpClient = context.ProxyPool.Rent();
var result = await url.FetchAsync(httpClient, context.Observer, context.Delay,
context.isReloadWithCookie);
context.ProxyPool.Return(httpClient);
return result;
}
public HttpClient Rent()
{
lock(_lockObject)
{
if (_uninitiliazedDatacenterProxiesAddresses.Count != 0)
{
var proxyAddress = _uninitiliazedDatacenterProxiesAddresses.Pop();
return proxyAddress.GetWebProxy(DataCenterProxiesCredentials).GetHttpClient();
}
return _proxiesQueue.Dequeue();
}
}
I am a novice at software developing, but the task of downloading using hundreds or thousands of proxies asynchronously looks like a trivial task that many should have been faced with and found a correct way to do it. So far I was unable to find any solutions to my problem on the internet. Any thoughts of how to achieve maximum download speed?

Let's take a look at what happens here:
var result = await url.FetchAsync(httpClient, context.Observer, context.Delay, context.isReloadWithCookie);
You are actually awaiting before you continue with the next item. That's why it is asynchronous and not parallel programming. async in Microsoft docs
The await keyword is where the magic happens. It yields control to the caller of the method that performed await, and it ultimately allows a UI to be responsive or a service to be elastic.
In essence, it frees the calling thread to do other stuff but the original calling code is suspended from executing, until the IO operation is done.
Now to your problem:
You can either use this excellent solution here: foreach async
You can use the Parallel library to execute your code in different threads.
Something like the following from Parallel for example
Parallel.For(0, urls.Count,
index => fetchBlcock.Post(urls[index])
});

Reading from multiple WebSockets with async/await

I'm writing a .NET Core Console App that needs to continuously read data from multiple WebSockets. My current approach is to create a new Task (via Task.Run) per WebSocket that runs an infinite while loop and blocks until it reads the data from the socket. However, since the data is pushed at a rather low frequency, the threads just block most of the time which seems quite inefficient.
From my understanding, the async/await pattern should be ideal for blocking I/O operations. However, I'm not sure how to apply it for my situation or even if async/await can improve this in any way - especially since it's a Console app.
I've put together a proof of concept (doing a HTTP GET instead of reading from WebSocket for simplicity). The only way I was able to achieve this was without actually awaiting. Code:
static void Main(string[] args)
{
Console.WriteLine($"ThreadId={ThreadId}: Main");
Task task = Task.Run(() => Process("https://duckduckgo.com", "https://stackoverflow.com/"));
// Do main work.
task.Wait();
}
private static void Process(params string[] urls)
{
Dictionary<string, Task<string>> tasks = urls.ToDictionary(x => x, x => (Task<string>)null);
HttpClient client = new HttpClient();
while (true)
{
foreach (string url in urls)
{
Task<string> task = tasks[url];
if (task == null || task.IsCompleted)
{
if (task != null)
{
string result = task.Result;
Console.WriteLine($"ThreadId={ThreadId}: Length={result.Length}");
}
tasks[url] = ReadString(client, url);
}
}
Thread.Yield();
}
}
private static async Task<string> ReadString(HttpClient client, string url)
{
var response = await client.GetAsync(url);
Console.WriteLine($"ThreadId={ThreadId}: Url={url}");
return await response.Content.ReadAsStringAsync();
}
private static int ThreadId => Thread.CurrentThread.ManagedThreadId;
This seems to be working and executing on various Worker Threads on the ThreadPool. However, this definitely doesn't seem as any typical async/await code which makes me think there has to be a better way.
Is there a more proper / more elegant way of doing this?

You've basically written a version of Task.WhenAny that uses a CPU loop to check for completed tasks rather than... whatever magic the framework method uses behind the scenes.
A more idiomatic version might look like this. (Although it might not - I feel like there should be an easier method of "re-run the completed task" than the reverse dictionary I've used here.)
static void Main(string[] args)
{
Console.WriteLine($"ThreadId={ThreadId}: Main");
// No need for Task.Run here.
var task = Process("https://duckduckgo.com", "https://stackoverflow.com/");
task.Wait();
}
private static async Task Process(params string[] urls)
{
// Set up initial dictionary mapping task (per URL) to the URL used.
HttpClient client = new HttpClient();
var tasks = urls.ToDictionary(u => client.GetAsync(u), u => u);
while (true)
{
// Wait for any task to complete, get its URL and remove it from the current tasks.
var firstCompletedTask = await Task.WhenAny(tasks.Keys);
var firstCompletedUrl = tasks[firstCompletedTask];
tasks.Remove(firstCompletedTask);
// Do work with completed task.
try
{
Console.WriteLine($"ThreadId={ThreadId}: URL={firstCompletedUrl}");
using (var response = await firstCompletedTask)
{
var content = await response.Content.ReadAsStringAsync();
Console.WriteLine($"ThreadId={ThreadId}: Length={content.Length}");
}
}
catch (Exception ex)
{
Console.WriteLine($"ThreadId={ThreadId}: Ex={ex}");
}
// Queue the task again.
tasks.Add(client.GetAsync(firstCompletedUrl), firstCompletedUrl);
}
}
private static int ThreadId => Thread.CurrentThread.ManagedThreadId;

I've accepted Rawling's answer - I believe it is correct for the exact scenario I described. However, with a bit of inverted logic, I ended up with something way simpler - leaving it in case anyone needs something like this:
static void Main(string[] args)
{
string[] urls = { "https://duckduckgo.com", "https://stackoverflow.com/" };
HttpClient client = new HttpClient();
var tasks = urls.Select(async url =>
{
while (true) await ReadString(client, url);
});
Task.WhenAll(tasks).Wait();
}
private static async Task<string> ReadString(HttpClient client, string url)
{
var response = await client.GetAsync(url);
string data = await response.Content.ReadAsStringAsync();
Console.WriteLine($"Fetched data from url={url}. Length={data.Length}");
return data;
}

Maybe better question is: do you really need thread per socket in this case? You should think of threads as system-wide resource and you should take this into consideration when spawning them, especially if you don't really know the number of threads that your application will be using. This is a good read: What's the maximum number of threads in Windows Server 2003?
Few years ago .NET team introduced Asynchronous sockets.
...The client is built with an asynchronous socket, so execution of
the client application is not suspended while the server returns a
response. The application sends a string to the server and then
displays the string returned by the server on the console.
Asynchronous Client Socket Example
There are a lot more examples out there showcasing this approach. While it is a bit more complicated and "low level" it let's you be in control.

Write C# async and "sync" code together in one method [duplicate]

This question already has answers here:
How to call asynchronous method from synchronous method in C#?
(17 answers)
Closed 6 years ago.
I have the below method:
public string RetrieveHolidayDatesFromSource() {
var result = this.RetrieveHolidayDatesFromSourceAsync();
/** Do stuff **/
var returnedResult = this.TransformResults(result.Result); /** Where result gets used **/
return returnedResult;
}
private async Task<string> RetrieveHolidayDatesFromSourceAsync() {
using (var httpClient = new HttpClient()) {
var json = await httpClient.GetStringAsync(SourceURI);
return json;
}
}
The above does not work and seems to not return any results properly. I am not sure where I am missing a statement to force the await of a result? I want the RetrieveHolidayDatesFromSource() method to return a string.
The below works fine but it is synchronous and I believe it can be improved upon? Note that the below is synchronous in which I would like to change to Asynchronous but am unable to wrap my head around for some reason.
public string RetrieveHolidayDatesFromSource() {
var result = this.RetrieveHolidayDatesFromSourceAsync();
/** Do Stuff **/
var returnedResult = this.TransformResults(result); /** This is where Result is actually used**/
return returnedResult;
}
private string RetrieveHolidayDatesFromSourceAsync() {
using (var httpClient = new HttpClient()) {
var json = httpClient.GetStringAsync(SourceURI);
return json.Result;
}
}
Am I missing something?
Note: For some reason, when I breakpoint the above Async Method, when it gets to the line var json = await httpClient.GetStringAsync(SourceURI) it just goes out of breakpoint and I can't go back into the method.

Am I missing something?
Yes. Asynchronous code - by its nature - implies that the current thread is not used while the operation is in progress. Synchronous code - by its nature - implies that the current thread is blocked while the operation is in progress. This is why calling asynchronous code from synchronous code literally doesn't even make sense. In fact, as I describe on my blog, a naive approach (using Result/Wait) can easily result in deadlocks.
The first thing to consider is: should my API be synchronous or asynchronous? If it deals with I/O (as in this example), it should be asynchronous. So, this would be a more appropriate design:
public async Task<string> RetrieveHolidayDatesFromSourceAsync() {
var result = await this.DoRetrieveHolidayDatesFromSourceAsync();
/** Do stuff **/
var returnedResult = this.TransformResults(result); /** Where result gets used **/
return returnedResult;
}
As I describe in my async best practices article, you should go "async all the way". If you don't, you won't get any benefit out of async anyway, so why bother?
But let's say that you're interested in eventually going async, but right now you can't change everything, you just want to change part of your app. That's a pretty common situation.
In that case, the proper approach is to expose both synchronous and asynchronous APIs. Eventually, after all the other code is upgraded, the synchronous APIs can be removed. I explore a variety of options for this kind of scenario in my article on brownfield async development; my personal favorite is the "bool parameter hack", which would look like this:
public string RetrieveHolidayDatesFromSource() {
return this.DoRetrieveHolidayDatesFromSourceAsync(sync: true).GetAwaiter().GetResult();
}
public Task<string> RetrieveHolidayDatesFromSourceAsync() {
return this.DoRetrieveHolidayDatesFromSourceAsync(sync: false);
}
private async Task<string> DoRetrieveHolidayDatesFromSourceAsync(bool sync) {
var result = await this.GetHolidayDatesAsync(sync);
/** Do stuff **/
var returnedResult = this.TransformResults(result);
return returnedResult;
}
private async Task<string> GetHolidayDatesAsync(bool sync) {
using (var client = new WebClient()) {
return sync
? client.DownloadString(SourceURI)
: await client.DownloadStringTaskAsync(SourceURI);
}
}
This approach avoids code duplication and also avoids any deadlock or reentrancy problems common with other "sync-over-async" antipattern solutions.
Note that I would still treat the resulting code as an "intermediate step" on the path to a properly-asynchronous API. In particular, the inner code had to fall back on WebClient (which supports both sync and async) instead of the preferred HttpClient (which only supports async). Once all the calling code is changed to use RetrieveHolidayDatesFromSourceAsync and not RetrieveHolidayDatesFromSource, then I'd revisit this and remove all the tech debt, changing it to use HttpClient and be async-only.

public string RetrieveHolidayDatesFromSource() {
var result = this.RetrieveHolidayDatesFromSourceAsync().Result;
/** Do stuff **/
var returnedResult = this.TransformResults(result.Result); /** Where result gets used **/
return returnedResult;
}
If you add .Result to the async call, it will execute and wait for the result to arrive, forcing it to be synchronous
UPDATE:
private static string stringTest()
{
return getStringAsync().Result;
}
private static async Task<string> getStringAsync()
{
return await Task.FromResult<string>("Hello");
}
static void Main(string[] args)
{
Console.WriteLine(stringTest());
}
To address the comment: This works without any problems.

Why use HttpClient for Synchronous Connection

I am building a class library to interact with an API. I need to call the API and process the XML response. I can see the benefits of using HttpClient for Asynchronous connectivity, but what I am doing is purely synchronous, so I cannot see any significant benefit over using HttpWebRequest.
If anyone can shed any light I would greatly appreciate it. I am not one for using new technology for the sake of it.

but what i am doing is purely synchronous
You could use HttpClient for synchronous requests just fine:
using (var client = new HttpClient())
{
var response = client.GetAsync("http://google.com").Result;
if (response.IsSuccessStatusCode)
{
var responseContent = response.Content;
// by calling .Result you are synchronously reading the result
string responseString = responseContent.ReadAsStringAsync().Result;
Console.WriteLine(responseString);
}
}
As far as why you should use HttpClient over WebRequest is concerned, well, HttpClient is the new kid on the block and could contain improvements over the old client.

I'd re-iterate Donny V. answer and Josh's
"The only reason I wouldn't use the async version is if I were trying
to support an older version of .NET that does not already have built
in async support."
(and upvote if I had the reputation.)
I can't remember the last time if ever, I was grateful of the fact HttpWebRequest threw exceptions for status codes >= 400. To get around these issues you need to catch the exceptions immediately, and map them to some non-exception response mechanisms in your code...boring, tedious and error prone in itself. Whether it be communicating with a database, or implementing a bespoke web proxy, its 'nearly' always desirable that the Http driver just tell your application code what was returned, and leave it up to you to decide how to behave.
Hence HttpClient is preferable.

For anyone coming across this now, .NET 5.0 has added a synchronous Send method to HttpClient. https://github.com/dotnet/runtime/pull/34948
The merits as to why where discussed at length here: https://github.com/dotnet/runtime/issues/32125
You can therefore use this instead of SendAsync. For example
public string GetValue()
{
var client = new HttpClient();
var webRequest = new HttpRequestMessage(HttpMethod.Post, "http://your-api.com")
{
Content = new StringContent("{ 'some': 'value' }", Encoding.UTF8, "application/json")
};
var response = client.Send(webRequest);
using var reader = new StreamReader(response.Content.ReadAsStream());
return reader.ReadToEnd();
}
This code is just a simplified example - it's not production ready.

public static class AsyncHelper
{
private static readonly TaskFactory _taskFactory = new
TaskFactory(CancellationToken.None,
TaskCreationOptions.None,
TaskContinuationOptions.None,
TaskScheduler.Default);
public static TResult RunSync<TResult>(Func<Task<TResult>> func)
=> _taskFactory
.StartNew(func)
.Unwrap()
.GetAwaiter()
.GetResult();
public static void RunSync(Func<Task> func)
=> _taskFactory
.StartNew(func)
.Unwrap()
.GetAwaiter()
.GetResult();
}
Then
AsyncHelper.RunSync(() => DoAsyncStuff());
if you use that class pass your async method as parameter you can call the async methods from sync methods in a safe way.
it's explained here :
https://cpratt.co/async-tips-tricks/

If you're building a class library, then perhaps the users of your library would like to use your library asynchronously. I think that's the biggest reason right there.
You also don't know how your library is going to be used. Perhaps the users will be processing lots and lots of requests, and doing so asynchronously will help it perform faster and more efficient.
If you can do so simply, try not to put the burden on the users of your library trying to make the flow asynchronous when you can take care of it for them.
The only reason I wouldn't use the async version is if I were trying to support an older version of .NET that does not already have built in async support.

In my case the accepted answer did not work. I was calling the API from an MVC application which had no async actions.
This is how I managed to make it work:
private static readonly TaskFactory _myTaskFactory = new TaskFactory(CancellationToken.None, TaskCreationOptions.None, TaskContinuationOptions.None, TaskScheduler.Default);
public static T RunSync<T>(Func<Task<T>> func)
{
CultureInfo cultureUi = CultureInfo.CurrentUICulture;
CultureInfo culture = CultureInfo.CurrentCulture;
return _myTaskFactory.StartNew<Task<T>>(delegate
{
Thread.CurrentThread.CurrentCulture = culture;
Thread.CurrentThread.CurrentUICulture = cultureUi;
return func();
}).Unwrap<T>().GetAwaiter().GetResult();
}
Then I called it like this:
Helper.RunSync(new Func<Task<ReturnTypeGoesHere>>(async () => await AsyncCallGoesHere(myparameter)));

In the current era, the shortest answer to this question is quite straightforward: Literally all other prior .NET options other than HttpClient are now deprecated/obsolete.

HttpClient GetStringAsync - It never comes back

New ASP.NET Web API HttpClient has been giving me some strange results. Here is my code:
class Program {
static async void Main(string[] args) {
var address = "http://localhost:3895/api/urls";
Console.WriteLine(await getStringAsync(address));
Console.ReadLine();
}
public static async Task<string> getStringAsync(string uri) {
var httpClient = new HttpClient();
return await httpClient.GetStringAsync(uri);
}
}
This never comes back and the console suddenly appears and disappears. When I change the code as below, it works as it is supposed to:
static void Main(string[] args) {
var address = "http://localhost:3895/api/urls";
Console.WriteLine(getString(address));
Console.ReadLine();
}
public static string getString(string uri) {
var httpClient = new HttpClient();
return httpClient.GetStringAsync(uri).Result;
}
Any idea on what would be the issue?

async on Main is disallowed in the VS11/.NET 4.5 compiler, so I'm assuming you're using the Async CTP. If using .NET 4.5 is at all an option, do make the switch.
That aside, the reason it doesn't work is because async, or more generally, tasks, rely on being able to signal some way for the remainder of the code to be executed. It works with .Result because the code runs synchronously, so the problem doesn't apply.
There is no built-in support for console applications, because they don't normally use message loops in the way that for example WinForms does, but you can look at Microsoft Visual Studio Async CTP\Samples\(C# Testing) Unit Testing\AsyncTestUtilities, notably GeneralThreadAffineContext.cs, to get a basic example that works in console applications too.