Should I utilize multiple HttpClients for bulk async GET requests? - c#

I have a scenario where I need to make a large number of GET requests in as little time as possible (think around 1000).
I know generally it's best to keep a single client and reuse it as much as possible:
// Create Single HTTP Client
HttpClient client = new HttpClient();
// Create all tasks
for (int x = 0; x < 1000; x++)
{
tasks.Add(ProcessURLAsync($"https://someapi.com/request/{x}", client, x));
}
// wait for all tasks to complete.
Task.WaitAll(tasks.ToArray());
...
static async Task<string> ProcessURLAsync(string url, HttpClient client, int x)
{
var response = await client.GetStringAsync(url);
ParseResponse(response.Result, x);
return response;
}
But doing so takes approximately 70 seconds for all requests to complete.
On the other hand, If I create multiple clients beforehand and distribute the requests across them, the it takes about 3 seconds to complete:
// Create arbitrary number of clients
while (clients.Count < maxClients)
{
clients.Add(new HttpClient());
}
// Create all tasks
for (int x = 0; x < 1000; x++)
{
tasks.Add(ProcessURLAsync(
$"https://someapi.com/request/{x}", clients[x % maxClients], x));
}
// Same same code as above
Due to the nature of the data requested, I need to either keep the results sequential or pass along the index associated with the request.
Assuming the API cannot be changed to better format the requested data, and the all requests must complete before moving on, is this solution wise or am I missing a smarter alternative?
(For the sake of brevity I've used an arbitrary number of HttpClient whereas I would create a pool of HttpClient that releases a client once it receives a response and only create a new one when none are free)

I would suggest two main changes.
Remove the await so that multiple downloads can occur at the same
time.
Set DefaultConnectionLimit to a larger number (e.g. 50).

Related

C# parallel HTTP GET request "Too many requests" error

I'm trying to make several GET requests to an API, in parallel, but I'm getting an error ("Too many requests") when trying to do large volumes of requests (1600 items).
The following is a snippet of the code.
Call:
var metadataItemList = await GetAssetMetadataBulk(unitHashList_Unique);
Method:
private static async Task<List<MetadataModel>> GetAssetMetadataBulk(List<string> assetHashes)
{
List<MetadataModel> resultsList = new();
int batchSize = 100;
int batches = (int)Math.Ceiling((double)assetHashes.Count() / batchSize);
for (int i = 0; i < batches; i++)
{
var currentAssets = assetHashes.Skip(i * batchSize).Take(batchSize);
var tasks = currentAssets.Select(asset => EndpointProcessor<MetadataModel>.LoadAddress($"assets/{asset}"));
resultsList.AddRange(await Task.WhenAll(tasks));
}
return resultsList;
}
The method runs tasks in parallel in batches of 100, it works fine for small volumes of requests (<~300), but for greater amounts (~1000+), I get the aforementioned "Too many requests" error.
I tried stepping through the code, and to my surprise, it worked when I manually stepped through it. But I need it to work automatically.
Is there any way to slow down requests, or a better way to somehow circumvent the error whilst maintaining relatively good performance?
The request does not return a "Retry-After" header, and I also don't know how I'd implement this in C#. Any input on what code to edit, or direction to a doc is much appreciated!
The following is the Class I'm using to send HTTP requests:
class EndpointProcessor<T>
{
public static async Task<T> LoadAddress(string url)
{
using (HttpResponseMessage response = await ApiHelper.apiClient.GetAsync(url))
{
if (response.IsSuccessStatusCode)
{
T result = await response.Content.ReadAsAsync<T>();
return result;
}
else
{
//Console.WriteLine("Error: {0} ({1})\nTrailingHeaders: {2}\n", response.StatusCode, response.ReasonPhrase, response.TrailingHeaders);
throw new Exception(response.ReasonPhrase);
}
}
}
}
You can use a semaphore as a limiter for currently active threads. Add a field of Semaphore type to your API client and initialize it with a maximum count and an initial count of, say, 250 or what you determine as a safe maximum number of running requests. In your method ApiHelper.apiClient.GetAsync(), before making the real connection, try to acquire the semaphore, then release it after completing/failing the download. This will allow you enforce a maximum number of concurrently running requests.

Asp.net core Web API parallel call , batch processing with some data lost

Created one web API in asp.net core 3.x which is responsible to save data in to Azure service bus queue and that data we will process for reporting.
API load is too high so we decided to save data in-memory for each request. Once data limit increased up to certain limit (>50 count) next request (51) will get all data from memory and save in to service bus in one go and clear the memory cache.
for sequential request all logic works fine but when load coming in parallel few data lost and i think it is because of one batch request is taking some time and after that all data problem start.
I did some research and found article and used SemaphoreSlim. It's working fine but is that good approach? As you see in below code I am blocking each request but actually I want to lock when I am processing the batch. I tried to move the lock inside if condition but it was not working.
https://medium.com/swlh/async-lock-mechanism-on-asynchronous-programing-d43f15ad0b3
using (await lockThread.LockAsync())
{
var topVisitedTiles = _service.GetFromCache(CacheKey);
if (topVisitedTiles?.Count >= 50)
{
topVisitedTiles?.Add(link);
await _service.AddNewQuickLinkAsync(topVisitedTiles);
_service.SetToCache(CacheKey, new List<TopVisitedTilesItem>());
return Ok(link.Title);
}
topVisitedTiles?.Add(link);
_service.SetToCache(CacheKey, topVisitedTiles);
}
return Ok(link.Title);
I got something from research that concurrentbag and blockingcollection help but I am not aware how can i use in my case. Your small direction will help me.
You can use Task Parallel Library if you don't want to deep dive into parallel implementations of bags or queues.
In your case something like this can be used
// Define a buffer block with size = 10
var batchBlock = new BatchBlock<string>(10);
// Define an ActionBlock that processes batches received from BatchBlock
var processingBlock = new ActionBlock<string[]>((messages) =>
{
Console.WriteLine("-------------");
Console.WriteLine($"Number of messages: {messages.Length}");
Console.WriteLine($"Messages: {string.Join(", ", messages)}");
});
// Link processing block to batchBloack.
batchBlock.LinkTo(processingBlock);
batchBlock.Completion.ContinueWith((t) =>
{
processingBlock.Complete();
});
var task1 = Task.Run(async () =>
{
for (int i = 0; i < 50; i++)
{
await batchBlock.SendAsync($"Message {i}");
}
});
var task2 = Task.Run(async () =>
{
for (int i = 50; i < 100; i++)
{
await batchBlock.SendAsync($"Message {i}");
}
});
await Task.WhenAll(task1, task2);
// Complete pipeline. You can leave it as active if you want.
batchBlock.Complete();
processingBlock.Completion.Wait();

Parallel.ForEach faster than Task.WaitAll for I/O bound tasks?

I have two versions of my program that submit ~3000 HTTP GET requests to a web server.
The first version is based off of what I read here. That solution makes sense to me because making web requests is I/O bound work, and the use of async/await along with Task.WhenAll or Task.WaitAll means that you can submit 100 requests all at once and then wait for them all to finish before submitting the next 100 requests so that you don't bog down the web server. I was surprised to see that this version completed all of the work in ~12 minutes - way slower than I expected.
The second version submits all 3000 HTTP GET requests inside a Parallel.ForEach loop. I use .Result to wait for each request to finish before the rest of the logic within that iteration of the loop can execute. I thought that this would be a far less efficient solution, since using threads to perform tasks in parallel is usually better suited for performing CPU bound work, but I was surprised to see that the this version completed all of the work within ~3 minutes!
My question is why is the Parallel.ForEach version faster? This came as an extra surprise because when I applied the same two techniques against a different API/web server, version 1 of my code was actually faster than version 2 by about 6 minutes - which is what I expected. Could performance of the two different versions have something to do with how the web server handles the traffic?
You can see a simplified version of my code below:
private async Task<ObjectDetails> TryDeserializeResponse(HttpResponseMessage response)
{
try
{
using (Stream stream = await response.Content.ReadAsStreamAsync())
using (StreamReader readStream = new StreamReader(stream, Encoding.UTF8))
using (JsonTextReader jsonTextReader = new JsonTextReader(readStream))
{
JsonSerializer serializer = new JsonSerializer();
ObjectDetails objectDetails = serializer.Deserialize<ObjectDetails>(
jsonTextReader);
return objectDetails;
}
}
catch (Exception e)
{
// Log exception
return null;
}
}
private async Task<HttpResponseMessage> TryGetResponse(string urlStr)
{
try
{
HttpResponseMessage response = await httpClient.GetAsync(urlStr)
.ConfigureAwait(false);
if (response.StatusCode != HttpStatusCode.OK)
{
throw new WebException("Response code is "
+ response.StatusCode.ToString() + "... not 200 OK.");
}
return response;
}
catch (Exception e)
{
// Log exception
return null;
}
}
private async Task<ListOfObjects> GetObjectDetailsAsync(string baseUrl, int id)
{
string urlStr = baseUrl + #"objects/id/" + id + "/details";
HttpResponseMessage response = await TryGetResponse(urlStr);
ObjectDetails objectDetails = await TryDeserializeResponse(response);
return objectDetails;
}
// With ~3000 objects to retrieve, this code will create 100 API calls
// in parallel, wait for all 100 to finish, and then repeat that process
// ~30 times. In other words, there will be ~30 batches of 100 parallel
// API calls.
private Dictionary<int, Task<ObjectDetails>> GetAllObjectDetailsInBatches(
string baseUrl, Dictionary<int, MyObject> incompleteObjects)
{
int batchSize = 100;
int numberOfBatches = (int)Math.Ceiling(
(double)incompleteObjects.Count / batchSize);
Dictionary<int, Task<ObjectDetails>> objectTaskDict
= new Dictionary<int, Task<ObjectDetails>>(incompleteObjects.Count);
var orderedIncompleteObjects = incompleteObjects.OrderBy(pair => pair.Key);
for (int i = 0; i < 1; i++)
{
var batchOfObjects = orderedIncompleteObjects.Skip(i * batchSize)
.Take(batchSize);
var batchObjectsTaskList = batchOfObjects.Select(
pair => GetObjectDetailsAsync(baseUrl, pair.Key));
Task.WaitAll(batchObjectsTaskList.ToArray());
foreach (var objTask in batchObjectsTaskList)
objectTaskDict.Add(objTask.Result.id, objTask);
}
return objectTaskDict;
}
public void GetObjectsVersion1()
{
string baseUrl = #"https://mywebserver.com:/api";
// GetIncompleteObjects is not shown, but it is not relevant to
// the question
Dictionary<int, MyObject> incompleteObjects = GetIncompleteObjects();
Dictionary<int, Task<ObjectDetails>> objectTaskDict
= GetAllObjectDetailsInBatches(baseUrl, incompleteObjects);
foreach (KeyValuePair<int, MyObject> pair in incompleteObjects)
{
ObjectDetails objectDetails = objectTaskDict[pair.Key].Result
.objectDetails;
// Code here that copies fields from objectDetails to pair.Value
// (the incompleteObject)
AllObjects.Add(pair.Value);
};
}
public void GetObjectsVersion2()
{
string baseUrl = #"https://mywebserver.com:/api";
// GetIncompleteObjects is not shown, but it is not relevant to
// the question
Dictionary<int, MyObject> incompleteObjects = GetIncompleteObjects();
Parallel.ForEach(incompleteHosts, pair =>
{
ObjectDetails objectDetails = GetObjectDetailsAsync(
baseUrl, pair.Key).Result.objectDetails;
// Code here that copies fields from objectDetails to pair.Value
// (the incompleteObject)
AllObjects.Add(pair.Value);
});
}
A possible reason why Parallel.ForEach may run faster is because it creates the side-effect of throttling. Initially x threads are processing the first x elements (where x in the number of the available cores), and progressively more threads may be added depending on internal heuristics. Throttling IO operations is a good thing because it protects the network and the server that handles the requests from becoming overburdened. Your alternative improvised method of throttling, by making requests in batches of 100, is far from ideal for many reasons, one of them being that 100 concurrent requests are a lot of requests! Another one is that a single long running operation may delay the completion of the batch until long after the completion of the other 99 operations.
Note that Parallel.ForEach is also not ideal for parallelizing IO operations. It just happened to perform better than the alternative, wasting memory all along. For better approaches look here: How to limit the amount of concurrent async I/O operations?
https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.parallel.foreach?view=netframework-4.8
Basically the parralel foreach allows iterations to run in parallel so you are not constraining the iteration to run in serial, on a host that is not thread constrained this will tend to lead to improved throughput
In short:
Parallel.Foreach() is most useful for CPU bound tasks.
Task.WaitAll() is more useful for IO bound tasks.
So in your case, you are getting information from webservers, which is IO. If the async methods are implemented correctly, it won't block any thread. (It will use IO Completion ports to wait on) This way the threads can do other stuff.
By running the async methods GetObjectDetailsAsync(baseUrl, pair.Key).Result synchroniced, it will block a thread. So the threadpool will be flood by waiting threads.
So I think the Task solution will have a better fit.

Completion in TPL Dataflow Loops

I have a problem with determining how to detect completion within a looping TPL Dataflow.
I have a feedback loop in part of a dataflow which is making GET requests to a remote server and processing data responses (transforming these with more dataflow then committing the results).
The data source splits its results into pages of 1000 records, and won't tell me how many pages it has available for me. I have to just keep reading until i get less than a full page of data.
Usually the number of pages is 1, frequently it is up to 10, every now and again we have 1000s.
I have many requests to fetch at the start.
I want to be able to use a pool of threads to deal with this, all of which is fine, I can queue multiple requests for data and request them concurrently. If I stumble across an instance where I need to get a big number of pages I want to be using all of my threads for this. I don't want to be left with one thread churning away whilst the others have finished.
The issue I have is when I drop this logic into dataflow, such as:
//generate initial requests for activity
var request = new TransformManyBlock<int, DataRequest>(cmp => QueueRequests(cmp));
//fetch the initial requests and feedback more requests to our input buffer if we need to
TransformBlock<DataRequest, DataResponse> fetch = null;
fetch = new TransformBlock<DataRequest, DataResponse>(async req =>
{
var resp = await Fetch(req);
if (resp.Results.Count == 1000)
await fetch.SendAsync(QueueAnotherRequest(req));
return resp;
}
, new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 10 });
//commit each type of request
var commit = new ActionBlock<DataResponse>(async resp => await Commit(resp));
request.LinkTo(fetch);
fetch.LinkTo(commit);
//when are we complete?
QueueRequests produces an IEnumerable<DataRequest>. I queue the next N page requests at once, accepting that this means I send slightly more calls than I need to. DataRequest instances share a LastPage counter to avoid neadlessly making requests that we know are after the last page. All this is fine.
The problem:
If I loop by feeding back more requests into fetch's input buffer as I've shown in this example, then i have a problem with how to signal (or even detect) completion. I can't set completion on fetch from request, as once completion is set I can't feedback any more.
I can monitor for the input and output buffers being empty on fetch, but I think I'd be risking fetch still being busy with a request when I set completion, thus preventing queuing requests for additional pages.
I could do with some way of knowing that fetch is busy (either has input or is busy processing an input).
Am I missing an obvious/straightforward way to solve this?
I could loop within fetch, rather than queuing more requests. The problem with that is I want to be able to use a set maximum number of threads to throttle what I'm doing to the remote server. Could a parallel loop inside the block share a scheduler with the block itself and the resulting thread count be controlled via the scheduler?
I could create a custom transform block for fetch to handle the completion signalling. Seems like a lot of work for such a simple scenario.
Many thanks for any help offered!
In TPL Dataflow, you can link the blocks with DataflowLinkOptions with specifying the propagation of completion of the block:
request.LinkTo(fetch, new DataflowLinkOptions { PropagateCompletion = true });
fetch.LinkTo(commit, new DataflowLinkOptions { PropagateCompletion = true });
After that, you simply call the Complete() method for the request block, and you're done!
// the completion will be propagated to all the blocks
request.Complete();
The final thing you should use is Completion task property of the last block:
commit.Completion.ContinueWith(t =>
{
/* check the status of the task and correctness of the requests handling */
});
For now I have added a simple busy state counter to the fetch block:-
int fetch_busy = 0;
TransformBlock<DataRequest, DataResponse> fetch_activity=null;
fetch = new TransformBlock<DataRequest, ActivityResponse>(async req =>
{
try
{
Interlocked.Increment(ref fetch_busy);
var resp = await Fetch(req);
if (resp.Results.Count == 1000)
{
await fetch.SendAsync( QueueAnotherRequest(req) );
}
Interlocked.Decrement(ref fetch_busy);
return resp;
}
catch (Exception ex)
{
Interlocked.Decrement(ref fetch_busy);
throw ex;
}
}
, new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 10 });
Which I then use to signal complete as follows:-
request.Completion.ContinueWith(async _ =>
{
while ( fetch.InputCount > 0 || fetch_busy > 0 )
{
await Task.Delay(100);
}
fetch.Complete();
});
Which doesnt seem very elegant, but should work I think.

How to make concurrent requests without creating multiple threads?

Can someone please show how to make concurrent requests without creating multiple threads? E.g., I want a program that makes 100 web requests and I don't want more than 8 concurrent requests at any time. I don't want to create 8 threads for the 8 concurrent requests. When a thread makes an async request, the same thread can then be used to make the next request, and so on. I am sorry but I can't wrap my head around this, and would like to see the best solution out there. In case it wasn't clear, the requests I am talking about are async. I want to see a solution that does not use any locks, and uses the built-in classes to do the work.
This is some code I came up with but it does not do what it is supposed to do.
Task.Run(async () =>
{
var outstandingRequests = 0;
var requestCount = 0;
var tasks = new List<Task>(concurrentRequests);
while (requestCount < maxRequests)
{
if (outstandingRequests < concurrentRequests)
{
tasks.Add(svc.GetDataAsync()); // a method that makes an async request
Interlocked.Increment(ref outstandingRequests);
}
else
{
var t = await Task.WhenAny(tasks);
Interlocked.Decrement(ref outstandingRequests);
Interlocked.Increment(ref requestCount);
}
}
await Task.WhenAll(tasks);
}).Wait();
Output:
[] 1 Sending Request...Received Response 490,835.00 bytes in 15.6 sec
[] 2 Sending Request...
[] 3 Sending Request...
[] 4 Sending Request...
[] 5 Sending Request...
[] 6 Sending Request...
[] 7 Sending Request...
[] 8 Sending Request...
[] 9 Sending Request...
I have set concurrentRequests to 5, so there is some bug in above code as it is making 8 requests in parallel. Initially it made only 5 requests in parallel, but as soon as one request completed, it fired off 4 more requests (should have fired off only one more).
Had to fix some bugs, but it all works out now:
Task.Run(async () =>
{
var outstandingRequests = 0;
var requestCount = 0;
// adding and removing from a List<> at the same time is not thread-safe,
// so have to use a SynchronizedCollection<>
var tasks = new SynchronizedCollection<Task>();
while (requestCount < maxRequests)
{
if (outstandingRequests < concurrentRequests)
{
tasks.Add(svc.GetDataAsync(uri)); // this will be your method that makes async web call and returns a Task to signal completion of async call
Interlocked.Increment(ref outstandingRequests);
Interlocked.Increment(ref requestCount);
}
else
{
**tasks.Remove(await Task.WhenAny(tasks));**
Interlocked.Decrement(ref outstandingRequests);
}
}
await Task.WhenAll(tasks);
}).Wait();
If there is a better way to do it, please let me know.
Looks like you are trying to reinvent the thread pool. Don't do that - just use existing functionality: http://msdn.microsoft.com/en-us/library/system.threading.threadpool.aspx
Or you can use async versions of request methods - they are based on the thread pool too.
How about this:
Parallel.Invoke (new ParallelOptions { MaxDegreeOfParallelism = 8 },
svcs.Select (svc => svc.GetDataAsync ()).ToArray ()) ;
There is a sample Microsoft implementation of a limited-concurrency task scheduler here. See SO questions System.Threading.Tasks - Limit the number of concurrent Tasks and .Net TPL: Limited Concurrency Level Task scheduler with task priority?.

Categories

Resources