I have a number of producer tasks that push data into a BlockingCollection, lets call it requestQueue.
I also have a consumer task that pops requests from the requestQueue, and forwards async http requests to a remote web service.
I need to throttle or block the number of active requests sent to the web service. On some machines that are far away from the service or have a slower internet connection, the http response time is long enough that the number of active requests fills up more memory than I'd like.
At the moment I am using a semaphore approach, calling WaitOne on the consumer thread multiple times, and Release on the HTTP response callback. Is there a more elegant solution?
I am bound to .net 4.0, and would like a standard library based solution.
You are already using a BlockingCollection why have a WaitHandle?
The way I would do it is to have a BlockingCollection with n as it's bounded capacity where n is the maximum number of concurrent requests you want to have at any given time.
You can then do something like....
var n = 4;
var blockingQueue = new BlockingCollection<Request>(n);
Action<Request> consumer = request =>
{
// do something with request.
};
var noOfWorkers = 4;
var workers = new Task[noOfWorkers];
for (int i = 0; i < noOfWorkers; i++)
{
var task = new Task(() =>
{
foreach (var item in blockingQueue.GetConsumingEnumerable())
{
consumer(item);
}
}, TaskCreationOptions.LongRunning | TaskCreationOptions.DenyChildAttach);
workers[i] = task;
workers[i].Start();
}
Task.WaitAll(workers);
I let you take care of cancellation and error handling but using this you can also control how many workers you want to have at any given time, if the workers are busy sending and processing the request any other producer will be blocked until more room is available in the queue.
Related
Consider a .Net Core 3.1 API with the following endpoints
GET : /computation - Performs a CPU intensive computation task
GET : /livecheck
High Load on /computation endpoint :
When there is high load on the '/computation' endpoint, 300 requests per second, other endpoints slow down as all threads are used up.
During the high load calling the '/livecheck' endpoint will return a request in 5-10 seconds, which is too much.
This is a problem, because if the '/livecheck' endpoint does not respond on time, the app is killed.(AWS ECS, kills the container when livecheck takes more than 5 seconds)
Is it possible to ensure '/livecheck' endpoint still returns data, by running the '/computation' endpoint on a separate thread pool. So that it does not use up all worker threads and they are available for other endpoints?
Note :
'/computation' has to be returned as a part of the same request,
don't want to queue it to background task.
Any other solutions also welcome .
I would suggest that the 'cpu intensive computation' is offloaded to another app and not done in the same app as the live check. An async message can be sent to trigger the computation processing to start and the app could subscribe to a finished or failed event from the processing app. That way heavy processing would not impact the response time of the web api.
Better than that, you can dockerize the apps.
To extend my comments under question
I still recommend you to follow the best practices to avoid blocking calls.
But for now, let's assume you really cannot, then you should at least offload computation from thread whenever it suits to give processor time to livecheck.
Consider this example:
void Computation()
{
for (var i = 0; i < 300; i++)
{
Thread.Sleep(1);
}
}
void LiveCheck()
{
Console.WriteLine("I'm alive.");
}
async Task Main()
{
var tasks = new List<Task>();
// Create 1000 blocking compuations to simulate busy thread pool
// or thread pool starvation
for (int i = 0; i < 1000; i++)
{
tasks.Add(Task.Run(Computation));
}
// Simulate 3 seconds after thread pool is busy, execute livecheck
Thread.Sleep(3000);
var sw = Stopwatch.StartNew();
await Task.Run(LiveCheck);
sw.Stop();
Console.WriteLine($"LiveCheck completed in {sw.Elapsed.TotalSeconds} seconds");
await Task.WhenAll(tasks);
}
In most cases, the return result on my machine is like:
LiveCheck completed in 31.2030817 seconds
If offload computation from thread:
async Task Computation()
{
for (var i = 0; i < 300; i++)
{
Thread.Sleep(1);
// Yield thread every 50ms
if ((i % 50) == 0)
{
await Task.Yield();
}
}
}
Output usually is:
LiveCheck completed in 2.4753268 seconds
The trade-off here is a single run of computation will be slower than sync version.
I'm writing a responsive API. We have to handle 10 requests per second.
The problem is, sending a response takes half a second. So you can imagine, the server is overwhelmed quickly.
I made the code process asynchronously, up to 10 tasks at once, to help mitigate this.
However I have concerns about whether using a single instance of HttpClient is the correct approach. The advice as soon as someone mentions HttpClient is always create a single instance of it.
I have a static instance of it. Although it is thread-safe, at least for PostAsync, should I really create 10 HttpClients (or a pool of HttpClients) to be able to send data out faster?
I assume that during the half a second it's sending out, that it won't let you send out other 'postasync's. However I can't confirm this behaviour.
Most benchmarks and resources simply look at sending requests synchronously, i.e. one after the other (i.e. await postasync)
However for my use case I need to send several simultaneously, i.e. from separate threads. The only way to reply to 10 messages per second that take half a second each is to to send five simultanous messages back - not five queued to go out one by one, but five simultaneous messages.
I cannot find any documentation on how HttpClient handles this. I've only seen a few references to it having a connection pool, but it's unclear whether it will actually perform multiple connections simultaneously, or if I need to create a small pool of 5 httpclients to rotate through.
Question: Does a single instance of HttpClient support multiple connections simultaneously?
And I don't mean just letting you call postasync lots of times in a thread-safe way before it has finished, but I mean truly opening five simultaneous connections and sending data through each of them at the exact same time?
An example would be, you're sending fifty 10 byte files to the moon, and there is a latency of 10 seconds. Your program scoops up all fifty files and makes fifty calls to HttpClient.PostAsync almost instantly.
Assuming the listening service can support it, would the cross-thread calls to HttpClient.PostAsync open fifty connections (or whatever, some limit, but more than 1) and send the data, meaning that the server receives all fifty files ~10 seconds later?
Or would it internally queue them up and you'd end up waiting 10x50=500 seconds?
Seems there is no limit, or at least, it's a high one.
I made a default web api application, modified the boilerplate controller method to this:
// GET api/values
public async Task<IEnumerable<string>> Get()
{
Debug.Print("Called");
Thread.Sleep(100000);
return new string[] { "value1", "value2" };
}
I then made a program that using a single instance of HttpClient, would make lots of simultaneous connections using Task.Run.
List<Task> tasks = new List<Task>();
var task1 = Task.Run(() => httpClient.GetAsync("http://localhost:57984/api/values"));
var task2 = Task.Run(() => httpClient.GetAsync("http://localhost:57984/api/values"));
var task3 = Task.Run(() => httpClient.GetAsync("http://localhost:57984/api/values"));
var task4 = Task.Run(() => httpClient.GetAsync("http://localhost:57984/api/values"));
var task5 = Task.Run(() => httpClient.GetAsync("http://localhost:57984/api/values"));
var task6 = Task.Run(() => httpClient.GetAsync("http://localhost:57984/api/values"));
var task7 = Task.Run(() => httpClient.GetAsync("http://localhost:57984/api/values"));
var task8 = Task.Run(() => httpClient.GetAsync("http://localhost:57984/api/values"));
var task9 = Task.Run(() => httpClient.GetAsync("http://localhost:57984/api/values"));
var taskA = Task.Run(() => httpClient.GetAsync("http://localhost:57984/api/values"));
var taskB = Task.Run(() => httpClient.GetAsync("http://localhost:57984/api/values"));
var taskC = Task.Run(() => httpClient.GetAsync("http://localhost:57984/api/values"));
var taskD = Task.Run(() => httpClient.GetAsync("http://localhost:57984/api/values"));
var taskE = Task.Run(() => httpClient.GetAsync("http://localhost:57984/api/values"));
await Task.WhenAll(task1, task2, task3, task4, task5, task6, task7, task8, task9, taskA, taskB, taskC, taskD);
I ran them and the word 'Called' was logged 14 times.
Since the Thread.Sleep will have blocked the response, it should mean there were 14 simultaneous connections.
There are two properties that might affect the maximum number of connections, that I've found by looking on google:
ServicePointManager.DefaultConnectionLimit which is defaulted to 2
and, the HttpClientHandler.MaxConnectionsPerServer which is also 2.
As I'm able to make many more than 2 connections, I really don't know if it's just ignored, or if these are the wrong settings, or what. Changing them appears to have no effect.
I noticed after a lot of stopping and starting my test projects that new connections were much slower to be made. I am guessing that I saturated the connection pool.
My conclusion is that if you set those two values to something higher (just in case, I mean, why not), then you can use a single httpclient concurrently where the connections will be truly concurrent, rather than sequential and thread safe.
However I can't confirm this behaviour.
Why not? Just create a webapi with a few seconds delay, and test calling it with HttpClient. Or you can use a service like Slowwly.
static async Task Main(string[] args)
{
var stopwatch = Stopwatch.StartNew();
await Serial(stopwatch);
Console.WriteLine($"Serial took {stopwatch.Elapsed}");
stopwatch.Restart();
await Concurrent(stopwatch);
Console.WriteLine($"Concurrent took {stopwatch.Elapsed}");
}
private static async Task Serial(Stopwatch stopwatch)
{
for (var i = 0; i != 5; ++i)
{
var client = new HttpClient();
await MakeRequest(stopwatch, client);
}
}
private static async Task Concurrent(Stopwatch stopwatch)
{
var client = new HttpClient();
var tasks = Enumerable.Range(0, 5).Select(async _ => { await MakeRequest(stopwatch, client); }).ToList();
await Task.WhenAll(tasks);
}
private static async Task MakeRequest(Stopwatch stopwatch, HttpClient client)
{
Console.WriteLine($"{stopwatch.Elapsed}: Issuing request.");
var response = await client.GetStringAsync("http://slowwly.robertomurray.co.uk/delay/3000/url/http://www.google.com");
Console.WriteLine($"{stopwatch.Elapsed}: Received {response.Length} bytes.");
}
Output for me (from the US):
00:00:00.0463664: Issuing request.
00:00:04.2560734: Received 49237 bytes.
00:00:04.2562498: Issuing request.
00:00:07.6731908: Received 49247 bytes.
00:00:07.6734158: Issuing request.
00:00:11.0882322: Received 49364 bytes.
00:00:11.0883803: Issuing request.
00:00:14.4990981: Received 49294 bytes.
00:00:14.4993977: Issuing request.
00:00:17.9082167: Received 49328 bytes.
Serial took 00:00:17.9083969
00:00:00.0025096: Issuing request.
00:00:00.0252402: Issuing request.
00:00:00.0422682: Issuing request.
00:00:00.0588887: Issuing request.
00:00:00.0755351: Issuing request.
00:00:03.4631815: Received 49278 bytes.
00:00:03.4632073: Received 49293 bytes.
00:00:03.4844698: Received 49313 bytes.
00:00:03.4913929: Received 49308 bytes.
00:00:03.4915415: Received 49280 bytes.
Concurrent took 00:00:03.4917199
Question: Does a single instance of HttpClient support multiple connections simultaneously?
Yes.
I am currently working on WebApi using .Net Core, one of my Api Method will call number of another Api (3rd party), and it will take some time to return response, but I don't want our Api consumers to wait for response, instead I want to return early response i.e The Operation is started. And I ll provide an endpoint to our Consumers through which they can get the status of that operation. For example our consumer calls the api to generate 100k records for which my Api will call around 20 parallel calls to 3rd party api. So I don't want consumer for these 20 apis response.
Currently I have this code:
public async Task<ActionResult> GenerateVouchers([FromBody][Required]CreateVoucherRequestModel request, string clientId)
{
_logger.LogInformation(Request.Method, Request.Path);
// await _voucherService.ValidateIdempotedKeyWithStatus(clientId, _idempotentHeader);
//TODO: Check Voucher type & Status before Generating Voucher
var watch = Stopwatch.StartNew();
var vouchers = new List<VoucherCreateResponseModel>();
var batchSize = 5000;
int numberOfBatches = (int)Math.Ceiling((double)request.quantity / batchSize);
int totalVoucherQuantity = request.quantity;
request.quantity = 5000;
var tasks = new List<Task<VoucherCreateResponseModel>>();
for (int i = 0; i < numberOfBatches; i++)
{
tasks.Add(_client.GenerateVoucher($"CouponsCreate", request));
vouchers.AddRange(await Task.WhenAll(tasks).ConfigureAwait(false));
}
// await _voucherService.GenerateVouchers(request, clientId, _idempotentHeader);
watch.Stop();
var totalMS = watch.ElapsedMilliseconds;
return Ok();
}
But the issue with above code even though I have ConfigureAwait(false), it waits for all 20 requests to execute and when response of all requests are returned than api consumer will get response, but each each of these 20 request will take around 5 seconds to execute, so our consumers may get request timeout while waiting for response.
How can I fix such issue in .Net Core.
It's not a good practice to wait for long running process inside controller.
My opinion is ,
put the data necessary (something like a Id for a batch) for long
running process to a Azure queue within the API
trigger a function app from the particular queue, So API's responsibility is
putting the data in to the queue
From there on it's function apps
responsibility to complete process .
May be using something like
signalR you can notify the frontend when process is completed
I have a client application that will get a large number of jobs to run, on the order of 10k, and foreach will make an http request to a web api. Each job is semi long running and unpredictable 7-90s response times.
I am trying to minimize the total time for all jobs. I notice that if I make too many requests at once, response times drastically increase because the server is essentially being DoSed. This is bringing the total for all jobs way up. I have been using SemaphoreSlim to statically set the order of parallelism but need to find a way to dynamically adjust based on current response times to get the lowest response times overall. Here is the code I have been using.
List<Task<DataTable>> tasks = new List<Task<DataTable>>();
SemaphoreSlim throttler = new SemaphoreSlim(40, 300);
foreach (DataRow entry in newEntries.Rows)
{
await throttler.WaitAsync();
tasks.Add(Task<DataTable>.Run(async () =>
{
try
{
return RunRequest(entry); //Http requests and other logic is done here
}
finally
{
throttler.Release();
}
}
));
}
await Task.WhenAll(tasks.ToArray());
I know that throttler.Release(); can be passed different numbers to increase the total number of outstanding request at one time and calling Wait() without Release() will subtract from the count.
I believe that need to keep a some sort of rolling average of response times. Using the rolling average some how determine how much to increase/decrease the total number of outstanding requests being allowed. I am not sure if this is the right direction or not.
Question
Given the information above, how can I keep the total number of outstanding requests at a level to have the minimum time spent for all jobs.
List<DataTable> dataTables = new List<DataTable>();
Parallel.ForEach(newEntries.AsEnumerable(), new ParallelOptions { MaxDegreeOfParallelism = 2 }, row => {
var request = RunRequest(row);
lock(dataTables)
{
dataTables.Add(request);
}
});
Now you can adjust the MaxDegreeOfParallelism //I wasn't understanding that you wanted to have this dynamically changed as the tasks were running.
I'll tell you from past experience when trying to allow users to change the amount of running threads when they are in process using a Semaphore, I wanted to jump in front of a moving truck. This was back before TPL.
Create a list of unstarted tasks. Start the number of tasks that you want, like 5 to start. Each task can return a duration from start to finish so you can use it to define your throttle logic. Now just loop the tasks with waitany as the block.
var runningTasks = 5;
//for loop to start 5 tasks.
while (taskList.count > 0)
{
var indexer = Task.WaitAny(taskList);
var myTask = taskList[indexer];
taskList.RemoveAt(indexer);
InterLocker.Decrement(ref runningTasks);
var durationResult = myTask.Result();
//do logic to determine if you need to start more.
//when you start another use InterLocker.Increment(ref runningTasks);
}
I have a task where I form thousands of requests which are later sent to a server. The server returns the response for each request and that response is then dumped to an output file line by line.
The pseudo code goes like this:
//requests contains thousands of requests to be sent to the server
string[] requests = GetRequestsString();
foreach(string request in requests)
{
string response = MakeWebRequest(request);
ParseandDump(response);
}
Now, as can be seen the serve is handling my requests one by one. I want to make this entire process fast. The server in question is capable of handling multiple requests at a time. I want to apply multi-threading and send let's say 4 requests to the server at a time and dump the response in same thread.
Can you please give me any pointer to possible approaches.
You can take advantage of Task from .NET 4.0 and the new toy HttpClient, sample code below is showed how you send requests in parallel, then dump response in the same thread by using ContinueWith:
var httpClient = new HttpClient();
var tasks = requests.Select(r => httpClient.GetStringAsync(r).ContinueWith(t =>
{
ParseandDump(t.Result);
}));
Task uses ThreadPool under the hood, so you don't need to specify how many threads should be used, ThreadPool will manage this for you in optimized way.
The easiest way would be to use Parallel.ForEach like this:
string[] requests = GetRequestsString();
Parallel.ForEach(requests, request => ParseandDump(MakeWebRequest(request)));
.NET framework 4.0 or greater is required to use Parallel.
I think this could be done in a consumer-producer-pattern. You could use a ConcurrentQueue (from the namespace System.Collections.Concurrent) as a shared resource between the many parallel WebRequests and the dumping thread.
The pseudo code would be something like:
var requests = GetRequestsString();
var queue = new ConcurrentQueue<string>();
Task.Factory.StartNew(() =>
{
Parallel.ForEach(requests , currentRequest =>
{
queue.Enqueue(MakeWebRequest(request));
}
});
Task.Factory.StartNew(() =>
{
while (true)
{
string response;
if (queue.TryDequeue(out response))
{
ParseandDump(response);
}
}
});
Maybe a BlockingCollection might serve you even better, depending on how you want to go about synchronizing the threads to signal the end of incoming requests.