I have a client application that will get a large number of jobs to run, on the order of 10k, and foreach will make an http request to a web api. Each job is semi long running and unpredictable 7-90s response times.
I am trying to minimize the total time for all jobs. I notice that if I make too many requests at once, response times drastically increase because the server is essentially being DoSed. This is bringing the total for all jobs way up. I have been using SemaphoreSlim to statically set the order of parallelism but need to find a way to dynamically adjust based on current response times to get the lowest response times overall. Here is the code I have been using.
List<Task<DataTable>> tasks = new List<Task<DataTable>>();
SemaphoreSlim throttler = new SemaphoreSlim(40, 300);
foreach (DataRow entry in newEntries.Rows)
{
await throttler.WaitAsync();
tasks.Add(Task<DataTable>.Run(async () =>
{
try
{
return RunRequest(entry); //Http requests and other logic is done here
}
finally
{
throttler.Release();
}
}
));
}
await Task.WhenAll(tasks.ToArray());
I know that throttler.Release(); can be passed different numbers to increase the total number of outstanding request at one time and calling Wait() without Release() will subtract from the count.
I believe that need to keep a some sort of rolling average of response times. Using the rolling average some how determine how much to increase/decrease the total number of outstanding requests being allowed. I am not sure if this is the right direction or not.
Question
Given the information above, how can I keep the total number of outstanding requests at a level to have the minimum time spent for all jobs.
List<DataTable> dataTables = new List<DataTable>();
Parallel.ForEach(newEntries.AsEnumerable(), new ParallelOptions { MaxDegreeOfParallelism = 2 }, row => {
var request = RunRequest(row);
lock(dataTables)
{
dataTables.Add(request);
}
});
Now you can adjust the MaxDegreeOfParallelism //I wasn't understanding that you wanted to have this dynamically changed as the tasks were running.
I'll tell you from past experience when trying to allow users to change the amount of running threads when they are in process using a Semaphore, I wanted to jump in front of a moving truck. This was back before TPL.
Create a list of unstarted tasks. Start the number of tasks that you want, like 5 to start. Each task can return a duration from start to finish so you can use it to define your throttle logic. Now just loop the tasks with waitany as the block.
var runningTasks = 5;
//for loop to start 5 tasks.
while (taskList.count > 0)
{
var indexer = Task.WaitAny(taskList);
var myTask = taskList[indexer];
taskList.RemoveAt(indexer);
InterLocker.Decrement(ref runningTasks);
var durationResult = myTask.Result();
//do logic to determine if you need to start more.
//when you start another use InterLocker.Increment(ref runningTasks);
}
Related
I'm determining between using TPL Dataflow blocks or some sort of producer/consumer approach for these tests. Producing a list of tasks will be super-fast, as each task will just be a string containing a list of test parameters such as the setup parameters, the measurements required, and time between measurements. This list of tasks will simply be files loaded through the GUI (1 file per test).
When at test is started, it should start right away. The tests could be very long and very asynchronous in that an action could take seconds or tens of minutes (e.g. heating up a device), followed by a measurement that takes a few seconds (or minutes), followed by a long period of inaction (24 hours) before the test is repeated again.
I could have up to 16 tests running at the same time, but I need the flexibility to be able to cancel any one of those tests at any time. I also need to be able to ADD a new test at any time (i.e. try to picture testing of 16 devices, or the span of a month in which individual test devices are added and removed throughout the month).
(Visual C#) I tried this example code for TPL dataflow where I tell it to run 32 simple tasks all at the same time. Each task is just a 5 second delay to simulate work. It appears to be processing the tasks in parallel as the time to complete the tasks took 15 seconds. I assume all 32 tasks did not finish in 5 seconds due to scheduling and any other overhead, but I am a bit worried that some task might of been blocked.
class Program
{
// Performs several computations by using dataflow and returns the elapsed
// time required to perform the computations.
static TimeSpan TimeDataflowComputations(int messageCount)
{
// Create an ActionBlock<int> that performs some work.
var workerBlock = new ActionBlock<int>(
// Simulate work by suspending the current thread.
millisecondsTimeout => Thread.Sleep(millisecondsTimeout),
// Specify a maximum degree of parallelism.
new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = messageCount
});
// Compute the time that it takes for several messages to
// flow through the dataflow block.
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
for (int i = 0; i < messageCount; i++)
{
workerBlock.Post(5000); // simulated work: a delay of 5 seconds.
}
workerBlock.Complete();
// Wait for all messages to propagate through the network.
workerBlock.Completion.Wait();
// Stop the timer and return the elapsed number of milliseconds.
stopwatch.Stop();
return stopwatch.Elapsed;
}
static void Main(string[] args)
{
int messageCount = 32;
TimeSpan elapsed;
// set processors maximum degree of parallelism. This causes
// multiple messages to be processed in parallel.
Console.WriteLine("START:\r\n");
elapsed = TimeDataflowComputations(messageCount);
Console.WriteLine("message count = {1}; " +
"elapsed time = {2}ms.", messageCount,
(int)elapsed.TotalMilliseconds);
Console.ReadLine();
}
}
The demo seems to work, but I am not sure if any of the tasks were blocked until one or more of the 5 second tasks were completed. I am also not sure how one would go about identifying each action block in order to cancel a specific one.
The reason that you don't get the expected performance is because your workload is synchronous and blocks the thread-pool threads. Do you expect to actually have synchronous (blocking) workload in your production environment? If yes, you could try boosting the ThreadPool reserve of available threads before starting the TPL Dataflow pipeline:
ThreadPool.SetMinThreads(workerThreads: 100, completionPortThreads: 100);
If your actual workload is asynchronous, then you could better simulate it with Task.Delay instead of Thread.Sleep.
var workerBlock = new ActionBlock<int>(async millisecondsTimeout =>
{
await Task.Delay(millisecondsTimeout);
}, new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = messageCount
});
I didn't test it, but you should get completion times at around 5 sec with both these approaches.
I'm writing an application which interact with Azure Cosmos DB. I need commit 30,000 Records to CosmosDB in a Session. Because I used .NET Core so I cannot use BulkInsert dll. So, I use Foreach loop to Insert to CosmosDB. But I see too many request per Second and it overload RU limit by CosmosDB.
foreach(item in listNeedInsert){
await RequestInsertToCosmosDB(item);
}
I want to Pause foreach loop when Number of request reach 100. After done 100 request. foreach will continue.
You can partition the list and await the results:
var tasks = new List<Task>();
foreach(item in listNeedInsert)
{
var task = RequestInsertToCosmosDB(item);
tasks.Add(task);
if(tasks.Count == 100)
{
await Task.WhenAll(tasks);
tasks.Clear();
}
}
// Wait for anything left to finish
await Task.WhenAll(tasks);
Every time you've got 100 tasks running the code will wait for them all to finish before executing the last batch.
You could set a delay on every hundred iteration
int i = 1;
foreach(item in listNeedInsert)
{
await RequestInsertToCosmosDB(item);
if (i % 100 == 0)
{
i = 0;
await Task.Delay(100); // Miliseconds
}
i++;
}
If you really want to maximize efficiency and can't do bulk updates, look into using SemaphorSlim in this post:
Throttling asynchronous tasks
Hammering a medium-sized database with 100 concurrent requests at a time isn't a great idea because it's not equipped to handle that kind of throughput. You could try playing with a different throttling number and seeing what's optimal, but I would guess it's in the single digit range.
If you want to do something quick and dirty, you could probably use Sean's solution. But I'd set the Task count to 5 starting out, not 100.
https://github.com/thomhurst/EnumerableAsyncProcessor
I've written a library to help with this sort of logic.
Usage would be:
await AsyncProcessorBuilder.WithItems(listNeedInsert) // Or Extension Method: listNeedInsert.ToAsyncProcessorBuilder()
.ForEachAsync(item => RequestInsertToCosmosDB(item), CancellationToken.None)
.ProcessInBatches(batchSize: 100);
I have a number of producer tasks that push data into a BlockingCollection, lets call it requestQueue.
I also have a consumer task that pops requests from the requestQueue, and forwards async http requests to a remote web service.
I need to throttle or block the number of active requests sent to the web service. On some machines that are far away from the service or have a slower internet connection, the http response time is long enough that the number of active requests fills up more memory than I'd like.
At the moment I am using a semaphore approach, calling WaitOne on the consumer thread multiple times, and Release on the HTTP response callback. Is there a more elegant solution?
I am bound to .net 4.0, and would like a standard library based solution.
You are already using a BlockingCollection why have a WaitHandle?
The way I would do it is to have a BlockingCollection with n as it's bounded capacity where n is the maximum number of concurrent requests you want to have at any given time.
You can then do something like....
var n = 4;
var blockingQueue = new BlockingCollection<Request>(n);
Action<Request> consumer = request =>
{
// do something with request.
};
var noOfWorkers = 4;
var workers = new Task[noOfWorkers];
for (int i = 0; i < noOfWorkers; i++)
{
var task = new Task(() =>
{
foreach (var item in blockingQueue.GetConsumingEnumerable())
{
consumer(item);
}
}, TaskCreationOptions.LongRunning | TaskCreationOptions.DenyChildAttach);
workers[i] = task;
workers[i].Start();
}
Task.WaitAll(workers);
I let you take care of cancellation and error handling but using this you can also control how many workers you want to have at any given time, if the workers are busy sending and processing the request any other producer will be blocked until more room is available in the queue.
I am trying to implement some kind of task queue with different task weights allowing different amount of tasks to run concurrently based on their weights.
There are two types of tasks: long task and short task.
Maximum of N short tasks can be executed together.
When the long task appears, it should start immediately if there are no other long task running or wait for it to complete otherwise.
Concurrent short task count limit should be reduced to M if there is long task running.
Short tasks running already should keep running to completion; but no new short tasks should start if the current limit is less or equal than current running short tasks amount.
As it seems, I basically need the ability to dynamically change semaphore "capacity".
It would be easy to decrease/increase the capacity by just taking/freeing (N - M) "slots" when needed, but that would cause "hanging" of the queue before (N - M) short tasks complete if there are N short tasks already running.
I could also implement some kind of "scheduler" awaking every 100 ms (for example) and checking the queue for any tasks that can start now. Disadvantage of this approach is having up to 100 ms delay between enqueueing the task and starting it.
So I am stuck with that puzzle and hope that someone will have some fresh idea about how to implement this.
Update:
Tasks are not going to create any significant CPU load.
They are HTTP requests in fact. Long requests are uploading files and short requests are common HTTP requests.
I answered a very similar question a few days ago, the solution for you is pretty much the exact same, use QueuedTaskScheduler
from "ParallelExtensionsExtras"
private static void Main(string[] args)
{
int highPriorityMaxConcurrancy = 1
QueuedTaskScheduler qts = new QueuedTaskScheduler();
var highPriortiyScheduler = qts.ActivateNewQueue(0);
var lowPriorityScheduler = qts.ActivateNewQueue(1);
BlockingCollection<HttpRequestWrapper> fileRequest= new BlockingCollection<Foo>();
BlockingCollection<HttpRequestWrapper> commonRequest= new BlockingCollection<Foo>();
List<Task> processors = new List<Task>(2);
processors.Add(Task.Factory.StartNew(() =>
{
Parallel.ForEach(fileRequest.GetConsumingPartitioner(), //.GetConsumingPartitioner() is also from ParallelExtensionExtras, it gives better performance than .GetConsumingEnumerable() with Parallel.ForEeach(
new ParallelOptions() { TaskScheduler = highPriortiyScheduler, MaxDegreeOfParallelism = highPriorityMaxConcurrancy },
ProcessWork);
}, TaskCreationOptions.LongRunning));
processors.Add(Task.Factory.StartNew(() =>
{
Parallel.ForEach(commonRequest.GetConsumingPartitioner(),
new ParallelOptions() { TaskScheduler = lowPriorityScheduler},
ProcessWork);
}, TaskCreationOptions.LongRunning));
//Add some work to do here to the fileRequest or commonRequest collections
//Lets the blocking collections know we are no-longer going to be adding new items so it will break out of the `ForEach` once it has finished the pending work.
fileRequest.CompleteAdding();
commonRequest.CompleteAdding();
//Waits for the two collections to compleatly empty before continueing
Task.WaitAll(processors.ToArray());
}
private static void ProcessWork(HttpRequestWrapper request)
{
//...
}
I have a WCF Service that is responseible for taking in an offer and 'reaching' out and dynamically provide this offer to X amount of potential buyers (typically 15-20) which are essentially external APIs.
Each of the buyers currently has 35 seconds to return a response, or they lose the ability to buy the offer,
In order to accomplish this, I have the following code which has been in production for 8 months and has worked and scaled rather well.
As we have been spending a lot of time on improving recently so that we can scale further, I have been interested in whether I have a better option for how I accomplishing this task. I am hesitant in making changes because it is workign well right now,however I may be able to squeeze additional performance out of it right now while I am able to focus on it.
The following code is responsible for creating the tasks which make the outbound requests to the buyers.
IBuyer[] buyer = BuyerService.GetBuyers(); /*Obtain potential buyers for the offer*/
var tokenSource = new CancellationTokenSource();
var token = tokenSource.Token;
Tasks = new Task<IResponse>[Buyers.Count];
for(int i = 0; i < Buyers.Count;i++)
{
IBuyer buyer = Buyers[i];
Func<IResponse> makeOffer = () => buyer.MakeOffer()
Tasks[i] = Task.Factory.StartNew<IResponse>((o) =>
{
try
{
var result = MakeOffer();
if (!token.IsCancellationRequested)
{
return result;
}
}
catch (Exception exception
{
/*Do Work For Handling Exception In Here*/
}
return null;
}, token,TaskCreationOptions.LongRunning);
};
Task.WaitAll(Tasks, timeout, token); /*Give buyers fair amount of time to respond to offer*/
tokenSource.Cancel();
List<IResponse> results = new List<IResponse>(); /*List of Responses From Buyers*/
for (int i = 0; i < Tasks.Length; i++)
{
if (Tasks[i].IsCompleted) /*Needed so it doesnt block on Result*/
{
if (Tasks[i].Result != null)
{
results.Add(Tasks[i].Result);
}
Tasks[i].Dispose();
}
}
/*Continue Processing Buyers That Responded*/
On average, this service is called anywhere from 400K -900K per day, and sometimes up to 30-40 times per second.
We have made a lot of optimizations in an attempt to tune performance, but I want to make sure that this piece of code does not have any immediate glaring issues.
I read alot about the power of TaskScheduler and messing with the SynchronizationContext and working async, and I am not sure how I can make that fit and if it is worth an improvement or not.
Right now, you're using thread pool threads (each Task.Factory.StartNew call uses a TP thread or a full .NET thread, as in your case, due to the LongRunning hint) for work that is effectively IO bound. If you hadn't specified TaskCreationOptions.LongRunning, you'd have seen a problem very early on, and you'd be experiencing thread pool starvation. As is, you're likely using a very large number of threads, and creating and destroying them very quickly, which is a waste of resources.
If you were to make this fully asynchronous, and use the new async/await support, you could perform the same "work" asynchronously, without using threads. This would scale significantly better, as the amount of threads used for a given number of requests would be significantly reduced.
As a general rule of thumb, Task.Factory.StartNew (or Task.Run in .NET 4.5, as well as the Parallel class) should only be used for CPU bound work, and async/await should be used for IO bound work, especially for server side operations.