Develop Reference

Threads with low priority prevents other threads with normal priority from execution

We have found that several CPU intensive queries mean that our API server no longer responses to simple requests. The API server is an .net core application with kestrel, which is executed in a Kubernetes cluster.
However, if the application is running on a Windows or Linux host, the task prioritization seems to work perfectly. The service is responding even if there are dozens of CPU intensive requests. So there seems to be a significant difference between the Docker environment and the host environment.
I use this API method for test purposes:
public void SimulateHighCpuLoad()
{
var previousPriority = Thread.CurrentThread.Priority;
try
{
Thread.CurrentThread.Priority = ThreadPriority.Lowest;
var until = DateTime.Now.AddSeconds(30);
var num = 0L;
var random = new Random();
// do senseless work for 30 seconds
while (DateTime.Now < until)
{
num = (random.Next() + Environment.TickCount + num) % (random.Next(10000) + 1);
num *= num++;
}
}
finally
{
Thread.CurrentThread.Priority = previousPriority;
}
}
My goal is to prioritize CPU intensive methods lower so that the application can always respond to other requests (such as health requests for the LivenessProbe).
The Thread.Priority seems to be completely ignored within docker enviroment

According to this forum thread, we are not the only ones with this problem.
Thread scheduling seems to work fundamentally differently within a Docker container (or K8s cluster).
We tried a lot and were able to solve the problem with a mixture of throttling and Thread.Sleep(0).
Note the following documentation for Thread.Sleep
If the value of the millisecondsTimeout argument is zero, the thread
relinquishes the remainder of its time slice to any thread of equal
priority that is ready to run
Without Thread.Sleep(0), CPU-intensive threads cause that no other threads have been processed. No timers were called either. Setting a low thread priority is ineffective within a Docker container.

Using Parallel Processing in C# to test a site's ability to withstand a DDOS

I have a website and I am also exploring Parallel Processing in C# and I thought it would be a good idea to see if I could write my own DDOS test script to see how the site would handle a DDOS attack.
However when I run it, there only seems to be 13 threads in use and they always return 200 status codes, never anything to suggest the response wasn't quick and accurate and when going to the site and refreshing at the same time as the script runs the site loads quickly.
I know there are tools out there for penetration tests and so on but I was just wondering why I couldn't use a Parallel loop to make enough concurrent HTTP requests to a site that it would struggle to load fast and return a response. It seems I get more problems from a Twitter Rush just by tweeting out a link to a new page on the site and the 100s of BOTS that all rush concurrently to the site to rip, scan, check it etc than anything I can throw at it using a Parallel loop.
Is there something I am doing wrong that limits the number of concurrent threads or is this something I cannot control. I could just throw numerous long winded search queries that I know would scan the whole DB returning 0 results in each request as I have seen this in action and depending on the size of the data to be scanned and the complexity of the search query it can cause CPU spikes and slow loads.
So without a lecture on using other tools is there a way to throw a 100+ parallel requests for a page to be loaded rather than a max of 13 threads which it handles perfectly.
Here is the code, the URL and no of HTTP requests to make are passed in as command line parameters.
static void Attack(string url, int limit)
{
Console.WriteLine("IN Attack = {0}, requests = {1}", url, limit);
try
{
Parallel.For(0, limit, i =>
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.ServicePoint.ConnectionLimit = limit;
HttpWebResponse webResponse = webRequest.GetResponse() as HttpWebResponse;
int statuscode = Convert.ToInt32(webResponse.StatusCode);
Console.WriteLine("iteration {0} on thread {1} Status {2}", i,
Thread.CurrentThread.ManagedThreadId, statuscode);
});
}
catch (AggregateException exc)
{
exc.InnerExceptions.ToList().ForEach(e =>
{
Console.WriteLine(e.Message);
});
}
catch (Exception ex)
{
Console.WriteLine("In Exception: " + ex.Message.ToString());
}
finally
{
Console.WriteLine("All finished");
}
}

you can try it like:
var socketsHandler = new SocketsHttpHandler
{
PooledConnectionLifetime = TimeSpan.FromSeconds(1),
PooledConnectionIdleTimeout = TimeSpan.FromSeconds(1),
MaxConnectionsPerServer = 10
};
var client = new HttpClient(socketsHandler);
for (var i = 0; i < limit; i++)
{
_ = await client.GetAsync(url);
}

The Parallel.For method is using threads from the ThreadPool. The initial number of threads in the pool is usually small (comparable to the number of logical processors in the machine). When the pool is starved, new threads are injected at a rate of one every 500 msec. The easy way to solve your problem is simply to increase the number of the create-immediately-on-demand threads, using the SetMinThreads method:
ThreadPool.SetMinThreads(1000, 10);
This is not scalable though, because each thread allocates 1MB of memory for its stack, so you can't have millions of them. The scalable solution is to go async, which makes minimal use of threads.

Throttle/block async http request

I have a number of producer tasks that push data into a BlockingCollection, lets call it requestQueue.
I also have a consumer task that pops requests from the requestQueue, and forwards async http requests to a remote web service.
I need to throttle or block the number of active requests sent to the web service. On some machines that are far away from the service or have a slower internet connection, the http response time is long enough that the number of active requests fills up more memory than I'd like.
At the moment I am using a semaphore approach, calling WaitOne on the consumer thread multiple times, and Release on the HTTP response callback. Is there a more elegant solution?
I am bound to .net 4.0, and would like a standard library based solution.

You are already using a BlockingCollection why have a WaitHandle?
The way I would do it is to have a BlockingCollection with n as it's bounded capacity where n is the maximum number of concurrent requests you want to have at any given time.
You can then do something like....
var n = 4;
var blockingQueue = new BlockingCollection<Request>(n);
Action<Request> consumer = request =>
{
// do something with request.
};
var noOfWorkers = 4;
var workers = new Task[noOfWorkers];
for (int i = 0; i < noOfWorkers; i++)
{
var task = new Task(() =>
{
foreach (var item in blockingQueue.GetConsumingEnumerable())
{
consumer(item);
}
}, TaskCreationOptions.LongRunning | TaskCreationOptions.DenyChildAttach);
workers[i] = task;
workers[i].Start();
}
Task.WaitAll(workers);
I let you take care of cancellation and error handling but using this you can also control how many workers you want to have at any given time, if the workers are busy sending and processing the request any other producer will be blocked until more room is available in the queue.

Timeout Exception - Queuing of Requests? Not enough threads?

Background:
I have a service which aggregates data from multiple other services. To make things happen in a timely manner I use async throughout the code, and then gather the various requests into a list of tasks.
Here is some excerpts from the code:
private async Task<List<Foo>> Baz(..., int timeout)
{
var tasks = new List<Task<IEnumerable<Foo>>>();
Tasks.Add(GetFoo1(..., timeout));
Tasks.Add(GetFoo2(..., timeout));
// Up to 6, depending on other parameters. Some tasks return multiple objects.
return await Task.WhenAll(tasks).ContinueWith((antecedent) => { return antecedent.Result.AsEnumerable().SelectMany(f => f).ToList(); }).ConfigureAwait(false);
}
private async Task<IEnumerable<Foo>> GetFoo1(..., int timeout)
{
Stopwatch sw = new Stopwatch();
sw.Start();
var value = await SomeAsyncronousService.GetAsync(..., timeout).ConfigureAwait(false);
sw.Stop();
// Record timing...
return new[] { new Foo(..., value) };
}
private async Task<IEnumerable<Foo>> GetFoo2(..., int timeout)
{
return await Task.Run(() => {
Stopwatch sw = new Stopwatch();
sw.Start();
var r = new[] { new Foo(..., SomeSyncronousService.Get(..., timeout)) };
sw.Start();
sw.Stop();
// Record timing...
return r;
}).ConfigureAwait(false);
}
// In class SomeAsyncronousService
public async Task<string> GetAsync(..., int timeout)
{
...
try
{
using (var httpClient = HttpClientFactory.Create())
{
// I have tried it with both timeout and CTS. The behavior is the same.
//httpClient.Timeout = TimeSpan.FromMilliseconds(timeout);
var cts = new CancellationTokenSource();
cts.CancelAfter(timeout);
var content = ...;
var responseMessage = await httpClient.PostAsync(Endpoint, content, cts.Token).ConfigureAwait(false);
if (responseMessage.IsSuccessStatusCode)
{
var contentData = await responseMessage.Content.ReadAsStringAsync().ConfigureAwait(false);
...
return ...
}
...
}
}
catch (OperationCanceledException ex)
{
// Log statement ...
}
catch (Exception ex)
{
// Log statement ...
}
return ...;
}
The Symptoms:
This code works great on my local machine, and it works fine on our test servers most of the time. However, occasionally we get a bunch of mass recorded timeouts - recorded by the "Record timing" comments above and the Log statements on OperationCanceledExceptions. I do not have any way of telling if the services I call actually timed out.
Now, when I say a series of timeouts I mean that most or all of the tasks (and the HttpClients that all but one use, the other uses a WCF service) all timeout at about the same time.
Now, I know what you are thinking, I am passing in the same timeout. Thats right, but I pass in 250 ms and the run time that is being reported by the various stop watches are around 800 ms or higher.
Now, I do see the OperationCanceledExceptions in the log, but the time stamp of the exception is the same as the time stamp of when the stopwatch ended (or within 2-3 ms) and my service is failing because clients are expecting it to respond in 500 ms or less, not 800 ms.
Now, normally the various services respond in less than 100 ms, with a wide variance among the results. When we a problem occurs, and most / all return in 800 ms or more, they vary only by ~10 ms. The dependencies I call are all on different domains. It seems highly unlikely that all of them are really taking that long to respond, all at the same time.
I suppose there could be a network issue, affecting all requests at the same time, but the other services in our network do not experience the same behavior - it is limited to the new service I am writing.
Even if that was the case, I would expect the cancellation exceptions to occur after 250 ms, then for the task to end and the stopwatch to record 250 (plus 5-20ms or so for exception handling).
So I do not think that it is a network issue. Now I am sure that at least part of the problem is related to me not cancelling / timing out correctly, but it seem to me that all of the out going requests from the service are being affected at the same time independent of HttpClient.
The reason I say that is because the WCF service also shows 800+ ms (according to the stopwatch) when the rest of the requests timeout. The WCF service is not asynchronous. The timeout is set like this:
var binding = new BasicHttpBinding()
{
Security = new BasicHttpSecurity()
{
Mode = BasicHttpSecurityMode.TransportCredentialOnly,
Transport = new HttpTransportSecurity()
{
ClientCredentialType = HttpClientCredentialType.Ntlm
}
},
ReceiveTimeout = TimeSpan.FromMilliseconds(timeout)
};
The Problem:
So, in short I think that something is causing all outgoing requests to any domain to pause or queue which is causing the observed behavior.
I have spent days trying to figure out what is going on, but have had no luck. Any ideas?
EDIT
I think what is happening is that the requests are being put put on hold because there isn't a thread available, and then a few hundred milliseconds later a thread is available and the task starts. Timing the method call shows that it is taking 800 ms, but the timeout on the HttpClient doesn't start until a thread is available to run the async call.
It would also explain why I see that the method takes 800+ ms, but sometimes it still completes without showing a timeout exception. Other times it does throw a timeout exception and does not complete.
I have tried setting the ServicePointManager.DefaultConnectionLimit to 200 in Application_Start, but that did not solve the issue.
The service isn't taking that much traffic compared to our other services, and none of the others appear to have the same problem.
Any ideas?
Edit 2
I logged into the box and monitored netstat while doing (minor) load tests.
Using HttpClient, with 1-2 requests per second the ports would show ESTABLISHED, then move to TIME_WAIT for about 4 minutes. With 3+ requests per second I would end up with about a constant 100 x requests per second ESTABLISHED ports (so 300 for a 3 per second load test), and then I would start seeing them go to CLOSE_WAIT instead of TIME_WAIT - indicating an error condition on close. At the same time I would see the spike in the number of exceptions and time to execute the requests. (TcpTimedWaitDelay does not apply to CLOSE_WAIT).
So I rewrote the whole thing to use HttpWebRequests in serial, instead of HttpClient in parallel. Then I ran the same tests.
Now the ESTABLISHED ports equal 0-2 x requests per second, and the ports then move on to TIME_CLOSE as expected. The performance and throughput improved, but didn't clear up completely.
Then I set TcpTimedWaitDelay to 30 (default 240). The performance has increased dramatically. I have a primitive load test that hits it with 40 requests per second without any issues. I will get a more thorough test setup but I think the problem has been solved.
I don't know what is going on, but it appears that the HttpClient was not closing the ephemoral ports correctly underneath. Many of the developers and architects at my company looked at it and couldn't not see anything wrong with the code. I tried having a single HttpClient in a using statement per request, as well as having a single HttpClient per api I call on the back end. I have tried using HttpClient in parallel and serial. I have tried it with async/await and without. No matter what I tried the behavior was the same.
I would like to be able to use HttpClient, but I can't spend anymore time on this issue as I have it working with HttpWebRequest. My next step is to make the HttpWebRequests occur in Parallel.
Thank you for your input.

I've experienced similar frustrations with the HttpClient. In my scenario I found setting MaxServicePointIdleTime to a much lower value and DefaultConnectionLimit to a high value on the ServicePointManager resolved my issues. I believe in my case I was experiencing pool starvation as the connections were being held open.
You may also want to test without the debugger attached, in release, if you are not already doing so, as the TaskScheduler behaves differently when debugging.
The following MSDN article is very helpful: http://blogs.msdn.com/b/jpsanders/archive/2009/05/20/understanding-maxservicepointidletime-and-defaultconnectionlimit.aspx

Task.Factory.StartNew - confused about the pool

Hi I'm getting myself tied up with Task.Factory.StartNew. Just as I think I get the idea of it someone has suggested I write the following code;
bool exitLoop = false;
while (!exitLoop)
{
exitLoop = true;
var messages = Queue.GetMessages(20);
foreach (var message in messages)
{
exitLoop = false;
Task.Factory.StartNew(() =>
{
DeliverMessage(message);
});
}
}
In theory this is going to drain a queue, 20 messages at a time, attempting to creat a Task for every message in the queue. So if we had a 1000 messages in the queue then in an instant we'd have 25 tasks and it would eat its way through all the msgs. I previously thought I understood this, I thought StartNew would block once it ran out of entries - in the old days that would have been ~ 25. But given this is .net 4.5 which I'm now under the impression that the upper limit for a pool is now pretty high. What puzzles me is that I would have assumed that is going to flood the pool with new tasks and start blocking, i.e. in an instant I now have 1000 tasks running. So if the pool limit is now hardly a limit why am I not seeing 1000 tasks?
[Edit]
ok, so what I'm seeing is that 1000 tasks are queued to run, rather than are running. So how do I determine the number of running/runnable tasks?

I know this is quite a while after your post, but I hope this may help someone facing your specific challenge. Your last comment stated that the 'DeliverMessage' method was making HTTP requests.
If you are using the 'WebClient' object (for example) to make your requests, it will be bound by the ServicePointManager.DefaultConnectionLimit property. This means it will create at most two (by default) concurrent connections to the host. If you created 1,000 parallel tasks, all 1,000 of those would have to be serviced by those two connections.
You'll have to play around with different values for this setting to find the right balance between throughput in your application and load on the web server.