Develop Reference

Run HttpClient POST in parallel

I am using .net HTTPClient to save some data using POST REST API. The payload size for this API is around 10 MB. I am splitting my data in chunks and calling this POST API for each chunk. I have question mostly around approach:
I am planning to create single static instance of HTTPClient and will use same instance across application. What should be my approach? (create singleton or new client per chunk POST API call)
I would like to call all these chunk POST calls in parallel (using TASKS in .net). Is there any way to stop remaining tasks if any one task fails. I am looking for some sample code.
_factory = new TaskFactory();
_factory.StartNew(() =>
//Call to async POST API using HttpClient
).ContinueWith((response) =>
{
if (!response.IsFaulted)
{
//Do something
}
else {
this._logger.Error("log the error");
}
});

If your calls are all to the same host, use a shared HttpClient instance.
There is no need to explicitly Create tasks using TaskFactory.StartNew in order to do I/O-bound asynchronous work in parallel. I would suggest using Task.WhenAll. Something like this:
try {
await Task.WhenAll(chunks.Select(MakeCall));
}
catch (Exception) {
_client.CancelPendingRequests();
}
private async Task MakeCall(string chunk) {
var response = await _client.PostAsync(chunk);
if (!response.IsFaulted) {
//Do something
}
else {
this._logger.Error("log the error");
throw new Exception("call failed!");
}
}

C# async await for pooling

I need to do some WebRequest to a certain endpoint every 2 seconds. I tried to do it with a Timer, the problem is that every call to the callback function is creating a different Thread and I'm havind some concurrence problems. So I decided to change my implementation and I was thinking about using a background worker with a sleep of two seconds inside or using async await but I don't see the advantages of using async await. Any advice? thank you.
This is the code that I will reimplement.
private void InitTimer()
{
TimerCallback callback = TimerCallbackFunction;
m_timer = new Timer(callback, null, 0, m_interval);
}
private void TimerCallbackFunction(Object info)
{
Thread.CurrentThread.Name = "Requester thread ";
m_object = GetMyObject();
}
public MyObject GetMyObject()
{
MyObject myobject = new MyObject();
try
{
MemoryStream responseInMemory = CreateWebRequest(m_host, ENDPOINT);
XmlSerializer xmlSerializer = new XmlSerializer(typeof(MyObject));
myObject = (MyObject) xmlSerializer.Deserialize(responseInMemory);
}
catch (InvalidOperationException ex)
{
m_logger.WriteError("Error getting MyObject: ", ex);
throw new XmlException();
}
return myObject;
}
private MemoryStream CreateWebRequest(string host, string endpoint)
{
WebRequest request = WebRequest.Create(host + endpoint);
using (var response = request.GetResponse())
{
return (MemoryStream) response.GetResponseStream();
}
}
EDIT: I have read this SO thread Async/await vs BackgroundWorker

async await is also concurrence. If you have concurrence problems and you want your application to have only one thread, you should avoid using async await.
However the best way to do WebRequest is to use async await, which does not block the main UI thread.
Use the bellow method, it will not block anything and it is recommended by Microsoft. https://msdn.microsoft.com/en-us/library/86wf6409(v=vs.110).aspx
private async Task<MemoryStream> CreateWebRequest(string host, string endpoint)
{
WebRequest request = WebRequest.Create(host + endpoint);
using (var response = await request.GetResponseAsync())
{
return (MemoryStream)response.GetResponseStream();
}
}

You don't mention what the concurrency problems are. It may be that the request takes so long that the next one starts before the previous one finishes. It could also be that the callback replaces the value in my_Object while readers are accessing it.
You can easily make a request every X seconds, asynchronously and without blocking, by using Task.Delay, eg:
ConcurrentQueue<MyObject> m_Responses=new ConcurrentQueue<MyObject>();
public async Task MyPollMethod(int interval)
{
while(...)
{
var result=await SomeAsyncCall();
m_Responses.Enqueue(result);
await Task.Delay(interval);
}
}
This will result in a polling call X seconds after the last one finishes.
It also avoids concurrency issues by storing the result in a concurrent queue instead of replacing the old value, perhaps while someone else was reading int.
Consumers of MyObject would call Dequeue to retrieve MyObject instances in the order they were received.
You could use the ConcurrentQueue to fix the current code too:
private void TimerCallbackFunction(Object info)
{
Thread.CurrentThread.Name = "Requester thread ";
var result=GetMyObject();
m_Responses.Enqueue(result);
}
or
private async void TimerCallbackFunction(Object info)
{
Thread.CurrentThread.Name = "Requester thread ";
var result=await GetMyObjectAsync();
m_Responses.Enqueue(result);
}
if you want to change your GetObject method to work asynchronously.
Since your request seems to take a long time, it's a good idea to make it asynchronous and avoid blocking the timer's ThreadPool thread while waiting for a network response.

Limit number of parallel running methods

In my class I have a download function. Now in order to not allow a too high number of concurrent downloads, I would like to block this function until a "download-spot" is free ;)
void Download(Uri uri)
{
currentDownloads++;
if (currentDownloads > MAX_DOWNLOADS)
{
//wait here
}
DoActualDownload(uri); // blocks long time
currentDownloads--;
}
Is there a ready made programming pattern for this in C# / .NET?
edit: unfortunatelyt i cant use features from .net4.5 but only .net4.0

May be this
var parallelOptions = new ParallelOptions
{
MaxDegreeOfParallelism = 3
};
Parallel.ForEach(downloadUri, parallelOptions, (uri, state, index) =>
{
YourDownLoad(uri);
});

You should use Semaphore for this concurrency problem, see more in the documentation:
https://msdn.microsoft.com/en-us/library/system.threading.semaphore(v=vs.110).aspx

For such cases, I create an own Queue<MyQuery> in a custom class like QueryManager, with some methods :
Each new query is enqueued in Queue<MyQuery> queries
After each "enqueue" AND in each query answer, I call checkIfQueryCouldBeSent()
The checkIfQueryCouldBeSent() method checks your conditions : number of concomitant queries, and so on. In your case you accept to launch a new query if global counter is less than 5. And you increment the counter
Decrement the counter in query answer
It works only if all your queries are asynchronous.
You have to store Callback in MyQuery class, and call it when query is over.

You're doing async IO bound work, there's no need to be using multiple threads with a call such as Parallel.ForEach.
You can simply use naturally async API's exposed in the BCL, such ones that make HTTP calls using HttpClient. Then, you can throttle your connections using SemaphoreSlim and it's WaitAsync method which asynchronously waits:
private readonly SemaphoreSlim semaphoreSlim = new SemaphoreSlim(3);
public async Task DownloadAsync(Uri uri)
{
await semaphoreSlim.WaitAsync();
try
{
string result = await DoActualDownloadAsync(uri);
}
finally
{
semaphoreSlim.Release();
}
}
And your DoActualyDownloadAsync will use HttpClient to do it's work. Something along the lines of:
public Task<string> DoActualDownloadAsync(Uri uri)
{
var httpClient = new HttpClient();
return httpClient.GetStringAsync(uri);
}

How do you create an asynchronous method in C#?

Every blog post I've read tells you how to consume an asynchronous method in C#, but for some odd reason never explain how to build your own asynchronous methods to consume. So I have this code right now that consumes my method:
private async void button1_Click(object sender, EventArgs e)
{
var now = await CountToAsync(1000);
label1.Text = now.ToString();
}
And I wrote this method that is CountToAsync:
private Task<DateTime> CountToAsync(int num = 1000)
{
return Task.Factory.StartNew(() =>
{
for (int i = 0; i < num; i++)
{
Console.WriteLine("#{0}", i);
}
}).ContinueWith(x => DateTime.Now);
}
Is this, the use of Task.Factory, the best way to write an asynchronous method, or should I write this another way?

I don't recommend StartNew unless you need that level of complexity.
If your async method is dependent on other async methods, the easiest approach is to use the async keyword:
private static async Task<DateTime> CountToAsync(int num = 10)
{
for (int i = 0; i < num; i++)
{
await Task.Delay(TimeSpan.FromSeconds(1));
}
return DateTime.Now;
}
If your async method is doing CPU work, you should use Task.Run:
private static async Task<DateTime> CountToAsync(int num = 10)
{
await Task.Run(() => ...);
return DateTime.Now;
}
You may find my async/await intro helpful.

If you didn't want to use async/await inside your method, but still "decorate" it so as to be able to use the await keyword from outside, TaskCompletionSource.cs:
public static Task<T> RunAsync<T>(Func<T> function)
{
if (function == null) throw new ArgumentNullException(“function”);
var tcs = new TaskCompletionSource<T>();
ThreadPool.QueueUserWorkItem(_ =>
{
try
{
T result = function();
tcs.SetResult(result);
}
catch(Exception exc) { tcs.SetException(exc); }
});
return tcs.Task;
}
From here and here
To support such a paradigm with Tasks, we need a way to retain the Task façade and the ability to refer to an arbitrary asynchronous operation as a Task, but to control the lifetime of that Task according to the rules of the underlying infrastructure that’s providing the asynchrony, and to do so in a manner that doesn’t cost significantly. This is the purpose of TaskCompletionSource.
I saw it's also used in the .NET source, e.g. WebClient.cs:
[HostProtection(ExternalThreading = true)]
[ComVisible(false)]
public Task<string> UploadStringTaskAsync(Uri address, string method, string data)
{
// Create the task to be returned
var tcs = new TaskCompletionSource<string>(address);
// Setup the callback event handler
UploadStringCompletedEventHandler handler = null;
handler = (sender, e) => HandleCompletion(tcs, e, (args) => args.Result, handler, (webClient, completion) => webClient.UploadStringCompleted -= completion);
this.UploadStringCompleted += handler;
// Start the async operation.
try { this.UploadStringAsync(address, method, data, tcs); }
catch
{
this.UploadStringCompleted -= handler;
throw;
}
// Return the task that represents the async operation
return tcs.Task;
}
Finally, I also found the following useful:
I get asked this question all the time. The implication is that there must be some thread somewhere that’s blocking on the I/O call to the external resource. So, asynchronous code frees up the request thread, but only at the expense of another thread elsewhere in the system, right? No, not at all.
To understand why asynchronous requests scale, I’ll trace a (simplified) example of an asynchronous I/O call. Let’s say a request needs to write to a file. The request thread calls the asynchronous write method. WriteAsync is implemented by the Base Class Library (BCL), and uses completion ports for its asynchronous I/O. So, the WriteAsync call is passed down to the OS as an asynchronous file write. The OS then communicates with the driver stack, passing along the data to write in an I/O request packet (IRP).
This is where things get interesting: If a device driver can’t handle an IRP immediately, it must handle it asynchronously. So, the driver tells the disk to start writing and returns a “pending” response to the OS. The OS passes that “pending” response to the BCL, and the BCL returns an incomplete task to the request-handling code. The request-handling code awaits the task, which returns an incomplete task from that method and so on. Finally, the request-handling code ends up returning an incomplete task to ASP.NET, and the request thread is freed to return to the thread pool.
Introduction to Async/Await on ASP.NET
If the target is to improve scalability (rather than responsiveness), it all relies on the existence of an external I/O that provides the opportunity to do that.

One very simple way to make a method asynchronous is to use Task.Yield() method. As MSDN states:
You can use await Task.Yield(); in an asynchronous method to force the
method to complete asynchronously.
Insert it at beginning of your method and it will then return immediately to the caller and complete the rest of the method on another thread.
private async Task<DateTime> CountToAsync(int num = 1000)
{
await Task.Yield();
for (int i = 0; i < num; i++)
{
Console.WriteLine("#{0}", i);
}
return DateTime.Now;
}

How to pass parameter to asynchronous task in c#

During an ASP.NET web request I need to kick off an asynchronous task and then return control back to the web page. Can someone show me the code for calling the below method? Just to be clear, I want the function called async and the web request to complete returning control to the calling page while the function is still processing.
private void FunctionToCall(IInterface objectIWantToPassIn)
{
// do stuff
}

You'll want to spawn the thread by creating a
Task task = Task.Factory.StartNew(action, "arg");
then you'll want to do something maybe when the task is done:
task.ContinueWith(anotheraction);
Of course action and otheraction would be void methods in this case.

private void FunctionToCall(object state)
{
IInterface objectIWantToPassIn= (IInterface ) state;
// do stuff
}
System.Threading.ThreadPool.QueueUserWorkItem(FunctionToCall,
objectIWantToPassIn);