Develop Reference

How to send http requests in ForEach loop with async / await [duplicate]

This question already has answers here:
How can I use Async with ForEach?
(11 answers)
Running multiple async tasks and waiting for them all to complete
(10 answers)
Closed 2 months ago.
The following code is a linqpad program.
The payloadList contains json-objects like {"id": 1, "foo": "one" }.
Each object of payloadList should be sent to a server with httpClient.SendAsync()
The response for each request should be stored in responseList
The code below does partly work. But i do not understand why some parts are not working. I assume that the responses are not completed when responseList.Add(foo) is executed.
This request shall be send for each json-object {"id": 1, "foo": "one" }
public static async Task<string> DoRequest(HttpClient client, string payload)
{
var request = new HttpRequestMessage(HttpMethod.Post,
"http://httpbin.org/anything");
request.Content = new StringContent(payload
, Encoding.UTF8, "application/json");
var response = await client.SendAsync(request);
string responseContent = await response.Content.ReadAsStringAsync();
return responseContent;
}
The DoRequest()-method wraps the request and can be used inside main like this
static async Task Main()
{
var responseList = new List<string>();
var payloadList = new List<string>{"{ 'id': 1, 'bar': 'One'}",
"{ 'id': 2, 'bar': 'Two'}",
"{ 'id': 3, 'bar': 'Three'}"};
var client = new HttpClient();
payloadList.ForEach(async (payload) => {
var responseFoo = await DoRequest(client, payload);
responseFoo.Dump("response"); // contains responseFoo
responseList.Add(responseFoo); // adding responseFoo to list fails
});
responseList.Dump(); // is empty
}
The responseList is empty.
Expected responseList.Dump() contains all responses responseFoo.
Actual responseList is empty.
Questions
How can each response for await client.SendAsync(request) be added to a responseList?
Why is responseList empty despite that foo.Dump() works?
How to confirm or check if every client.SendAsync is finished?
Would you write the code above different - why?

List.ForEach is not Task-aware and will execute everything in parallel without waiting for the results (i.e. it will create tasks for all items in payloadList and will continue to the next statement responseList.Dump(); without awaiting them).
In newer versions of .NET you can use Parallel.ForEachAsync(for example as in this answer) combined with use of appropriate collection from System.Collections.Concurrent, for example ConcurrentBag<T>. List is not thread-safe and modifying it concurrently can lead to a lot of problems.
Code can look like the following:
var responseList = new ConcurrentBag<string>();
await Parallel.ForEachAsync(payloadList, async (payload, ct) =>
{
var foo = await DoRequest(client, payload, ct);
responseList.Add(foo);
});
static async Task<string> DoRequest(HttpClient client, string payload, CancellationToken ct)
{
var request = new HttpRequestMessage(HttpMethod.Post, "http://httpbin.org/anything");
request.Content = new StringContent(payload, Encoding.UTF8, "application/json");
var response = await client.SendAsync(request, ct);
string responseContent = await response.Content.ReadAsStringAsync(ct);
return responseContent;
}
If you are fine with all requests running in parallel - just create a enumerable of tasks and use Task.WhenAll<T>():
var tsks = payloadList
.Select(payload => DoRequest(client, payload));
string[] result = await Task.WhenAll(tsks);
If you want to execute requests one after another - just switch to ordinary foreach:
var responseList = new List<string>(); // no need for concurrent collection in this case
foreach (var payload in payloadList)
{
var foo = await DoRequest(client, payload);
responseList.Add(foo);
}
If you want to dive deeper here some links:
Asynchronous programming with async and await documentation
Asynchronous Programming in .NET - Introduction, Misconceptions, and Problems
Threading in C# by Joseph Albahari (a bit dated but still amazing)

How to POST to a rest api and consume the non ending stream in c#?

I need to consume a stream of a rest api. The thing is that this stream is continuous and i need to POST some JSON to it, so it knows what to stream back.
Can you guys help me? This is what i have right now, but the problem is that the _httpClient.PostAsync() is never completing.
var content = new StringContent("whatever", Encoding.UTF8, "application/json");
var response = await _httpClient.PostAsync("URL", content);
//not reaching this point here
var stream = await response.Content.ReadAsStreamAsync();

Not sure if it will help you, but you can tell the HTTPClient whether to wait for the complete response, or only for the headers, if you use the SendAsync method:
var request = new HttpRequestMessage(HttpMethod.Post, "URL");
request.Content = new StringContent("whatever", Encoding.UTF8, "application/json");
var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
var stream = await response.Content.ReadAsStreamAsync();

Although it's not recommended to use http for this purpose, but you can achieve it with this code
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, "URL");
request.Content = new StringContent("whatever", Encoding.UTF8, "application/json");
await _httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
the key part here is HttpCompletionOption.ResponseHeadersRead which completes SendAsync as soon as the response stream is ready.

ASP.Net Forms HttpClient PostAsync no response, no error message

I have this code:
using (var client = new HttpClient())
{
var values = new Dictionary<string, string>
{
{"site_id","001"},
{"apikey","abc01201az1024"},
{"trans_id","45364136"},
};
// Get the parameters in the url encoded format
var content = new FormUrlEncodedContent(values);
//Send request
var response = await client.PostAsync(url, content);
DataRoot<Transaction> outPut = null;
if (response.IsSuccessStatusCode)
{
//Get Response
var result = await response.Content.ReadAsStringAsync();
JsonConvert.DeserializeObject<DataRoot<Transaction>>(result);
}
return outPut;
}
In the debug mode at this stage, the code does not produce any response, no error code but stops running:
//Send request
var response = await client.PostAsync(url, content);

there could be a better way but this solved my problem. Call your method from another one using wait:-
public static async Task<string> AuthenticateUser()
{
var t = Task.Run(() => ClassObject.AuthenticateUser("me"));
t.Wait();
Console.WriteLine(t.Result);
Console.ReadLine();
return "ok";
}

Using await like this, can end up in a deadlock.
You can use ConfigureAwait(false) in async methods for preventing such a deadlock.
Update code to:
var response = await client.PostAsync(url, content).ConfigureAwait(false);
This will solve the issue.

Are you sure that it isn't actually returning the response and continuing execution? Because the client.PostAsync() call is awaited execution may continue on a different thread. Therefore, if you're just debugging line by line (via F10 or similar) it may appear that the method never returns; in actuality the entire method has finished execution and your program is running.
You may need to add another breakpoint in the method (after the PostAsync method call). When the PostAsync method returns on a different thread, your debugger should hit the next breakpoint.

How can I get JSON data from Google API and store it in a variable?

I am trying to send an Http Get message to the Google location Api which is supposed to have Json data such as this
https://maps.googleapis.com/maps/api/geocode/json?address=Los%20Angeles,CA=AIzaSyDABt
as you noticed the response is in Json. I want to give a a Http Call to that URL and save the Json content in a variable or string. My code does not give out any errors but it is also not returning anything
public async System.Threading.Tasks.Task<ActionResult> GetRequest()
{
var client = new HttpClient();
HttpResponseMessage response = await client.GetAsync("https://maps.googleapis.com/maps/api/geocode/json?address=Los%20Angeles,CA=AIzaSyDABt");
string data = response.Content.ToString();
return data;
}
I want to send out a Get Request using HttpClient() or anything that will send out the URL request and then save that content into a string variable . Any suggestions would be appreciated, again my code gives no errors but it is not returning anything.

Use ReadAsStringAsync to get the json response...
static void Main(string[] args)
{
HttpClient client = new HttpClient();
Task.Run(async () =>
{
HttpResponseMessage response = await client.GetAsync("https://maps.googleapis.com/maps/api/geocode/json?address=Los%20Angeles,CA=AIzaSyDABt");
string responseString = await response.Content.ReadAsStringAsync();
Console.WriteLine(responseString);
});
Console.ReadLine();
}
If you use response.Content.ToString() it is actually converting the datatype of the Content to string so you will get System.Net.Http.StreamContent

Try this.
var client = new HttpClient();
HttpResponseMessage httpResponse = await client.GetAsync("https://maps.googleapis.com/maps/api/geocode/json?address=Los%20Angeles,CA=AIzaSyDABt");
string data = await httpResponse.Content.ReadAsStringAsync();

How about using the Json framework for .NET powered by Newtonsoft ?
You can try to parse your content to a string with this.

Well, your code can be simplified:
public async Task<ActionResult> GetRequest()
{
var client = new HttpClient();
return await client.GetStringAsync("https://maps.googleapis.com/maps/api/geocode/json?address=Los%20Angeles,CA=AIzaSyDABt");
}
However...
My code does not give out any errors but it is also not returning anything
This is almost certainly due to using Result or Wait further up the call stack. Blocking on asynchronous code like that causes a deadlock that I explain in full on my blog. The short version is that there is an ASP.NET request context that only permits one thread at a time; await by default will capture the current context and resume in that context; and since Wait/Result is blocking a thread in the request context, the await cannot resume executing.

Try this
public async static Task<string> Something()
{
var http = new HttpClient();
var url = "https://maps.googleapis.com/maps/api/geocode/json?address=Los%20Angeles,CA=AIzaSyDABt";
var response = await http.GetAsync(url);
if (response.IsSuccessStatusCode)
{
var result = await response.Content.ReadAsStringAsync();
result = JsonConvert.DeserializeObject<string>(result);
return result;
}
return "";
}
var result = Task.Run(() => Something()).Result;

Use HttpClient to send two POST requests 2 seconds apart

I'm trying to write an integration test for a DelegatingHandler that prevents duplicate requests. The handler checks a database to see if the request is already being processed and returns a 407-Conflict if a duplicate request is made while the previous request is still running.
I have the following code in my test:
HttpClient client = new HttpClient();
var responseTask1 = client.PostAsJsonAsync(RequestUriWithDuplicatePrevention, ReadRequestContent("DuplicateRequestJsonContent.json"));
var responseTask2 = client.PostAsJsonAsync(RequestUriWithDuplicatePrevention, ReadRequestContent("DuplicateRequestJsonContent.json"));
var response1 = responseTask1.Result;
var response2 = responseTask2.Result;
Both requests are being logged into the database at the exact same time. How can I delay the second request for a period of time?
I've tried adding a Thread.Sleep(500) but it didn't seem to make a difference.
Revised Code
This code seems to work most of the time but it not 100% reliable.
[TestMethod]
public void ShouldReturn407ConflictWhenDuplicateRequestSubmitted()
{
var results = ExecutePostRequests().Result;
Assert.AreEqual(HttpStatusCode.OK, results[0].StatusCode);
Assert.AreEqual(HttpStatusCode.Conflict, results[1].StatusCode);
}
private async Task<HttpResponseMessage[]> ExecutePostRequests()
{
HttpClient client = new HttpClient();
var task1 = ExecutePost(client, 0);
var task2 = ExecutePost(client, 4000);
var response1 = await task1;
var response2 = await task2;
return new[] {response1, response2};
}
private async Task<HttpResponseMessage> ExecutePost(HttpClient client, int delay)
{
await Task.Delay(delay);
return await client.PostAsync(RequestUriWithDuplicatePrevention,
ReadRequestContent("DuplicateRequestJsonContent.json"));
}
The web service being executed has a Thread.Sleep(5000).

The specific problem in your original code is that it is sleeping between getting results, when you should be sleeping between starting the asynchronous operations.
It could be corrected like so:
var responseTask1 = client.PostAsJsonAsync(...);
Thread.Sleep(2000);
var responseTask2 = client.PostAsJsonAsync(...);
var response1 = responseTask1.Result;
var response2 = responseTask2.Result;
Your revised code does not suffer this issue and should work. Though, I would change this:
var response1 = await task1;
var response2 = await task2;
return new[] {response1, response2};
To a more efficient:
return await Task.WhenAll(task1, task2);