Develop Reference

Is it better to use HttpClient or WebRequest in ASP.NET Core to read the content of a file line by line asynchronously remotely?

I plan to read a remote file line by line asynchronously using https://github.com/Dasync/AsyncEnumerable (since there is not yet Async Streams [C# 8 maybe]: https://github.com/dotnet/csharplang/blob/master/proposals/async-streams.md):
public static class StringExtensions
{
public static AsyncEnumerable<string> ReadLinesAsyncViaHttpClient(this string uri)
{
return new AsyncEnumerable<string>(async yield =>
{
using (var httpClient = new HttpClient())
{
using (var responseStream = await httpClient.GetStreamAsync(uri))
{
using (var streamReader = new StreamReader(responseStream))
{
while(true)
{
var line = await streamReader.ReadLineAsync();
if (line != null)
{
await yield.ReturnAsync(line);
}
else
{
return;
}
}
}
}
}
});
}
public static AsyncEnumerable<string> ReadLinesAsyncViaWebRequest(this string uri)
{
return new AsyncEnumerable<string>(async yield =>
{
var request = WebRequest.Create(uri);
using (var response = request.GetResponse())
{
using (var responseStream = response.GetResponseStream())
{
using (var streamReader = new StreamReader(responseStream))
{
while(true)
{
var line = await streamReader.ReadLineAsync();
if (line != null)
{
await yield.ReturnAsync(line);
}
else
{
return;
}
}
}
}
}
});
}
}
It seems that they both run just fine in a simple Console application like below:
public class Program
{
public static async Task Main(string[] args)
{
// Or any other remote file
const string url = #"https://gist.githubusercontent.com/dgrtwo/a30d99baa9b7bfc9f2440b355ddd1f75/raw/700ab5bb0b5f8f5a14377f5103dbe921d4238216/by_tag_year.csv";
await url.ReadLinesAsyncViaWebRequest().ForEachAsync(line =>
{
Console.WriteLine(line, Color.GreenYellow);
});
await url.ReadLinesAsyncViaHttpClient().ForEachAsync(line =>
{
Console.WriteLine(line, Color.Purple);
});
}
}
... but I have some concerns if it is used as part of an ASP.NET Core WebAPI to process the lines and then push them using PushStreamContent:
https://learn.microsoft.com/en-us/previous-versions/aspnet/hh995285(v=vs.108)
https://blog.stephencleary.com/2016/10/async-pushstreamcontent.html
The idea would be to have a pipeline of data which leverages async / await so that the number of threads in use is as low as possible and also to avoid an increase in memory (which leverage the enumerable-like feature of AsyncEnumerable).
I read several articles but it seems it's all non .NET Core versions and I don't really know if there would be some potential performance issues / caveats in regard to what I would like to achieve?
Difference between HttpRequest, HttpWebRequest and WebRequest
http://www.diogonunes.com/blog/webclient-vs-httpclient-vs-httpwebrequest/
An example of "business" case would be:
using System;
using System.Collections.Async;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
namespace WebApplicationTest.Controllers
{
[Route("api/[controller]")]
[ApiController]
public class DumbValuesController : ControllerBase
{
private static readonly Random Random = new Random();
// GET api/values
[HttpGet]
public async Task<IActionResult> DumbGetAsync([FromQuery] string fileUri)
{
using (var streamWriter = new StreamWriter(HttpContext.Response.Body))
{
await fileUri.ReadLinesAsyncViaHttpClient().ForEachAsync(async line =>
{
// Some dumb process on each (maybe big line)
line += Random.Next(0, 100 + 1);
await streamWriter.WriteLineAsync(line);
});
}
return Ok();
}
}
}

We have access to the source code for .NET Core. So you can look.
The underlying implementation of both end up using HttpClientHandler (the implementation of that class is split up into 4 files).
You can see this from the source code of both HttpClient and HttpWebRequest (which WebRequest uses).
So I suspect you won't notice any difference in the performance of either.
HttpClient is the latest one to be written, so that's why its use is encouraged. And for the reasons mentioned in the article you linked to: http://www.diogonunes.com/blog/webclient-vs-httpclient-vs-httpwebrequest/

With the latest release of .Net Core 6.0, WebRequest will be declared as deprecated. Microsoft recommended to use HttpClient instead
https://learn.microsoft.com/en-us/dotnet/core/compatibility/networking/6.0/webrequest-deprecated

C# stops working when performing async post request

I am working on a mobile app, and the problem is that when I perform a async request ( PostAsync ) using Net.Http my program stops running.
Here is my request class, where I perform the requests using Net.Http.
...
namespace BSoft.Requests
{
public class Requests
{
public Requests(){}
public static string HostName = "https://dev5.360businesssoft.com/";
private static readonly HttpClient httpClient = new HttpClient();
public static async Task<string> PerformPostRequest(Dictionary<string, string> values, string path)
{
string url = HostName + path;
FormUrlEncodedContent content = new FormUrlEncodedContent(values);
HttpResponseMessage response = await httpClient.PostAsync(url, content);
string responseString = await response.Content.ReadAsStringAsync();
return responseString;
}
}
}
and here is my login class, where I call call the request and display the result as string.
...
namespace BSoft.Login
{
public class Login
{
public Login()
{
}
public static void PerformLogin(string username, string password, bool remember)
{
var values = new Dictionary<string, string>();
values.Add("User", username);
values.Add("Password", password);
var ReturnedObj = Requests.Requests.PerformPostRequest(values, "test.php").Result;
System.Diagnostics.Debug.WriteLine(ReturnedObj);
}
}
}
This is a screenshot of the app, you can notice that the button is freezed

The call to Result is blocking the gui thread. Instead, await the result:
var ReturnedObj = await Requests.Requests.PerformPostRequest(values, "test.php");
System.Diagnostics.Debug.WriteLine(ReturnedObj);
Your call to Result will block the gui thread until PerformPostRequest completes, so there's not really a lot of point using the async features here. If you really don't want the code to execute asynchronously then you might as well remove the calls to the async methods and make the calls synchronous.

Try
string returnedString = await Requests.Requests.PerformPostRequest(values, "test.php");

HttpClient Async Method return value

I studied over the Internet regarding Task Async method but cannot seem to find an approach to assign my return value in Task Async to another object. The first method is to prepare HTTP Request header and Uri.
public static async Task MainAsync()
{
string token = await AuthHelper.AcquireToken(tenantId, clientId, clientSecret);
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Authorization", "Bearer " + token);
client.BaseAddress = new Uri("https://foo.net");
await GetValue(client);
}
}
The second method is to use GetAsync to call to an API to get the JSON and the two last lines I extract only value from the "Value" field in the JSON body.
public static async Task<String> GetValue(HttpClient client)
{
string url = $"/mykey/key01";
using (var httpResponse = await client.GetAsync(url))
{
httpResponse.EnsureSuccessStatusCode();
string responsContent = await httpResponse.Content.ReadAsStringAsync();
JObject json = JObject.Parse(responsContent);
string value = json["value"].ToString();
return value;
}
}
Now I would like to use this value to assign to another object, but not sure how to do so. I managed to return the valid value. Is it possible to retrieve the value from another method or even different class?
[Updated] The main function is:
static void Main(string[] args)
{
try
{
MainAsync().Wait();
}
catch (Exception e)
{
Console.WriteLine(e.GetBaseException().Message);
}
}
Update
To be more clear. The HTTP response message is a JSON format and I can return the value from Value property in this JSON. Now how I can to reuse the value from an external method or class

I'm not sure exactly what you are trying to achieve. And there would be thorough debates about your architecture, you can do something like this..
Update
Because your MainAsync is static it can be called form anywhere.
You just need to modify it a bit to return your result as follows :
public static async Task<string> MainAsync()
{
...
return await GetValue(client);
...
And somewhere else
public class MyAwesomeClass
{
public async Task DoMagic()
{
var newValueOfSomething = await MainAsync();
// hilarity ensues
}
}

You can Make it more generic and useful, something like below :
Your initial method can be changes to :
public async Task<T> GetContentAsync<T>(HttpClient client)
{
string url = $"/mykey/key01";
using (var httpResponse = await client.GetAsync(url))
{
httpResponse.EnsureSuccessStatusCode();
string responsContent = await httpResponse.Content.ReadAsStringAsync();
return Deserialize<T>(json);
}
}
private T Deserialize<T>(string json)
{
return JsonConvert.DeserializeObject<T>(json, SerializationSettings);
}
You can now call method like :
var person = await GetContentAsync<Person>(/*http client*/)

HttpClient vs HttpWebRequest for better performance, security and less connections

I discovered that a single HttpClient could be shared by multiple requests. If shared, and the requests are to the same destination, multiple requests could reuse the connections. WebRequest needs to recreate the connection for each request.
I also looked up some documentation on other ways to use HttpClient in examples.
The following article summarizes the high-speed NTLM-authenticated connection sharing: HttpWebRequest.UnsafeAuthenticatedConnectionSharing
 
Possible implementations that I tried out are shown below
A)
private WebRequestHandler GetWebRequestHandler()
{
CredentialCache credentialCache = new CredentialCache();
credentialCache.Add(ResourceUriCanBeAnyUri, "NTLM", CredentialCache.DefaultNetworkCredentials);
WebRequestHandler handler = new WebRequestHandler
{
UnsafeAuthenticatedConnectionSharing = true,
Credentials = credentialCache
};
return handler;
}
using (HttpClient client = new HttpClient(GetWebRequestHandler(), false))
{
}
B)
using (HttpClient client = new HttpClient)
{
}
C)
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("some uri string")
I would appreciate any help in making me understand which approach I should take so as to achieve max performance, minimizing connections and making sure security is not impacted.

If you use either of them with async it should be good for the performance point of view as it will not block the resources waiting for the response and you will get good throughput.
HttpClient is preferred over HttpWebRequest due to async methods available out of the box and you would not have to worry about writing begin/end methods.
Basically when you use async call (using either of the class), it will not block the resources waiting for the response and any other request would utilise the resources to make further calls.
Another thing to keep in mind that you should not be using HttpClient in the 'using' block to allow reuse of same resources again and again for other web requests.
See following thread for more information
Do HttpClient and HttpClientHandler have to be disposed?

This is my ApiClient which creates the HttpClient for only once. Register this object as singleton to your dependency injection library. It's safe to reuse because it's stateless. Do NOT recreate HTTPClient for each request. Reuse Httpclient as much as possible
using System;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Threading.Tasks;
//You need to install package Newtonsoft.Json > https://www.nuget.org/packages/Newtonsoft.Json/
using Newtonsoft.Json;
using Newtonsoft.Json.Serialization;
public class MyApiClient : IDisposable
{
private readonly TimeSpan _timeout;
private HttpClient _httpClient;
private HttpClientHandler _httpClientHandler;
private readonly string _baseUrl;
private const string ClientUserAgent = "my-api-client-v1";
private const string MediaTypeJson = "application/json";
public MyApiClient(string baseUrl, TimeSpan? timeout = null)
{
_baseUrl = NormalizeBaseUrl(baseUrl);
_timeout = timeout ?? TimeSpan.FromSeconds(90);
}
public async Task<string> PostAsync(string url, object input)
{
EnsureHttpClientCreated();
using (var requestContent = new StringContent(ConvertToJsonString(input), Encoding.UTF8, MediaTypeJson))
{
using (var response = await _httpClient.PostAsync(url, requestContent))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
}
public async Task<TResult> PostAsync<TResult>(string url, object input) where TResult : class, new()
{
var strResponse = await PostAsync(url, input);
return JsonConvert.DeserializeObject<TResult>(strResponse, new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver()
});
}
public async Task<TResult> GetAsync<TResult>(string url) where TResult : class, new()
{
var strResponse = await GetAsync(url);
return JsonConvert.DeserializeObject<TResult>(strResponse, new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver()
});
}
public async Task<string> GetAsync(string url)
{
EnsureHttpClientCreated();
using (var response = await _httpClient.GetAsync(url))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
public async Task<string> PutAsync(string url, object input)
{
return await PutAsync(url, new StringContent(JsonConvert.SerializeObject(input), Encoding.UTF8, MediaTypeJson));
}
public async Task<string> PutAsync(string url, HttpContent content)
{
EnsureHttpClientCreated();
using (var response = await _httpClient.PutAsync(url, content))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
public async Task<string> DeleteAsync(string url)
{
EnsureHttpClientCreated();
using (var response = await _httpClient.DeleteAsync(url))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
public void Dispose()
{
_httpClientHandler?.Dispose();
_httpClient?.Dispose();
}
private void CreateHttpClient()
{
_httpClientHandler = new HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip
};
_httpClient = new HttpClient(_httpClientHandler, false)
{
Timeout = _timeout
};
_httpClient.DefaultRequestHeaders.UserAgent.ParseAdd(ClientUserAgent);
if (!string.IsNullOrWhiteSpace(_baseUrl))
{
_httpClient.BaseAddress = new Uri(_baseUrl);
}
_httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(MediaTypeJson));
}
private void EnsureHttpClientCreated()
{
if (_httpClient == null)
{
CreateHttpClient();
}
}
private static string ConvertToJsonString(object obj)
{
if (obj == null)
{
return string.Empty;
}
return JsonConvert.SerializeObject(obj, new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver()
});
}
private static string NormalizeBaseUrl(string url)
{
return url.EndsWith("/") ? url : url + "/";
}
}
The usage;
using ( var client = new MyApiClient("http://localhost:8080"))
{
var response = client.GetAsync("api/users/findByUsername?username=alper").Result;
var userResponse = client.GetAsync<MyUser>("api/users/findByUsername?username=alper").Result;
}

There is a problem in your implementation 'A'. The lifetime of the instance returned from GetWebRequestHandler() is shortlived (maybe just for the sake of the example?). If this was done on purpose, it negates the passing of false for the 2nd param of HttpClient constructor. The value of false tells the HttpClient to not dispose the underlying HttpMessageHandler (which helps with scaling since it does not close the port for the request). This is, of course, assuming that the lifetime of the HttpMessageHandler is long enough for you to take advantage of the benefit of not opening/closing ports (which is a big impact on the scalability of your server). Thus, I have a recommendation below of option 'D'.
There is also an option 'D' that you do not list above - to make the Httpclient instance static and re-used across all api calls. This is much more efficient from a memory allocation and GC perspective - as well as the opening of ports on the client. You don't have the overhead of allocating memory for and creating HttpClient instances (and all of its underlying objects) and thus avoid the cleanup via GC for them too.
Please refer to my answer provided about a similar question - What is the overhead of creating a new HttpClient per call in a WebAPI client?

Async task does not end

I'm trying to start async task (on .NET 4.5) which downloads content of web page, but somehow this task never finishes.
My PageDownloader class:
using System.Net;
using System.Text;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;
using System;
namespace ParserConsole.WebClient
{
public class PageDownloader
{
private System.Net.Http.HttpClient _client;
public PageDownloader()
: this(Encoding.UTF8) { }
private Encoding _encoding;
public PageDownloader(Encoding encoding)
{
_encoding = encoding;
_client = new HttpClient() { Timeout = TimeSpan.FromSeconds(10)};
}
private HttpRequestMessage _request;
private HttpResponseMessage _response;
private string _responseString;
public string GetPageData(string link)
{
_request = new HttpRequestMessage(HttpMethod.Get, link);
_request.Headers.Add("User-Agent", "Chrome/21.0.1180.89");
_request.Headers.Add("Accept", "text/html");
GetResponse().Wait();
GetStringFromResponse().Wait();
return _responseString;
}
private async Task<HttpResponseMessage> GetResponse() {
return _response = await _client.GetAsync(_request.RequestUri);
}
private async Task<string> GetStringFromResponse() {
return _responseString = await _response.Content.ReadAsStringAsync();
}
}
}
I start downloading page by calling
new PageDownloader().GetPageData(url);
When I'm trying to debug the code, everything is fine till GetResponse().Wait(). But somehow GetResponse() task never finishes - breakpoint on the next line is never reached. I get no exceptions, application continues running. Any suggestions?

This is a standard deadlock condition you get when you start an async operation and then block on the returned task.
Here is a blog post discussion the topic.
Basically, the await call ensures that the continuation it wires up of the task will run in the context you were originally in (which is very helpful) but because you are calling Wait in that same context it's blocking, so the continuation never runs, and that continuation needs to run for the wait to end. Classic deadlock.
As for the fix; usually it means you just shouldn't be doing a blocking wait on the async operation; it's contrary to the design of the whole system. You should, "async all the way up". In this case it would mean that GetPageData should return a Task<string> rather than a string, and rather than waiting on the other operations that return a task you should await on them.
Now, having said that, there are ways of doing a blocking wait on the async operations without deadlocking. While it can be done, it honestly defeats the purpose of using async/await in the first place. The primary advantage of using that system is that the main context isn't blocked; when you block on it that entire advantage goes away, and you might as well just use blocking code all the way through. async/await is really more of an all-or-nothing paradigm.
Here is how I would structure that class:
public class PageDownloader
{
private System.Net.Http.HttpClient _client;
private Encoding _encoding;
public PageDownloader()
: this(Encoding.UTF8) { }
public PageDownloader(Encoding encoding)
{
_encoding = encoding;
_client = new HttpClient() { Timeout = TimeSpan.FromSeconds(10) };
}
public async Task<string> GetPageData(string link)
{
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, link);
request.Headers.Add("User-Agent", "Chrome/21.0.1180.89");
request.Headers.Add("Accept", "text/html");
HttpResponseMessage response = await _client.GetAsync(request.RequestUri);
return await response.Content.ReadAsStringAsync(); ;
}
}

Why not just do this if you want to have a function like that.
public string GetPageData(string link)
{
_request = new HttpRequestMessage(HttpMethod.Get, link);
_request.Headers.Add("User-Agent", "Chrome/21.0.1180.89");
_request.Headers.Add("Accept", "text/html");
var readTask = _client.GetStringAsync(link);
readTask.Wait();
return readTask.Result;
}
It would be better to return the Task all the way back and handle it with async/await in the calling code.
public Task<string> GetPageData(string link)
{
_request = new HttpRequestMessage(HttpMethod.Get, link);
_request.Headers.Add("User-Agent", "Chrome/21.0.1180.89");
_request.Headers.Add("Accept", "text/html");
return _client.GetStringAsync(link);
}