Develop Reference

HttpClient getAsync just works once

I'm trying to call a endpoint to get some info in my website but when I run the getAsync it's only works the first time in the current process.
static async Task Main(string[] args)
{
await GetUserInfo("srmilton"); //Working
await GetUserInfo("srmilton"); //Not Working
while (true)
{
var res = await GetUserInfo("srmilton");
}
}
public static async Task<(string,string)> GetUserInfo(string username)
{
string url = "https://www.mywebsite.com/api/user/detail";
var baseAddress = new Uri(url);
using (var handler2 = new HttpClientHandler { UseCookies = false })
using (var client2 = new HttpClient(handler2) { BaseAddress = baseAddress })
{
client2.DefaultRequestHeaders.Clear();
client2.DefaultRequestHeaders.Add("Cookie", "session=zgbEqIjfSC7M7QdTTpHDkpWLt");
client2.DefaultRequestHeaders.Add("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36");
client2.DefaultRequestHeaders.Add("x-requested-with", "XMLHttpRequest");
client2.DefaultRequestHeaders.Add("referer", "https://www.mywebsite.com/");
var result = await client2.GetAsync("");
string responseString = await result.Content.ReadAsStringAsync();
dynamic jsonresponse = JObject.Parse(responseString);
id = jsonresponse.userInfo.user.id;
sec_id = jsonresponse.userInfo.user.secUid;
return (id, sec_id);
}
}
The first time the fuction GetUserInfo is called it's return the correct json response from the api but the second loop gets stuck in GetAsync. I have already tried .ConfigureAwait(false), .Result and even creating HttpClient just once and reusing but it's aways hang on the second loop.
I don't know what i'm doing wrong, if someone can explain and show the right way to make this works i'll be thankful.

Solved. I was missing a Accept header on my getAsync headers.
client2.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9");

Is there a more elegant/clean way to set HttpRequestMessage Headers using TryAddWithoutValidation method?

I'm using HttpRequestMessage from HttpClient in a couple of methods and currently, I'm repeating the following piece of code all over my code:
This code was converted by https://curl.olsh.me/ so I'm not sure if best practices were used here.
// using System.Net.Http;
using (var httpClient = new HttpClient(handler))
{
using (var request = new HttpRequestMessage(new HttpMethod("POST"), "https://www.url.com/"))
{
request.Headers.TryAddWithoutValidation("authority", "www.url.com");
request.Headers.TryAddWithoutValidation("pragma", "no-cache");
request.Headers.TryAddWithoutValidation("cache-control", "no-cache");
request.Headers.TryAddWithoutValidation("dnt", "1");
request.Headers.TryAddWithoutValidation("x-requested-with", "XMLHttpRequest");
request.Headers.TryAddWithoutValidation("x-odesk-csrf-token", "19b91748869456a4ae700ffb69077745");
request.Headers.TryAddWithoutValidation("user-agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36");
request.Headers.TryAddWithoutValidation("accept", "*/*");
request.Headers.TryAddWithoutValidation("origin", "https://www.url.com");
request.Headers.TryAddWithoutValidation("sec-fetch-site", "same-origin");
request.Headers.TryAddWithoutValidation("sec-fetch-mode", "cors");
request.Headers.TryAddWithoutValidation("sec-fetch-dest", "empty");
request.Headers.TryAddWithoutValidation("referer", "https://www.url.com/");
request.Headers.TryAddWithoutValidation("accept-language", "pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7,fr-FR;q=0.6,fr;q=0.5");
request.Headers.TryAddWithoutValidation("cookie", "G_AUTHUSER_H=1; AccountSecurity_cat=fc4d14f1.oauth2v2_812293");
request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json");
var response = await httpClient.SendAsync(request);
IEnumerable<string> cookies = new List<string>();
response.Headers.TryGetValues("Set-Cookie", out cookies);

I assume you are trying to understand how to extract the code in question into a helper method given its use of using blocks. If not, please clarify and I will adjust my answer. Here is what I would do in this case:
Task<HttpResponseMessage> PostAsyncWithHeaders(Uri uri)
{
var response = new Task<HttpResponseMessage>();
using (var httpClient = new HttpClient(handler))
using (var request = new HttpRequestMessage(HttpMethod.Post, uri))
{
//Add your headers as before
response = await httpClient.SendAsync(request);
}
return response;
}
You can then call the method from anywhere it is accessible like this:
response = await PostAsyncWithHeaders(new Uri("https://www.url.com/"));
IEnumerable<string> cookies = new List<string>();
response.Headers.TryGetValues("Set-Cookie", out cookies);
//Presumably consume cookies (yum!)

I made this test in .NET Core 3 for reference.
If you want to be able to reuse your method throughout the code-base, you can try something like the following:
static async Task Main(string[] args)
{
var client = new HttpClient();
// .NET core setting the content type.
client.DefaultRequestHeaders.TryAddWithoutValidation("Content-Type", "application/json; charset=utf-8");
// this is wrapped in the using statement.
using var requestMessage = GetRequestMessage("https://www.url.com/", HttpMethod.Post);
await client.SendAsync(requestMessage);
}
// Get the request message for reuse. You can then reuse this for maybe different end-points and method types.
// the "referer" or "origin" values can be passed in as parameters too.
static HttpRequestMessage GetRequestMessage(string url, HttpMethod method)
{
var request = new HttpRequestMessage(method, url);
request.Headers.TryAddWithoutValidation("authority", "www.url.com");
request.Headers.TryAddWithoutValidation("pragma", "no-cache");
request.Headers.TryAddWithoutValidation("cache-control", "no-cache");
request.Headers.TryAddWithoutValidation("dnt", "1");
request.Headers.TryAddWithoutValidation("x-requested-with", "XMLHttpRequest");
request.Headers.TryAddWithoutValidation("x-odesk-csrf-token", "19b91748869456a4ae700ffb69077745");
request.Headers.TryAddWithoutValidation("user-agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36");
request.Headers.TryAddWithoutValidation("accept", "*/*");
request.Headers.TryAddWithoutValidation("origin", "https://www.url.com");
request.Headers.TryAddWithoutValidation("sec-fetch-site", "same-origin");
request.Headers.TryAddWithoutValidation("sec-fetch-mode", "cors");
request.Headers.TryAddWithoutValidation("sec-fetch-dest", "empty");
request.Headers.TryAddWithoutValidation("referer", "https://www.url.com/");
request.Headers.TryAddWithoutValidation("accept-language", "pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7,fr-FR;q=0.6,fr;q=0.5");
request.Headers.TryAddWithoutValidation("cookie", "G_AUTHUSER_H=1; AccountSecurity_cat=fc4d14f1.oauth2v2_812293");
return request;
}
If you are using this in a console program, you can be fine with creating HttpClient in a using statement. However, if you are intending to use the client in something like a web app in .net core, you want to use the services.AddHttpClient method and dependency injection. This is because the HttpClient isn't intended to be disposed of after every use, if you are using it multiple times in a given call especially.

really appreciate the answers from #Jaquez and #Kuroiyatsu, from both I get to the following
public async Task<string> postAsync(param1, param2)
{
using (var httpClient = new HttpClient(handler))
{
using (var request = new HttpRequestMessage(new HttpMethod("POST"), "https://www.url.com/"))
{
....
var response = await PostAsync(param1, param2);
var variable = JsonSerializer.Deserialize<Obj>(response);
WebScrapFunc(response);
...
Although it seems weird to return a Task<string> from PostAsync it fits fine my intents.

How do I post a http request in c# using FormUrlEncodedContent

I would like to post a http request in order to login to a website using HttpClient and FormUrlEncodedContent. I can track how the response is supposed to look on my chrome browser, but when i recreate the post request made by my browser, I don't get the anticipated response. The website I'm trying to log into is https://www.lectio.dk/lectio/31/login.aspx.
I guess I'm in doubt as to what I should supply the FormUrlEncodedContent with (using the network tab in chrome I can see request headers and form data, but how do I select what I should supply?). Currently my code looks like this.
CookieContainer container = new CookieContainer();
HttpClientHandler handler = new HttpClientHandler();
handler.CookieContainer = container;
HttpClient client = new HttpClient(handler);
Console.WriteLine(client.DefaultRequestHeaders + "\n" + "--------------------------");
Dictionary<string, string> vals = new Dictionary<string, string>
{
{"user-agent","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"},
{"accept-language","en-GB,en-AS;q=0.9,en-DK;q=0.8,en;q=0.7,da-DK;q=0.6,da;q=0.5,en-US;q=0.4"},
{"accept-encoding","gzip, deflate, br"},
{"accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"},
{"m$Content$username2","username"},
{"m$Content$passwordHidden","password"},
{"__EVENTTARGET","m$Content$submitbtn2"},
};
FormUrlEncodedContent content = new FormUrlEncodedContent(vals);
var response = client.PostAsync("https://www.lectio.dk/lectio/31/login.aspx", content);
var responseString = response.Result;
Console.WriteLine(responseString);
handler.Dispose();
client.Dispose();
The idea is to be logged into the website(I guess my Cookiecontainer will take care of that?) so that I can scrape som data.

HttpClient WebRequestHandler not capturing all cookies

I'm making a POST request to a web site to send username/login information in order to get a cooking containing an authentication token. I build, test, and debug my process running in a console application, then I set it to run as part of a Windows Service.
When I am running in the console app, there are 2 cookies being returned from the post request: the JSESSIONID and the AuthToken. After I deploy and run it in the Windows Service, I only see the JSESSIONID.
I set up Fiddler to watch the windows service, and I can see the AuthToken being passed back in the response to my post request, but I am unable to get it from the cookie container.
Additionally, if I install the process on my local machine, and run through my Fiddler proxy, I am able to get the AuthToken, but if I disable the proxy, no AuthToken is contained in the cookies.GetCookies result.
I'm at a loss as to why it will operate fine when running as a console app on my local machine, but the AuthToken fails to return when run as a service on a remote machine.
My local machine is running .net 4.7, and the server where the service is installed is running 4.5.2.
Here's the code I'm using:
public string SubmitPost(Uri uri, string action, string contentPost, bool putRequest)
{
CookieContainer cookies = new CookieContainer();
WebRequestHandler handler = new WebRequestHandler();
handler.Proxy = new WebProxy("http://<FIDDLER PROXY>:8888", false, new string[] {});
X509Certificate cert = X509Certificate2.CreateFromSignedFile(m_CertPath);
handler.ClientCertificates.Add(cert);
handler.CookieContainer = cookies;
string resultContent = "";
using (var client = new HttpClient(handler))
{
AddClientHeadersForPost(uri, client);
cookies.Add(uri, m_jar);
var content = new StringContent(contentPost, Encoding.UTF8);
content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
content.Headers.ContentLength = contentPost.Length;
HttpResponseMessage result = client.PostAsync(action, content).Result;
resultContent = result.Content.ReadAsStringAsync().Result;
IEnumerable<Cookie> responseCookies = cookies.GetCookies(uri).Cast<Cookie>();
Logger.InfoFormat("{0} cookies", responseCookies.Count());
foreach (Cookie item in responseCookies)
{
Logger.InfoFormat("Cookie: {0}", item.Name);
if (item.Name.Contains("auth"))
{
Logger.InfoFormat("Auth Token: {0}", item.Value);
}
m_jar.Add(item);
}
}
return (resultContent);
}
protected virtual void AddClientHeadersForPost(Uri uri, HttpClient client)
{
client.BaseAddress = uri;
client.DefaultRequestHeaders.TryAddWithoutValidation("Host", "<HOST URL>");
client.DefaultRequestHeaders.TryAddWithoutValidation("Origin", "<ORIGIN URL>");
client.DefaultRequestHeaders.TryAddWithoutValidation("X-Requested-With", "XMLHttpRequest");
client.DefaultRequestHeaders.TryAddWithoutValidation("Accept", "*/*");
client.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate, br");
client.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Language", "en-US,en;q=0.8");
client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
client.DefaultRequestHeaders.TryAddWithoutValidation("DNT", "1");
client.DefaultRequestHeaders.TryAddWithoutValidation("Referer", "<REFERER URL>");
client.DefaultRequestHeaders.TryAddWithoutValidation("X-CSRF-TOKEN", m_csrf);
client.DefaultRequestHeaders.TryAddWithoutValidation("X-2-CSRF-TOKEN", m_sppCSRF);
client.DefaultRequestHeaders.TryAddWithoutValidation("Connection", "keep-alive");
client.DefaultRequestHeaders.ExpectContinue = false;
}

Why can't I use HttpClient to log in to this ASP.NET website?

There's an ASP.NET website from a third party that requires one to log on. I need to get some data from the website and parse it, so I figured I'd use HttpClient to post the necessary credentials to the website, same as the browser would do it. Then, after that POST request, I figured I'd be able to use the cookie values I received to make further request to the (authorization-only) urls.
I'm down to the point where I can successfully POST the credentials to the login url and receive three cookies: ASP.NET_SessionId, .ASPXAUTH, and a custom value used by the website itself, each with their own values. I figured that since the HttpClient I set up is using an HttpHandler that is using a CookieContainer, the cookies would be sent along with each further request, and I'd remain logged in.
However, this does not appear to be working. If I use the same HttpClient instance to then request one of the secured areas of the website, I'm just getting the login form again.
The code:
const string loginUri = "https://some.website/login";
var cookieContainer = new CookieContainer();
var clientHandler = new HttpClientHandler() { CookieContainer = cookieContainer, AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate };
var client = new HttpClient(clientHandler);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
var loginRequest = new HttpRequestMessage(HttpMethod.Post, loginUri);
// These form values correspond with the values posted by the browser
var formContent = new FormUrlEncodedContent(new[]
{
new KeyValuePair<string, string>("customercode", "password"),
new KeyValuePair<string, string>("customerid", "username"),
new KeyValuePair<string, string>("HandleForm", "Login")
});
loginRequest.Content = formContent;
loginRequest.Headers.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393");
loginRequest.Headers.Referrer = new Uri("https://some.website/Login?ReturnUrl=%2f");
loginRequest.Headers.Host = "some.website";
loginRequest.Headers.Connection.Add("Keep-Alive");
loginRequest.Headers.CacheControl = new System.Net.Http.Headers.CacheControlHeaderValue() { NoCache = true };
loginRequest.Headers.AcceptLanguage.ParseAdd("nl-NL");
loginRequest.Headers.AcceptEncoding.ParseAdd("gzip, deflate");
loginRequest.Headers.Accept.ParseAdd("text/html, application/xhtml+xml, image/jxr, */*");
var response = await client.SendAsync(loginRequest);
var responseString = await response.Content.ReadAsStringAsync();
var cookies = cookieContainer.GetCookies(new Uri(loginUri));
When using the proper credentials, cookies contains three items, including a .ASPXAUTH cookie and a session id, which suggests that the login succeeded. However:
var text = await client.GetStringAsync("https://some.website/secureaction");
...this just returns the login form again, and not the content I get when I log in using the browser and navigate to /secureaction.
What am I missing?
EDIT: here's the complete request my application is making and the request chrome is making. They are identical, save for the cookie values. I ran them through windiff: the lines marked <! are the lines sent by my application, the ones marked !> are sent by Chrome.
GET https://some.website/secureaction
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36
Accept-Encoding: gzip, deflate, sdch, br
Upgrade-Insecure-Requests: 1
Host: some.website
Accept-Language:nl-NL,
>> nl;q=0.8,en-US;q=0.6,en;q=0.4
Accept: text/html,
>> application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Cookie:
<! customCookie=7CF190C0;
<! .ASPXAUTH=37D61E47(shortened for readability);
<! ASP.NET_SessionId=oqwmfwahpvf0qzpiextx0wtb
!> ASP.NET_SessionId=kn4t4rmeu2lfrgozjjga0z2j;
!> customCookie=8D43E263;
!> .ASPXAUTH=C2477BA1(shortened for readability)
The HttpClient application get a 302 referral to /login, Chrome gets a 200 response containing the requested page.

As requested, here's how I eventually made it work. I had to do a simple GET request to /login first, and then do a POST with the login credentials. I don't recall what value exactly is being set by that GET (I assume a cookie with some encoded value the server wants), but the HttpClient takes care of the cookies anyway, so it just works. Here's the final, working code:
const string loginUri = "https://some.website/login";
var cookieContainer = new CookieContainer();
var clientHandler = new HttpClientHandler()
{
CookieContainer = cookieContainer,
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
var client = new HttpClient(clientHandler);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
// First do a GET to the login page, allowing the server to set certain
// required cookie values.
var initialGetRequest = new HttpRequestMessage(HttpMethod.GET, loginUri);
await client.SendAsync(initialGetRequest);
var loginRequest = new HttpRequestMessage(HttpMethod.Post, loginUri);
// These form values correspond with the values posted by the browser
var formContent = new FormUrlEncodedContent(new[]
{
new KeyValuePair<string, string>("customercode", "password"),
new KeyValuePair<string, string>("customerid", "username"),
new KeyValuePair<string, string>("HandleForm", "Login")
});
loginRequest.Content = formContent;
loginRequest.Headers.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393");
loginRequest.Headers.Referrer = new Uri("https://some.website/Login?ReturnUrl=%2f");
loginRequest.Headers.Host = "some.website";
loginRequest.Headers.Connection.Add("Keep-Alive");
loginRequest.Headers.CacheControl = new System.Net.Http.Headers.CacheControlHeaderValue() { NoCache = true };
loginRequest.Headers.AcceptLanguage.ParseAdd("nl-NL");
loginRequest.Headers.AcceptEncoding.ParseAdd("gzip, deflate");
loginRequest.Headers.Accept.ParseAdd("text/html, application/xhtml+xml, image/jxr, */*");
var response = await client.SendAsync(loginRequest);
var responseString = await response.Content.ReadAsStringAsync();
var cookies = cookieContainer.GetCookies(new Uri(loginUri));