Develop Reference

HttpClient is sending additional Cookies

With .NET Core 3.1 DI, I use an HttpClient instance inside an Azure Function to send data to an external API.
var clientRequest = new HttpRequestMessage(HttpMethod.Post, clientConfig.BaseURL + RequestParam);
string payloadBody = GenerateLoginPayload(clientConfig.Username, clientConfig.Password);
clientRequest.Content = new StringContent(payloadBody, Encoding.UTF8, "application/json");
clientRequest.Headers.Clear();
var response = await _api.SendAsync(clientRequest);
As you can see, I even cleared the clientRequest Headers. However, checking on Fiddler, it is still sending extra Cookie Headers. This is messing up our requests, as to the API, no past Cookies must be sent.
Cookie: KEY_PHPSESSID=HehRVwcsmiUmoxO4zYaOpO1B9hm%2BfAK9igvuRw4YjgA%3D%3AJ7Bgip94EQjfJ%2FSt5RcnFTY3jcZ%2FlChCoyK54v%2Fu%2FJI%3D
Accept: application/json
Content-Type: application/json; charset=utf-8
Content-Length: 87
My question is, what is adding that extra Cookie header value, and how can I clear or remove it?
I also checked this SO question about extra cookies being sent which is similar to my situation, but t does not seem to apply to Azure Functions and .NET Core, as I do not have any reference to Windows.Web.Http.HttpClient
This is also how I registered my HttpClient, and the _api.SendAsync() is reused multiple times across multiple endpoints, and it's possible that the Cookie originated from a previous call. How do I ensure the Cookies don't bleed to other Requests?
builder.Services.AddHttpClient<IApiCaller, ApiCaller>()
.ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler() { UseCookies = false })
.AddPolicyHandler(HttpRetryPolicies.RetryWithJitter());

Maybe it's a DefaultRequestHeader of HttpClient itself.
You could try _api.DefaultRequestHeaders.Clear();

Put a breakpoint before the request and check the contents of
clientRequest.DefaultRequestHeaders.GetCookies()
If the cookie is in there, it might get set in the implementation of your ApiCaller. If there are no cookies there, it might be set by a setting/proxy in Azure

403 Forbidden with WebClient

The company I work for have a limited internet connection and we use the proxy (example: 10.10.10.10:8080) to access at some restricted connections.
I can use the API in Postman (putting the proxy in the Postman settings) but when putting in the C# WebClient code it gives me a 403-Forbidden error.
I only need the var sensorData field but I splitted in var data and var data2 to understand where was the problem. It gives me the error at the var data = ...
Uri uri = new Uri("https://XXXXXXXX/api/DatasourceData/DatasourceDataHistoryBySerialNumber/");
Token token = new Token();
token = GetToken(tokenAPI);
using (WebClient client = new WebClient())
{
try
{
client.Proxy = new WebProxy("10.10.10.10", 8080);
client.Headers.Add("Authorization", "Bearer " + token.AccessToken);
client.QueryString.Add("serialNumbersDatasource", "I2001258");
client.QueryString.Add("startDate", string.Format("{0:s}", "2019-12-01"));
client.QueryString.Add("endDate", string.Format("{0:s}", DateTime.Now));
client.QueryString.Add("isFilterDatesByDataDate", "false");
var data = client.DownloadData(uri);
var data2 = (Encoding.UTF8.GetString(data));
sensorData = (JsonConvert.DeserializeObject<List<Sensor>>(Encoding.UTF8.GetString(client.DownloadData(uri))))[0];
}
}

Seems the problem at this line
client.Headers.Add("Authorization", "Bearer " + "tokenTest");
here you wil add header Authorization with value Bearer tokenTest
so, 403 Forbidden returns by service which you are addressing, but not a proxy
change to
client.Headers.Add("Authorization", "Bearer " + tokenTest);
and check if tokenTest has valid value

Check to see if you need any additional properties on the proxy. You may possibly need to enable:
UseDefaultCredentials (Boolean) true if the default credentials are
used; otherwise, false. The default value is false
Also, check your full url and query string that you are producing - look at the outgoing request fabricated (in the debugger) or through Fiddler and make sure it all lines up, url, query string, headers, etc.
From the docs:
Address
Gets or sets the address of the proxy server.
BypassArrayList
Gets a list of addresses that do not use the proxy server.
BypassList
Gets or sets an array of addresses that do not use the proxy server.
BypassProxyOnLocal
Gets or sets a value that indicates whether to bypass the proxy server for local addresses.
Credentials
Gets or sets the credentials to submit to the proxy server for authentication.
UseDefaultCredentials
Gets or sets a Boolean value that controls whether the DefaultCredentials are sent with requests.

Probably a problem with authorization header.
Is the token valid? Does it work with the same token in Postman?
I bet the api can't validate the token and and gives you no authorization to the resources. This is what a 403 would mean (but don't know what the api programmer actually intended by giving you 403).
Do you have access to the api's source code?
The token is really a string "tokentest" and that works with Postman?

I would suggest you to go for xNet.dll instead of webclient Because xNet library are considered best for proxy and webrequest.
var request = new HttpRequest()
request.UserAgent = Http.ChromeUserAgent();
request.Proxy = Socks5ProxyClient.Parse("10.10.10.10:8080");//can use socks4/5 http

Based on this
Try adding User-Agent in the header
client.Headers.Add("User-Agent", "PostmanRuntime/7.26.1");

In my case i did not specify security protocol. Paste this line of code before running any WebClient requests.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
original answer: How to specify SSL protocol to use for WebClient class

Cookie for domain not being used for subdomains

I use HttpClient in my app to send my user/password to a service that returns some cookies that I can later use for all my other requests. The service is located at https://accounts.dev.example.com/login and returns two cookies that have Domain=.dev.example.com. The issue I'm finding is that, in some machines (Windows Domain Controllers), these cookies are not being used when I request resources in subdomains like https://accounts.dev.example.com/health-check, but according to the MDN docs a cookie for a domain can be used for requesting resources to subdomains:
Domain= Optional
Specifies those hosts to which the cookie will be sent. If not specified, defaults to the host portion of the current document location (but not including subdomains). Contrary to earlier specifications, leading dots in domain names are ignored. If a domain is specified, subdomains are always included.
Do you know how to properly configure HttpClient to pass the domain cookies to subdomain requests?
A bit more of details:
The cookies returned by my authentication service at https://accounts.dev.example.com/login look like this in the HTTP headers:
Set-Cookie: AK=112233;Version=1;Domain=.dev.example.com;Path=/;Max-Age=5400;Secure;HttpOnly,
Set-Cookie: AS=445566;Version=1;Domain=.dev.example.com;Path=/;Max-Age=5400;Secure;HttpOnly,
Then I can query C#'s CookieContainer with either of these calls in normal workstations:
cookies.GetCookies("https://accounts.dev.example.com")
cookies.GetCookies("https://dev.example.com")
Both of which will return the 2 cookies like:
$Version=1; AK=112233; $Path=/; $Domain=.dev.example.com
$Version=1; AS=445566; $Path=/; $Domain=.dev.example.com
But in the other machines (the Domain Controller's) the first call will return an empty list, while the second will return the 2 cookies.
Why this difference on the behaviour of CookieContainer.GetCookies depending on which machine is running the code?
My workstations are using Microsoft Windows 10 Home Single Language (.Net 4.0.30319.42000) and the DCs are using Microsoft Windows Server 2012 R2 Datacenter (.Net 4.0.30319.36399).
The code
This is a modified version of my code:
public static async Task<string> DoAuth(CookieContainer cookies,
Dictionary<string, string> postHeaders,
StringContent postBody)
{
try
{
using (var handler = new HttpClientHandler())
{
handler.CookieContainer = cookies;
using (var client = new HttpClient(handler, true))
{
foreach (var key in postHeaders.Keys)
client.DefaultRequestHeaders.Add(key, postHeaders[key]);
var response = await client.PostAsync("https://accounts.dev.example.com/login", postBody);
response.EnsureSuccessStatusCode();
// This line returns 0 in Domain Controllers, and 2 in all other machines
Console.Write(cookies.GetCookies("https://accounts.dev.example.com").Count);
return await response.Content.ReadAsStringAsync();
}
}
}
catch (HttpRequestException e)
{
...
throw;
}
}

As I couldn't find an answer to this (not in TechNet either), I decided to go with the following solution, which works, but not sure if there is a proper way of solving the issue:
foreach (Cookie cookie in cookies.GetCookies(new Uri("https://dev.example.com")))
{
cookies.Add(new Uri("https://accounts.dev.example.com"), new Cookie(cookie.Name, cookie.Value, cookie.Path, ".accounts.dev.example.com"));
}
So, I'm duplicating the cookie for each one of the subdomains that my app should send these cookies to.

The underlying issue seems to be a bug in the Set-Cookie header. It seems the cause of the issue is the Version= component in the Set-Cookie header. This makes the CookieContainer fall on its face and results in the strange $Version and $Domain cookies then being sent in subsequent client requests. As far as I can tell there is no way to remove these broken cookies either. Iterating GetCookies() with the originating domain does not reveal the erroneous cookies.

Identity Framework Authenticated Request for Spidering

I am working on a project to build a web spider that crawls our internal web pages. The security is built using the Identity Framework. The non-protected pages is easy by using this code:
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("http://www.yoursite.com/resource/file.htm");
using (StreamWriter streamWriter = new StreamWriter(webRequest.GetRequestStream(), Encoding.UTF8))
{
streamWriter.Write(requestData);
}
string responseData = string.Empty;
HttpWebResponse httpResponse = (HttpWebResponse)webRequest.GetResponse();
using (StreamReader responseReader = new StreamReader(httpResponse.GetResponseStream()))
{
responseData = responseReader.ReadToEnd();
}
But this code isn't using the OWIN authentication so when it hits a protected page it can't access it.
If I am logged in and try HttpContext.Request then I see through the debugger the Request is authenticated. How do I use this request object to get a new protected page to parse? I am missing a very simple method I am sure.
Update
I am still struggling with this. Maybe I should ask this another way. I am calling a page from the same web application. From the calling controller this returns true which is good:
HttpContext.Current.User.Identity.IsAuthenticated
But through the debugger and check this in the receiving controller and now this is false, which is bad:
HttpContext.Current.User.Identity.IsAuthenticated
How can get the receiving controller to be authenticated?

It really depends on what authentication method you implementation is being used on server side. Asp.net Identity is essentially a membership system.
For example if it's using bearer authentication tokens then you'd need to set a bearer authentication token in the Authorization header of the request you are sending. And I'd rather suggest using HttpClient for these purposes.
As you mentioned OWIN, Im assuming this is an bearer token scenario.
var httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", "Your Oauth token");
Update
If you're using the default project template of Asp.net MVC with Asp.net Identity as a added dependency later, you can hook up a httpClient like the following one here:
class Program
{
static void Main()
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri("http://yourapp");
var content = new FormUrlEncodedContent(new[]
{
// Push your stuff here, your username, password fields
// as you coded in your server
new KeyValuePair<string, string>("", "login")
});
var result = client.PostAsync("login", content).Result;
string resultContent = result.Content.ReadAsStringAsync().Result;
Console.WriteLine(resultContent);
}
}
}
And you'd of course need to save the auth cookie and you can see how here and to attach it when you do the request to get the secured page, you have to do something like this

Looks like the permissions are based on roles. Check to see if the user you are logged in as has the permission to access the controller method getting hit when the user is redirected to that page.
If you have Attribute parameter for the method that specifies the role for the user required in order to hit then the logged in user should have those permissions to access the page.
As you are getting authenticated may be the issue is with authorization. The user might not have enough permissions to access the page.

HttpClient is adding it's own cookie header

I have rather strange problem. With all that questions over internet how to add and get cookies, I want the opposite ;)
When I try to send request via HttpHandler it is adding it's own Cookie header. I have to get rid of it. Without digging into details - when it is added, server I am trying to request is giving me wrong answer. It works without this cookie (tried in fiddler).
But back to the problem, code:
string domain = "someMysteriousDomain";
var handler = new HttpClientHandler();
handler.UseDefaultCredentials = false;
handler.AllowAutoRedirect = true;
handler.ClientCertificateOptions = ClientCertificateOption.Manual;
handler.UseCookies = false;
var httpClient = new HttpClient(handler);
var request = new HttpRequestMessage(HttpMethod.Get, domain);
request.Headers.UserAgent.Add(new ProductInfoHeaderValue("Mozilla", "5.0"));
request.Headers.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
var response = await httpClient.SendAsync(request);
Raw request seen in fiddler:
GET https://domain HTTP/1.1
Accept-Encoding: gzip
User-Agent: Mozilla/5.0
Host: domain
Connection: Keep-Alive
Cookie: cadata477E7C1824F44800AF0077724F65345="51595d316-0286-44bb-bc6f-ffb1fd311a92SqJA36rA69YW7aBg+iHXYi9LAcBLN6DBWE8a3MLejd2VCluO/UQ5eF6F6T4NWh4NhdRcv4rea15Hs0e2q6GatMac59UVbljhREYdH6PRbzZC/2qn8QHtpc6go5B56R"; mobile=0
I don't want to add that cookie! How to delete/clear/whatever it?
I am using Visual Studio Community 2015, with Windows Universal Project.
What is interesting, after rebooting my pc after few hours, I was able to make 2 or 3 requests without this cookie (using THE SAME code) and then mysterios cookie returned.
What it is about? How to get rid of it?

Thank you for reporting this issue - this is a known issue with the System.Net.Http.HttpClientHandler API implementation on Windows 10 and we are working on fixing it in an upcoming release.
In the meanwhile, a possible workaround is to use the Windows.Web.Http.HttpClient API with the underlying HttpBaseProtocolFilter class. This class has a property called CookieManager that stores all the cookies for each URI. You can write a method to delete the cookies from the CookieManager for the destination URI before sending a request. This will ensure that no cookies get sent. You can see this sample for how to delete cookies from the CookieManager: https://github.com/Microsoft/Windows-universal-samples/tree/master/httpclient
Thanks
Sidharth [MSFT]