I have rather strange problem. With all that questions over internet how to add and get cookies, I want the opposite ;)
When I try to send request via HttpHandler it is adding it's own Cookie header. I have to get rid of it. Without digging into details - when it is added, server I am trying to request is giving me wrong answer. It works without this cookie (tried in fiddler).
But back to the problem, code:
string domain = "someMysteriousDomain";
var handler = new HttpClientHandler();
handler.UseDefaultCredentials = false;
handler.AllowAutoRedirect = true;
handler.ClientCertificateOptions = ClientCertificateOption.Manual;
handler.UseCookies = false;
var httpClient = new HttpClient(handler);
var request = new HttpRequestMessage(HttpMethod.Get, domain);
request.Headers.UserAgent.Add(new ProductInfoHeaderValue("Mozilla", "5.0"));
request.Headers.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
var response = await httpClient.SendAsync(request);
Raw request seen in fiddler:
GET https://domain HTTP/1.1
Accept-Encoding: gzip
User-Agent: Mozilla/5.0
Host: domain
Connection: Keep-Alive
Cookie: cadata477E7C1824F44800AF0077724F65345="51595d316-0286-44bb-bc6f-ffb1fd311a92SqJA36rA69YW7aBg+iHXYi9LAcBLN6DBWE8a3MLejd2VCluO/UQ5eF6F6T4NWh4NhdRcv4rea15Hs0e2q6GatMac59UVbljhREYdH6PRbzZC/2qn8QHtpc6go5B56R"; mobile=0
I don't want to add that cookie! How to delete/clear/whatever it?
I am using Visual Studio Community 2015, with Windows Universal Project.
What is interesting, after rebooting my pc after few hours, I was able to make 2 or 3 requests without this cookie (using THE SAME code) and then mysterios cookie returned.
What it is about? How to get rid of it?
Thank you for reporting this issue - this is a known issue with the System.Net.Http.HttpClientHandler API implementation on Windows 10 and we are working on fixing it in an upcoming release.
In the meanwhile, a possible workaround is to use the Windows.Web.Http.HttpClient API with the underlying HttpBaseProtocolFilter class. This class has a property called CookieManager that stores all the cookies for each URI. You can write a method to delete the cookies from the CookieManager for the destination URI before sending a request. This will ensure that no cookies get sent. You can see this sample for how to delete cookies from the CookieManager: https://github.com/Microsoft/Windows-universal-samples/tree/master/httpclient
Thanks
Sidharth [MSFT]
Related
With .NET Core 3.1 DI, I use an HttpClient instance inside an Azure Function to send data to an external API.
var clientRequest = new HttpRequestMessage(HttpMethod.Post, clientConfig.BaseURL + RequestParam);
string payloadBody = GenerateLoginPayload(clientConfig.Username, clientConfig.Password);
clientRequest.Content = new StringContent(payloadBody, Encoding.UTF8, "application/json");
clientRequest.Headers.Clear();
var response = await _api.SendAsync(clientRequest);
As you can see, I even cleared the clientRequest Headers. However, checking on Fiddler, it is still sending extra Cookie Headers. This is messing up our requests, as to the API, no past Cookies must be sent.
Cookie: KEY_PHPSESSID=HehRVwcsmiUmoxO4zYaOpO1B9hm%2BfAK9igvuRw4YjgA%3D%3AJ7Bgip94EQjfJ%2FSt5RcnFTY3jcZ%2FlChCoyK54v%2Fu%2FJI%3D
Accept: application/json
Content-Type: application/json; charset=utf-8
Content-Length: 87
My question is, what is adding that extra Cookie header value, and how can I clear or remove it?
I also checked this SO question about extra cookies being sent which is similar to my situation, but t does not seem to apply to Azure Functions and .NET Core, as I do not have any reference to Windows.Web.Http.HttpClient
This is also how I registered my HttpClient, and the _api.SendAsync() is reused multiple times across multiple endpoints, and it's possible that the Cookie originated from a previous call. How do I ensure the Cookies don't bleed to other Requests?
builder.Services.AddHttpClient<IApiCaller, ApiCaller>()
.ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler() { UseCookies = false })
.AddPolicyHandler(HttpRetryPolicies.RetryWithJitter());
Maybe it's a DefaultRequestHeader of HttpClient itself.
You could try _api.DefaultRequestHeaders.Clear();
Put a breakpoint before the request and check the contents of
clientRequest.DefaultRequestHeaders.GetCookies()
If the cookie is in there, it might get set in the implementation of your ApiCaller. If there are no cookies there, it might be set by a setting/proxy in Azure
I am getting to the stage of hair pulling with this one, I'm hoping someone can see if I'm doing anything wrong.
I'm trying to POST some form data to website using Azure data factory web activity however whilst I get a response (I get the page and some headers) it is different to the response I get if I make the exact same request using C# and HttpClient code. I've used fiddler to view the request being post'd using my C# script and according to the request information given in data factory they are exactly the same - so same headers, same content format etc...
This POST request is to login to a website which has a custom login mechanism, so no OAuth or anything like that unfortunately. It is supposed to return a cookie, which it does if I use my C# script, but if I make the same POST request using data factory web activity then I get different html sent back (it just returns the same login screen) and also different set of response headers in the "ADFWebActivityResponseHeaders" part of the activity output!?! See below for what is returned in the web activity output response headers:-
"ADFWebActivityResponseHeaders": {
"Pragma": "no-cache",
"Vary": "Accept-Encoding",
"X-Frame-Options": "DENY",
"Cache-Control": "no-store, must-revalidate, no-cache, post-check=0, pre-check=0",
"Date": "Wed, 09 Sep 2020 08:09:30 GMT",
"Server": "Microsoft-IIS/8.5"
}
If I do this via C# I also get a 'Set-Cookie' as well (strangely if I make a 'GET' request for the homepage of this site I do get a 'Set-Cookie' in the response!!!), but never when doing this via data factory. I'm struggling to see how this is possible unless data factory is modifying my request in some fashion? Below is my C# code, pretty simple/standard:-
var handler = new HttpClientHandler();
handler.CookieContainer = new CookieContainer();
handler.UseCookies = true;
handler.UseDefaultCredentials = false;
// Create our http client which will perform our web requests
var HttpClient = new HttpClient(handler);
HttpClient.BaseAddress = new Uri("**REMOVED**");
// Some of the extracts take a LONG time, so set the timeout for default of 30mins
HttpClient.Timeout = TimeSpan.FromMinutes(30);
// Set the 'form' parameters we're going to POST to the server in the request
var parameters = new Dictionary<string, string>
{
{ "username", "**REMOVED**" },
{ "password", "**REMOVED**" }
};
// URL encode the parameters
var content = new FormUrlEncodedContent(parameters);
// Submit our POST with the parameters
var response = await HttpClient.PostAsync("**REMOVED**", content);
Running this code and using fiddler I see the following request with headers, these are the only headers:-
Content-Length: 80
Content-Type: application/x-www-form-urlencoded
username=REMOVED&password=REMOVED
and in the 'input' side of the web activity is the details of the request, I've added the headers in the web activity and these are correct:-
"method": "POST",
"headers": {
"Content-Type": "application/x-www-form-urlencoded",
"Content-Length": 80
},
"body": "username=REMOVED&password=REMOVED"
Note that in the data factory I'm using a self hosted integration runtime as this website blocks addresses that do not come from the specific IP addresses used externally by our on-prem network/firewall. I know that is not the problem as I'm getting a response with the normal login page from the site (if I use the Azure integration runtime I get a denied response).
Here is a screen shot of the web activity in data factory:-
Really hope someone out there can see what I'm missing or whatever...
Turns out this does work and will list the cookies in the JSON output from the activity as shown below (note this is to be found in the output of the ADF activity, so you would pick up the cookie from the output a bit like... #activity('Login and get cookie').output.ADFWebActivityResponseHeaders["Set-Cookie"] )
However, in my case the url I was POSTing to was responding with a 302 (moved temporarily) but the 'Location' header which should be there is not in the ADFWebActivityResponseHeaders - which is why I missed it. I tried using Chrome with the developer tools and looked at the response directly which is where I found the 302 response code. After that, I just used the new URL given in the response headers (i.e. the url in the 'Location') that I found when using the browser dev tools.
Unfortunately at the time of writing, the Azure data factory HTTP activity does not follow redirects (and doesn't list all the response headers either!) so if anyone encounters the same problem they will need to manually find out and get the url's for any redirects. In other words, try using a tool like browser/postman and look at the response if it doesn't work in ADF... you might find there is a redirect going on :-)
There is a feature request logged for this here, be sure to add your vote :)
edited to update the Azure feedback change of URL after MS decided to change things on the feedback site!?!
I've not used HttpClient before so apologies if it's an obvious one.
I'm poking about with the airbnb api http://airbnbapi.org/#view-listing-info
My understanding of the endpoint is that I don't need an auth token, as this is a public endpoint I'm trying to use. Unfortunately I'm getting a 403 no matter what I try to do and I'm not entirely sure why.
I've got the following code:
client.DefaultRequestHeaders.Add("client_id", "<My client Id>");
client.DefaultRequestHeaders.Add("locale", "en-gb");
client.DefaultRequestHeaders.Add("currency", "gbp");
var request = new HttpRequestMessage()
{
RequestUri = new Uri($"https://api.airbnb.com/v2/listings/{id}"),
Method = HttpMethod.Get,
};
var task = client.SendAsync(request)
.ContinueWith((taskwithmsg) =>
{
var response = taskwithmsg.Result;
//var jsonTask = response.Content.ReadAsAsync<JsonResult>();
//jsonTask.Wait();
//var jsonObject = jsonTask.Result;
return response.Content;
});
task.Wait();
return task.Result;
And I'm getting the following response:
- response {StatusCode: 403, ReasonPhrase: 'Forbidden', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Connection: close
Date: Tue, 04 Apr 2017 19:44:40 GMT
Server: AkamaiGHost
Mime-Version: 1.0
Content-Length: 291
Content-Type: text/html
Expires: Tue, 04 Apr 2017 19:44:40 GMT
}} System.Net.Http.HttpResponseMessage
Any advice?
EDIT:
Macceturra wisely suggested I try to make the call with postman.
I've now established that I can make a call in postman and get a correct response back.
The request you're sending has the client_id as a HTTP header, when Airbnb is expecting it as a URL parameter.
Additionally, Airbnb requires the client to send an Accept (or User-Agent) header, or else it will still return "403 Forbidden" (probably should be "400 Bad Request").
Putting that together (and deleting the unnecessary headers):
var id = ...;
var clientId = ...;
var uri = new Uri($"https://api.airbnb.com/v2/listings/{id}?client_id={Uri.EscapeDataString(clientId)}&locale=en-gb¤cy=GBP");
HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Add("Accept", "application/json");
await client.GetAsync(uri);
I've not been able to mark a single answer as the solution as it came through in the comments.
Solution
Start by making calls in postman. This helped me realise that I should have been passing the client id as a url parameter and not as a header. Bradley Grainger went on to answer this too.
Try and make that exact call with the HttpClient. This still returned the 403, but now we have confidence in the url we're sending.
Change the content type to application/json. This beats the 403 and gets us a 200 with the requested data.
Big thanks to all who helped piece these steps together.
You may need to add a cookie to the request called _aat which contains the airbnb access token. This is generated during login. You'll be able to see it by looking at the headers in Fiddler or the Net section of the browser dev tools when browsing one of your own listings when logged in. Just copy and paste it from there into your code to test if you can get the request working properly and then later you can automate the process of getting the _aat cookie within your code. Make sure the http headers in your code also match what is being sent via the browser.
I am tearing my the remainings of my hair off trying to solve an issue with DotNetOpenOAuth. Below are the details of the problem.
Consumer = new DesktopConsumer(xeroProviderDescription, _tokenManager);
var endPoint = new MessageReceivingEndpoint(apiEndPoint, HttpDeliveryMethods.PostRequest | HttpDeliveryMethods.AuthorizationHeaderRequest);
var request = Consumer.PrepareAuthorizedRequest(endPoint, accessToken);
var content = Encoding.UTF8.GetBytes(payload);
request.ContentType = "application/json";
request.ContentLength = content.Length;
This works perfectly fine, the server accepts the request and replies. Now, I want to switch to using xml to talk to the remote service (this is their prefered format over json) So I changed all the code into sending XML data instead of json and changed the content-type to application/x-www-form-urlencoded as per the web service provider documentation. The problem now is that the service is returning 401 unauthorized request - Reason: invalid signature whenever application/x-www-form-urlencoded is used
Questions:
Is the Content-Type taken into account when generating the oauth signature (I looked into the DotNetOpenOAuth source code with no luck)?
Anyone has ever encountered this issue and resolved it?
i am just working on my first Windows Phone 8.1 app (Universal if this matters, but only Windows Phone implemented at the moment). And at first all is working very smooth but as soon as my app is running for about 25-30 Minutes I can no longer use my HttpClient. I use the Windows.Web.Http.HttpClient.
In my first trys I used a singleHttpClientand reused it all the time. As I became aware that this is not working I started using a newHttpClient` for each request. But still no luck.
This is my method to get a new HttpClient:
private HttpClient GetClient()
{
var filter = new HttpBaseProtocolFilter
{
AllowUI = false,
CacheControl = { WriteBehavior = HttpCacheWriteBehavior.NoCache },
ServerCredential =
new PasswordCredential(
BaseApiUri.ToString(),
credentials.UserName,
credentials.Password),
};
var httpClient = new HttpClient(filter);
var headers = httpClient.DefaultRequestHeaders;
var httpConnectionOptionHeaderValueCollection = headers.Connection;
httpConnectionOptionHeaderValueCollection.Clear();
headers.Accept.TryParseAdd("application/json");
headers.CacheControl.TryParseAdd("no-cache");
headers.Add("Pragma", "no-cache");
headers.Add("Keep-Alive", "false");
headers.Cookie.Clear();
return httpClient;
}
The extra code setting the headers and clearing cookies are my attempts to stop some kind of caching of connections under the surface that might happen. But still no luck.
My method to make requests my API is like the following:
private async Task<bool> PostNoResponseRequestTo(string relativeUri, object requestContent, CancellationToken cancellationToken)
{
var targetUri = new Uri(BaseApiUri, relativeUri);
var requestJson = JsonConvert.SerializeObject(requestContent);
var content = new HttpStringContent(requestJson, UnicodeEncoding.Utf8, "application/json");
try
{
using (var httpClient = this.GetClient())
{
var post =
await httpClient.PostAsync(targetUri, content).AsTask(cancellationToken).ContinueWith(
async request =>
{
using (var response = await request)
{
return response.IsSuccessStatusCode;
}
},
cancellationToken);
return await post;
}
}
catch (Exception)
{
return false;
}
}
This works fine for about 25-30 Minutes after which the calls to the api suddenly start to fail. I start getting a 401 but as you can see i have specified credentials and because those are working and do not change (hardcoded them to test this) i start believing that the problem is on the API side.
This is the response I get:
StatusCode: 401, ReasonPhrase: 'Unauthorized', Version: 2, Content: Windows.Web.Http.HttpStreamContent, Headers:
{
Server: Microsoft-IIS/8.5
Date: Fri, 20 Mar 2015 14:25:06 GMT
WWW-Authenticate: Digest qop="auth",algorithm=MD5-sess,nonce="+Upgraded+NounceRemoved",charset=utf-8,realm="Digest", Negotiate, NTLM
X-Powered-By: ASP.NET
}
{
Content-Length: 1344
Content-Type: text/html
}
My API consists of a Asp.Net project with ServiceStack for its API functionality.
This is running on an IIS with activated digest authentication (all other are disabled).
By inspecting the logs i became aware of a failing API call in front of each successful call. But if i'm right this is by design of digest auth because i have not found a way to tell the client that the other side is using digest auth. I was able to specify this kind of information in my other .Net projects but for some reason Microsoft changed the code (and namespace) for the HttpClient. I am also aware of the HttpClient in the original namespace that you can get through nuget but this is not working for me as i get an error in my output window as soon as i make any call. This closes my app without any kind of information.
Back to the log i was able to get some information with the help of the extended logging and the tool to analyze them. The error is something like (can't access it right now will edit it later):'Invalid token passed to function/method'.
I really hope that someone can help me to solve this problem as it makes the app nearly unusable. My users have to restart the app every 15 Minutes to be on the save site.
Thanks for all advices that help me.
Try Checking the Machine Key setting in IIS. Automatically generate at runtime if tick will generate a new key every time the app pool is restarted. This might be causing your issue. The Machine Key can be set on the server, website or application level. As activated digest authentication is encrypted this might be the issue.
Managing Websites with IIS Manager (part 6) - The Machine Key and Windows Authentication