Develop Reference

Any added headers to DefaultRequestHeaders will not make it through to the API

In my API project I have the following controller that works fine when called with Postman:
[HttpPost]
public async Task<ActionResult> Upload([FromHeader] string authorization, IFormFile payLoad) { ... }
When I use Postman, I add a string (Token) in the Auth section and specify the Type as Bearer Token. I then go to the Body section and set the Key payLoad as a File and choose a file to upload.
Postman generates C# - RestSharp code as follows:
var client = new RestClient("http://localhost:11764/api/logdata");
client.Timeout = -1;
var request = new RestRequest(Method.POST);
request.AddHeader("Authorization", "Bearer exampleTokenString");
request.AddFile("payLoad", "/C:/path/oqwi.zip");
IRestResponse response = client.Execute(request);
Console.WriteLine(response.Content);
I'm not using RestSharp so I haven't verified that the code above works but the Postman post itself within the tool works fine and my API gets all the data as I would expect.
In a separate client application, whenever I attempt to make a POST call, the API controller (at the top of this page) always receives null for the authorization parameter. The file loads in fine. Here is the client code trying to POST to the API with every example I attempted to add the header (I did not try them all at once):
Uri EndPoint = new Uri("http://localhost:11764/api/logdata");
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Authorization
= new AuthenticationHeaderValue("Bearer", AccessToken);
client.DefaultRequestHeaders.Add("Authorization", "Bearer " + AccessToken);
var request = new HttpRequestMessage(HttpMethod.Post, EndPoint)
{
Content = fileAsFormData
};
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", AccessToken);
request.Headers.Add("Authorization", "Bearer " + AccessToken);
var response = await client.SendAsync(request);
...
Note: The above is an example of every different attempt I made at adding the authorization and token; I didn't do all of the above at once.
I have inspected the object in the client during runtime and it appears as if the header(s) are added on where I would expect them to be. Using Fiddler, I can confirm this:
UPDATE: I've tried adding other headers, like CacheControl, and none of them make it through to the API. I see it on the client side during runtime, I see it in Fiddler, but then they're all scrubbed by the time they get to the API. I'm wondering if this Github discussion has anything to do with it:
https://github.com/dotnet/runtime/issues/26475

According to this Github ticket within the dotnet runtime team,
karelz commented on Oct 24, 2018 •
FYI: 2 weeks ago we released a security fix to remove Authorization request headers from redirects.
Thinking that redirects may be at the heart of the issue, by using Fiddler or the Visual Studio inspection tool, I was able to observe that the callback to my client was from https://localhost:5001. I was not expecting this...
When I was constructing the original client code to POST to my API, I was simply copying all of the values I had used when I was exercising these calls from Postman. As an example from Postman's C# RestSharp:
var client = new RestClient("http://localhost:11764/api/logdata");
This ultimately was a red herring caused by the intelligent way Postman handles redirects. Postman was indeed posting to http://localhost:11764 -- but then getting a secure redirect to https://localhost:5001. Postman would then subtly resend the original request with reattached headers to this new secure endpoint.
So after updating the endpoint that the client will POST to, from http://localhost:11764/api/logdata to https://localhost:5001/api/logdata, everything works as expected.
But why https://localhost:5001? It's setup this way in (most) launchSettings.json:
"MyProj.UploadApi.WebApi": {
"commandName": "Project",
"launchBrowser": false,
"environmentVariables": {
"ASPNETCORE_ENVIRONMENT": "Development"
},
"applicationUrl": "https://localhost:5001;http://localhost:11764"
}

POSTing form data using Azure data factory web activity returns different results to C# script

I am getting to the stage of hair pulling with this one, I'm hoping someone can see if I'm doing anything wrong.
I'm trying to POST some form data to website using Azure data factory web activity however whilst I get a response (I get the page and some headers) it is different to the response I get if I make the exact same request using C# and HttpClient code. I've used fiddler to view the request being post'd using my C# script and according to the request information given in data factory they are exactly the same - so same headers, same content format etc...
This POST request is to login to a website which has a custom login mechanism, so no OAuth or anything like that unfortunately. It is supposed to return a cookie, which it does if I use my C# script, but if I make the same POST request using data factory web activity then I get different html sent back (it just returns the same login screen) and also different set of response headers in the "ADFWebActivityResponseHeaders" part of the activity output!?! See below for what is returned in the web activity output response headers:-
"ADFWebActivityResponseHeaders": {
"Pragma": "no-cache",
"Vary": "Accept-Encoding",
"X-Frame-Options": "DENY",
"Cache-Control": "no-store, must-revalidate, no-cache, post-check=0, pre-check=0",
"Date": "Wed, 09 Sep 2020 08:09:30 GMT",
"Server": "Microsoft-IIS/8.5"
}
If I do this via C# I also get a 'Set-Cookie' as well (strangely if I make a 'GET' request for the homepage of this site I do get a 'Set-Cookie' in the response!!!), but never when doing this via data factory. I'm struggling to see how this is possible unless data factory is modifying my request in some fashion? Below is my C# code, pretty simple/standard:-
var handler = new HttpClientHandler();
handler.CookieContainer = new CookieContainer();
handler.UseCookies = true;
handler.UseDefaultCredentials = false;
// Create our http client which will perform our web requests
var HttpClient = new HttpClient(handler);
HttpClient.BaseAddress = new Uri("**REMOVED**");
// Some of the extracts take a LONG time, so set the timeout for default of 30mins
HttpClient.Timeout = TimeSpan.FromMinutes(30);
// Set the 'form' parameters we're going to POST to the server in the request
var parameters = new Dictionary<string, string>
{
{ "username", "**REMOVED**" },
{ "password", "**REMOVED**" }
};
// URL encode the parameters
var content = new FormUrlEncodedContent(parameters);
// Submit our POST with the parameters
var response = await HttpClient.PostAsync("**REMOVED**", content);
Running this code and using fiddler I see the following request with headers, these are the only headers:-
Content-Length: 80
Content-Type: application/x-www-form-urlencoded
username=REMOVED&password=REMOVED
and in the 'input' side of the web activity is the details of the request, I've added the headers in the web activity and these are correct:-
"method": "POST",
"headers": {
"Content-Type": "application/x-www-form-urlencoded",
"Content-Length": 80
},
"body": "username=REMOVED&password=REMOVED"
Note that in the data factory I'm using a self hosted integration runtime as this website blocks addresses that do not come from the specific IP addresses used externally by our on-prem network/firewall. I know that is not the problem as I'm getting a response with the normal login page from the site (if I use the Azure integration runtime I get a denied response).
Here is a screen shot of the web activity in data factory:-
Really hope someone out there can see what I'm missing or whatever...

Turns out this does work and will list the cookies in the JSON output from the activity as shown below (note this is to be found in the output of the ADF activity, so you would pick up the cookie from the output a bit like... #activity('Login and get cookie').output.ADFWebActivityResponseHeaders["Set-Cookie"] )
However, in my case the url I was POSTing to was responding with a 302 (moved temporarily) but the 'Location' header which should be there is not in the ADFWebActivityResponseHeaders - which is why I missed it. I tried using Chrome with the developer tools and looked at the response directly which is where I found the 302 response code. After that, I just used the new URL given in the response headers (i.e. the url in the 'Location') that I found when using the browser dev tools.
Unfortunately at the time of writing, the Azure data factory HTTP activity does not follow redirects (and doesn't list all the response headers either!) so if anyone encounters the same problem they will need to manually find out and get the url's for any redirects. In other words, try using a tool like browser/postman and look at the response if it doesn't work in ADF... you might find there is a redirect going on :-)
There is a feature request logged for this here, be sure to add your vote :)
edited to update the Azure feedback change of URL after MS decided to change things on the feedback site!?!

Sending Json from Postman returns code 200, Sending Json from HttpClient returns 503(Your chunked POST data is too large to upload)

I'm making an api call to UPS using their Address Validation call(https://onlinetools.ups.com/rest/AV). I have had success calling different Gets from UPS before(example: UPS Address Streen Level Validation).
Here is the json that is being sent:
{
"AccessRequest": {
"AccessLicenseNumber": "removed for security reasons",
"UserId": "removed for security reasons",
"Password": "removed for security reasons"
},
"AddressValidationRequest": {
"Request": {
"TransactionReference": {
"CustomerContext": ""
},
"RequestAction": "AV"
},
"Address": {
"City": "",
"StateProvinceCode": "",
"PostalCode": "98272"
}
}
}
In PostMan, there are only the default Headers being used:
This returns successfully with the data I am looking for with code: 200.
In my .Net Standard 2.0 Library, I am using HTTPClient(Microsoft.AspNet.WebApi.Client 5.2.7) to make a call to the api. Here is how I initiate the HTTP Client(variable called "ApiClient"):
string api = _config["UPS_API_LINK"];
_apiClient = new HttpClient();
_apiClient.BaseAddress = new Uri(api);
_apiClient.DefaultRequestHeaders.Accept.Clear();
_apiClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
Now I have created the models to line up so it sends the exact same Json as Postman does. You will see with the code below that I double check that by serializing it myself(I take that string, copy it into postman, and works like a charm). So I send my request. I shortened the code a little bit make my point:
string json = JsonConvert.SerializeObject(request);
Debug.Print(json);
using (HttpResponseMessage response = await ApiClient.PostAsJsonAsync("/rest/AV", request))
{
var content = await response.Content.ReadAsStringAsync();
}
Here is what the content returns:
<HTML><HEAD><TITLE>Weblogic Bridge Message</TITLE></HEAD> <BODY><H2>Failure of Web Server bridge:</H2><P><hr>Your chunked POST data is too large to upload.<hr> </BODY></HTML>
The return code is 503. So can someone lead me into the right direction? I just don't know what HTTP Client is sending that is making it so large that I get this response. Whats surprising to me is that I have another call to the api that sends more data and it works perfectly. Any suggestions would be appreciated.
Note: I have double checked that I am requesting the call using the right URL
Update: Thanks to Nehorai, I discovered that I can use third party dll called RestSharp for making Api calls. This library worked flawlessly! Though I would love to still know why HTTP Client does not work in this situation. Here is my code below using RestSharp incase anyone wanted to see it:
var client = new RestClient("https://onlinetools.ups.com/rest/AV");
client.Timeout = -1;
var requestRest = new RestRequest(Method.POST);
requestRest.AddHeader("Content-Type", "application/json");
requestRest.AddJsonBody(request);
var result = await client.ExecutePostAsync<ZipInfo_RootResponse>(requestRest);

In postman, you can view a code snippet of the request, click on "Code":
Then you can select the language, select C# and you can see the request in C# code (you will need to add a reference to RestSharp dll)

C# HttpClient.PostAsJsonAsync() fails, even though the exact same request is accepted when made through PostMan

I have found similar questions both here as well as on the Elastic discussion forum, but unfortunately none of the answers helped.
I am currently using ElasticSearch 7.0.
I want to make a bulk request to my ElasticSearch server. My JSON file contains information that looks something like this:
{ "index": { "_index": "website", "_id": "link1" }}
{ "label": "Link1" }
Each line is terminated by an LF line break, and there is also an additional LF line break at the end of the document.
In C#, here is how I make a POST request for my bulk data:
HttpResponseMessage response = await httpClient.PostAsJsonAsync($"http://127.0.0.1:9200/website/_bulk", jsonDocumentContents);
And yet I keep seeing this error message:
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\\n]"},"status":400}
How can I fix this error?
UPDATE:
A short description of how I read the JSON document contents into the jsonDocumentContents variable: The JSON document was stored inside a zipped folder, so retrieving it requires unzipping:
ZipArchive archive = new ZipArchive(zippedFolderStream);
foreach (ZipArchiveEntry entry in archive.Entries)
{
string jsonDocumentContents = new StreamReader(entry.Open()).ReadToEnd();
HttpResponseMessage response = await httpClient.PostAsJsonAsync($"http://127.0.0.1:9200/website/_bulk", jsonDocumentContents);
Console.WriteLine(await response.Content.ReadAsStringAsync());
}
UPDATE:
I just made a bulk request with the exact same contents using PostMan, and the request was successful. However, the error message persists when I make the same bulk request in C# using httpClient.PostAsJsonAsync(...).

I got it working by changing my code to the following:
ZipArchive archive = new ZipArchive(zippedFolderStream);
foreach (ZipArchiveEntry entry in archive.Entries)
{
string jsonDocumentContents = new StreamReader(entry.Open()).ReadToEnd();
StringContent content = new StringContent(jsonDocumentContents, Encoding.ASCII, mediaType: "application/json");
HttpResponseMessage response = await httpClient.PostAsync($"http://127.0.0.1:9200/website/_bulk", content);
Console.WriteLine(await response.Content.ReadAsStringAsync());
}
Notice that I am using HttpClient.PostAsync() instead of HttpClient.PostAsJsonAsync(), with a StringContent instance that specifies "application/json" as its media type.
I looked into the source code for HttpClient, and noticed that a new instance of JsonMediaTypeFormatter is created every time HttpClient.PostAsJsonAsync is called.
Since my POST requests are successful when I make them through PostMan, the issue must be caused by how PostAsJsonAsync() is implemented. I suspect, but have not verified, that the problem is due to the default properties in the JsonMediaTypeFormatter class.
To circumvent the problem I decided to use Http.PostAsync() with a correctly-configured StringContent instance.
Lo and behold, I can now send bulk requests to my ElasticSearch server using C#.

403 when calling AirBnb api from C#

I've not used HttpClient before so apologies if it's an obvious one.
I'm poking about with the airbnb api http://airbnbapi.org/#view-listing-info
My understanding of the endpoint is that I don't need an auth token, as this is a public endpoint I'm trying to use. Unfortunately I'm getting a 403 no matter what I try to do and I'm not entirely sure why.
I've got the following code:
client.DefaultRequestHeaders.Add("client_id", "<My client Id>");
client.DefaultRequestHeaders.Add("locale", "en-gb");
client.DefaultRequestHeaders.Add("currency", "gbp");
var request = new HttpRequestMessage()
{
RequestUri = new Uri($"https://api.airbnb.com/v2/listings/{id}"),
Method = HttpMethod.Get,
};
var task = client.SendAsync(request)
.ContinueWith((taskwithmsg) =>
{
var response = taskwithmsg.Result;
//var jsonTask = response.Content.ReadAsAsync<JsonResult>();
//jsonTask.Wait();
//var jsonObject = jsonTask.Result;
return response.Content;
});
task.Wait();
return task.Result;
And I'm getting the following response:
- response {StatusCode: 403, ReasonPhrase: 'Forbidden', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Connection: close
Date: Tue, 04 Apr 2017 19:44:40 GMT
Server: AkamaiGHost
Mime-Version: 1.0
Content-Length: 291
Content-Type: text/html
Expires: Tue, 04 Apr 2017 19:44:40 GMT
}} System.Net.Http.HttpResponseMessage
Any advice?
EDIT:
Macceturra wisely suggested I try to make the call with postman.
I've now established that I can make a call in postman and get a correct response back.

The request you're sending has the client_id as a HTTP header, when Airbnb is expecting it as a URL parameter.
Additionally, Airbnb requires the client to send an Accept (or User-Agent) header, or else it will still return "403 Forbidden" (probably should be "400 Bad Request").
Putting that together (and deleting the unnecessary headers):
var id = ...;
var clientId = ...;
var uri = new Uri($"https://api.airbnb.com/v2/listings/{id}?client_id={Uri.EscapeDataString(clientId)}&locale=en-gb&currency=GBP");
HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Add("Accept", "application/json");
await client.GetAsync(uri);

I've not been able to mark a single answer as the solution as it came through in the comments.
Solution
Start by making calls in postman. This helped me realise that I should have been passing the client id as a url parameter and not as a header. Bradley Grainger went on to answer this too.
Try and make that exact call with the HttpClient. This still returned the 403, but now we have confidence in the url we're sending.
Change the content type to application/json. This beats the 403 and gets us a 200 with the requested data.
Big thanks to all who helped piece these steps together.

You may need to add a cookie to the request called _aat which contains the airbnb access token. This is generated during login. You'll be able to see it by looking at the headers in Fiddler or the Net section of the browser dev tools when browsing one of your own listings when logged in. Just copy and paste it from there into your code to test if you can get the request working properly and then later you can automate the process of getting the _aat cookie within your code. Make sure the http headers in your code also match what is being sent via the browser.