Develop Reference

Obtain the entire http Request including headers prior to a request being sent to the server

I can obtain the details of my http request including headers via Fiddler, however I'd like to display the value of my http request and response as part of my MVC web service to ensure the request is valid.
I want to return all of the details of the request as a string so I can return the result to the View for this specific controller.
I am using C# to create the request:
var _httpClientManager = new HttpClientManager(_httpClientPool, _Identity, _errorLogger);
_httpClientManager.HttpRequestHeaders = new Dictionary<string, string>();
_httpClientManager.HttpRequestHeaders.Add("Accept", "application/json");
... extra headers omitted.....
... url and request JSON is omitted
var response = await _httpClientManager.PostAsJsonAsync<IdentityVerificationRequest>(url, request, null);
return View(response);
Basically I'd like to obtain what I am about to Post to the server and include that in the View response.
Its probably a bit of a naïve question, I am quite new working with this.
Is it possible?
Thankyou.

POSTing form data using Azure data factory web activity returns different results to C# script

I am getting to the stage of hair pulling with this one, I'm hoping someone can see if I'm doing anything wrong.
I'm trying to POST some form data to website using Azure data factory web activity however whilst I get a response (I get the page and some headers) it is different to the response I get if I make the exact same request using C# and HttpClient code. I've used fiddler to view the request being post'd using my C# script and according to the request information given in data factory they are exactly the same - so same headers, same content format etc...
This POST request is to login to a website which has a custom login mechanism, so no OAuth or anything like that unfortunately. It is supposed to return a cookie, which it does if I use my C# script, but if I make the same POST request using data factory web activity then I get different html sent back (it just returns the same login screen) and also different set of response headers in the "ADFWebActivityResponseHeaders" part of the activity output!?! See below for what is returned in the web activity output response headers:-
"ADFWebActivityResponseHeaders": {
"Pragma": "no-cache",
"Vary": "Accept-Encoding",
"X-Frame-Options": "DENY",
"Cache-Control": "no-store, must-revalidate, no-cache, post-check=0, pre-check=0",
"Date": "Wed, 09 Sep 2020 08:09:30 GMT",
"Server": "Microsoft-IIS/8.5"
}
If I do this via C# I also get a 'Set-Cookie' as well (strangely if I make a 'GET' request for the homepage of this site I do get a 'Set-Cookie' in the response!!!), but never when doing this via data factory. I'm struggling to see how this is possible unless data factory is modifying my request in some fashion? Below is my C# code, pretty simple/standard:-
var handler = new HttpClientHandler();
handler.CookieContainer = new CookieContainer();
handler.UseCookies = true;
handler.UseDefaultCredentials = false;
// Create our http client which will perform our web requests
var HttpClient = new HttpClient(handler);
HttpClient.BaseAddress = new Uri("**REMOVED**");
// Some of the extracts take a LONG time, so set the timeout for default of 30mins
HttpClient.Timeout = TimeSpan.FromMinutes(30);
// Set the 'form' parameters we're going to POST to the server in the request
var parameters = new Dictionary<string, string>
{
{ "username", "**REMOVED**" },
{ "password", "**REMOVED**" }
};
// URL encode the parameters
var content = new FormUrlEncodedContent(parameters);
// Submit our POST with the parameters
var response = await HttpClient.PostAsync("**REMOVED**", content);
Running this code and using fiddler I see the following request with headers, these are the only headers:-
Content-Length: 80
Content-Type: application/x-www-form-urlencoded
username=REMOVED&password=REMOVED
and in the 'input' side of the web activity is the details of the request, I've added the headers in the web activity and these are correct:-
"method": "POST",
"headers": {
"Content-Type": "application/x-www-form-urlencoded",
"Content-Length": 80
},
"body": "username=REMOVED&password=REMOVED"
Note that in the data factory I'm using a self hosted integration runtime as this website blocks addresses that do not come from the specific IP addresses used externally by our on-prem network/firewall. I know that is not the problem as I'm getting a response with the normal login page from the site (if I use the Azure integration runtime I get a denied response).
Here is a screen shot of the web activity in data factory:-
Really hope someone out there can see what I'm missing or whatever...

Turns out this does work and will list the cookies in the JSON output from the activity as shown below (note this is to be found in the output of the ADF activity, so you would pick up the cookie from the output a bit like... #activity('Login and get cookie').output.ADFWebActivityResponseHeaders["Set-Cookie"] )
However, in my case the url I was POSTing to was responding with a 302 (moved temporarily) but the 'Location' header which should be there is not in the ADFWebActivityResponseHeaders - which is why I missed it. I tried using Chrome with the developer tools and looked at the response directly which is where I found the 302 response code. After that, I just used the new URL given in the response headers (i.e. the url in the 'Location') that I found when using the browser dev tools.
Unfortunately at the time of writing, the Azure data factory HTTP activity does not follow redirects (and doesn't list all the response headers either!) so if anyone encounters the same problem they will need to manually find out and get the url's for any redirects. In other words, try using a tool like browser/postman and look at the response if it doesn't work in ADF... you might find there is a redirect going on :-)
There is a feature request logged for this here, be sure to add your vote :)
edited to update the Azure feedback change of URL after MS decided to change things on the feedback site!?!

How to send a GET request exactly like I have it in fiddler?

So I am trying to build a program in C#. I want to send a really specific request. Exactly as I receive it in fiddler. Right now I am using fiddlers composer feature to send the request. Here is how I get the request in fiddler
How can I send it exactly as shown on the picture but in a C# console application?
Basically looking for a way to get a request in fiddler and implement it directly in to my code in the format that fiddler gives it to me.(In fiddler I can save the request in .txt format. If I could use that same .txt format to send the request then it would be really helpful)

You need to create HttpClient instance and set the headers shown in the fiddler to the same values. Hope this helps. Your URL was HTTPS hence you will need to set for certificate validation OR you will have to set avoid certificate errors on the call.
// Create a client
HttpClient httpClient = new HttpClient();
// Add a new Request Message
HttpRequestMessage requestMessage = new HttpRequestMessage(HttpMethod.Put, "https://yoursitehere/");
// Add our custom headers
requestMessage.Headers.Add("User-Agent", "User-Agent-Here");
requestMessage.Headers.Add("Connection", "MIME-Type-Here");
requestMessage.Headers.Add("Cache-Control", "value-from-fiddler");
requestMessage.Headers.Add("Accept-Language", "value-from-fiddler");
requestMessage.Headers.Add("Accept-Encoding", "value-from-fiddler");
requestMessage.Headers.Add("Accept", "value-from-fiddler");
requestMessage.Headers.Add("User-Agent", "value-from-fiddler");
// Send the request to the server
HttpResponseMessage response = await httpClient.SendAsync(requestMessage);
// Just as an example I'm turning the response into a string here
string responseAsString = await response.Content.ReadAsStringAsync();

BrokerProperties REST header values not showing up in .NET client code

I'm writing two small pieces of C# code. The first is for a client-side Portable Class Library. All it does is send messages to an Azure Service Bus topic via the Azure Service Bus REST API, using HttpClient.
I populate the BrokerProperties header on the REST call with valid JSON, and I expect that on the server side, when I receive the message through a subscription, that I'll get my instance of BrokeredMessage.Properties populated with the values I sent from the client.
The one problem I've had on this side is that the documentation says to set Content-Type to application/atom+xml;type=entry;charset=utf-8, but even when I do I get application/json; charset=utf-8, so I'm just using application/json.
With that aside, as far as I can tell, this does what it's supposed to do. It creates the client and the request message, sets the headers, and sends the message. I get a 201 Created every time. Here's all of it:
private async static void SendServiceBusMessage(Command command)
{
// Create the HttpClient and HttpRequestMessage objects
HttpClient client = new HttpClient();
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, topicUri);
// Add the authorization header (CreateAuthToken does the SHA256 stuff)
request.Headers.Add("Authorization", CreateAuthToken(topicUri, authSasKeyName, authSasKey));
// Add the content (command is a normal POCO)
// I've tried application/atom+xml;type=entry;charset=utf-8, always see application/json in the request
request.Content = new StringContent(JsonConvert.SerializeObject(command), Encoding.UTF8, "application/json");
// Add the command name and SessionId as BrokeredMessage properties
var brokeredMessageProperties = new Dictionary<string, string>();
brokeredMessageProperties.Add("CommandName", command.GetType().Name);
brokeredMessageProperties.Add("SessionId", Guid.NewGuid().ToString());
// Add the BrokerProperties header to the request
request.Content.Headers.Add("BrokerProperties", JsonConvert.SerializeObject(brokeredMessageProperties));
// I've also tried adding it directly to the request, nothing seems different
// request.Headers.Add("BrokerProperties", JsonConvert.SerializeObject(brokeredMessageProperties));
// Send it
var response = await client.SendAsync(request);
if (!response.IsSuccessStatusCode)
{
// Do some error-handling
}
}
and here's an example of the HTTP request it sends. Compare it to the example at the bottom of Send Message documentation... aside from the Content-Type, it looks (functionally) identical to me.
POST https://myawesomesbnamespace.servicebus.windows.net/commands/messages HTTP/1.1
Authorization: SharedAccessSignature sr=https%3A%2F%2Fmyawesomesbnamespace.servicebus.windows.net%2Fcommands%2Fmessages&sig=SomeValidAuthStuffHere
Content-Type: application/json; charset=utf-8
BrokerProperties: {"CommandName":"CreateJob_V1","SessionId":"94932660-54e9-4867-a020-883a9bb79fa1"}
Host: myawesomesbnamespace.servicebus.windows.net
Content-Length: 133
Expect: 100-continue
Connection: Keep-Alive
{"JobId":"6b76e7e6-9499-4809-b762-54c03856d5a3","Name":"Awesome New Job Name","CorrelationId":"47fc77d9-9470-4d65-aa7d-690b65a7dc4f"}
However, when I receive the message on the server, the .Properties are empty. This is annoying.
The server code looks like this. It just gets a batch of messages and does a foreach loop.
private async Task ProcessCommandMessages()
{
List<BrokeredMessage> commandMessages = (await commandsSubscriptionClient.ReceiveBatchAsync(serviceBusMessageBatchSize, TimeSpan.FromMilliseconds(waitTime_ms))).ToList();
foreach (BrokeredMessage commandMessage in commandMessages)
{
// commandMessage.Properties should have CommandName and SessionId,
// like I sent from the client, but it's empty
// that's not good
if (commandMessage.Properties.ContainsKey("CommandName"))
{
string commandName = commandMessage.Properties["CommandName"] as string;
// Do some stuff
}
else
{
// This is bad, log an error
}
}
}
So, I'm a bit stuck. Can anyone spot something I'm doing wrong here? Maybe it's the Content-Type problem and there's a way around it?
Thanks!
Scott
Seattle, WA, USA

OK, finally getting back to this. What I misunderstood (and I'd argue the documentation isn't clear about) is that arbitrary properties cannot be passed through the BrokerProperties header. Only named properties from the BrokeredMessage class (like SessionId, Label, etc.) will come through Service Bus to the server.
For properties to show up in BrokeredMessage.Properties, they have to be passed as custom headers on the request. So, in my case,
request.Headers.Add("CommandName", command.GetType().Name);
gets the CommandName property to show up on the server after the message is passed through Service Bus.
And to pass the SessionId value, I'll still want to pass it through BrokerProperties header.

RestSharp Accept header change

I am using RestSharp for developing on the client side. I am also using Ruby Grape gem for my custom API on server side. Grape gem can do versioning by setting Accept HTTP header f.e to application/vnd.twitter-v1+json
And test command via console works perfect
curl -H Accept=application/vnd.twitter-v1+json /statuses/public_timeline
But when I am trying to set up header for RestRequest I am getting error 404 on the server.
I have no idea why so. I have found another issue that server returns 406 error - but in my case 404.
How can I put custom value for Accept header?

You can set a custom Accept header with the AddHeader method...
var client = new RestClient("http://example.com/api");
var request = new RestRequest("statuses/public_timeline", Method.GET);
request.AddHeader("Accept", "application/vnd.twitter-v1+json");
var response = client.Execute(request);
var json = response.Content;
This should work fine if you are willing to deserialize the JSON yourself.
If you want to make use of the generic Execute<T> method, which does automatic deserialization for you, you will run into problems...
From the RestSharp documentation about deserialization:
RestSharp includes deserializers to process XML and JSON. Upon receiving a response, RestClient chooses the correct deserializer to use based on the Content Type returned by the server. The defaults can be overridden (see Customization). The built-in content types supported are:
application/json – JsonDeserializer
application/xml – XmlDeserializer
text/json – JsonDeserializer
text/xml – XmlDeserializer
* – XmlDeserializer (all other content types not specified)
This is saying that, by default, if the response's content type is not one of those listed, RestSharp will attempt to use the XmlDeserializer on your data. This is customizable though with extra work.