I have a system that needs to make several calls to an API as a single 'job'.
I connect to a proxy via a hostname to carry out the job so it doesn't assign me an IP address I can see. I essentially hand over the task.
At the moment I'm using HtmlWeb to make the call to my URL list like so:
var webProxy = new WebProxy(Address: [MyProxyHost]);
var web = new HtmlWeb();
web.UseCookies = false;
web.UserAgent = "[MyUserAgent]";
var htmlDoc = web.Load("[URL]","GET", webProxy, webCreds);
//...do stuff with results...
The problem: as it stands, every request I make will rotate the proxy (as every 'call' using the proxy host randomly changes the IP their side - I have no control over this).
What I want to be able to do is re-use the http request/connection over several calls (I appreciate I probably won't be able to use HtmlWeb for this).
So I would essentially say:
Create HTTP Request Object (w/ Proxy host)
Open/Initialize
Make Request 1
...
Make Request N
Close & Dispose
Is this possible?
Ok, I've got the following in place and so far it seems to be ok:
var webProxy = new WebProxy()
{
Address = new Uri([ServerHost] + ":" + [ServerPort]),
BypassProxyOnLocal = false,
UseDefaultCredentials = false,
Credentials = new NetworkCredential(
userName: [Username],
password: [Password])
};
var httpClientHandler = new HttpClientHandler()
{
Proxy = webProxy,
};
var _httpClient = new HttpClient(httpClientHandler);
...
var response = await _httpClient.GetAsync([Url]);
var pageContents = await response.Content.ReadAsStringAsync();
var htmlDoc.LoadHtml(pageContents);
...
This makes the request to each URL using the proxy and converts the page returned to a HtmlDocument so I can do the AgilityPack stuff I need to.
Related
I need to send the HTTP Request to the URL of online Data Management Systems. I am using the following codes:
public async Task StartCrawl()
{
var url = "http://pww.comopsdms.healthcare.philips.com/livelink/llisapi.dll?func=ll&objId=3890195&objAction=browse&sort=name&viewType=1";
var httpClient = new HttpClient();
var html = await httpClient.GetStringAsync(url);
var htmlDoc = new HtmlAgilityPack.HtmlDocument();
}
Here, it works with all the other URL links but does not work with the above link. HttpClient sends the request but receives nothing. And flow does not go to the next line.
The link only works inside the company network.
I'm trying to pass credentials back to a web service using HttpClient.
However, I keep getting an Unauthorized request.
However, when I try using a WebRequest it authenticates?
HttpClient:
var handler = new NativeMessageHandler
{
UseDefaultCredentials = true,
Credentials = credential
};
var client = new HttpClient(handler);
var content = _httpClientHelper.Serialize(data);
var response = await _client.PostAsync($"{_baseurl}/api/foos/List", content);
WebRequest:
HttpResponseMessage response = null;
try
{
var data = JsonConvert.SerializeObject(new
{
ViewTitle = "New",
PageCount = 60
});
var content = _httpClientHelper.Serialize(data);
using (var client = new WebClient { UseDefaultCredentials = true, Credentials = credentials })
{
client.Headers.Add(HttpRequestHeader.ContentType, "application/json; charset=utf-8");
client.UploadData("$"{baseurl}/api/foos/List", "POST", Encoding.UTF8.GetBytes(content));
}
I cannot figure out why one works and the other does not.
Any help or insight on this would be greatly appreciated
As noted here and here this behavior of HttpClient could be because of how HttpClientHandler is implemented.
"[..] the StartRequest method is executed in new thread with the credentials of the asp.net process (not the credentials of the impersonated user) [..]"
You might be seeing the difference in behavior of HttpClient and WebClient because
"HttpClient creates new threads
via the Task Factory. WebClient on the other hand, runs synchronously
on the same thread thereby forwarding its
credentials (i.e. the credentials of the impersonated user) ."
I have some code that is making a Server2Server call using an HttpClient. Here is some quick code
Code to make a request
private HttpRequestMessage GetRequest(Uri uri, NameValueCollection headers)
{
var method = HttpMethod.Get;
var request = new HttpRequestMessage(method, uri);
foreach (string v in headers)
{
var success = request.Headers.TryAddWithoutValidation(v, headers[v]);
if (!success)
{
// log something ( no logs are appearing )
}
}
return request;
}
Code to make the request
private void AsyncCallUrl(Uri url, NameValueCollection headers)
{
object result1 = null;
var handler = new HttpClientHandler() { AllowAutoRedirect = false };
using (HttpClient client = new HttpClient(handler))
{
var request = GetRequest(url, headers);
using (HttpResponseMessage response = client.SendAsync(request).Result) // when this line is executed the request object's domain is changed to something else
{
using (HttpContent content = response.Content)
{
result1 = content.ReadAsStringAsync().Result;
}
}
}
I've verified that the request object is created appropriately with the proper domain. I've also verified that the network traffic is going to the wrong domain, even the request object shows the new bad domain. What I don't is why this is happening. I've even set the AllowAutoRedirect to false
NOTE: As a note I notice that if I use GetAsync instead of SendAsync the domain change doesn't happen. However this is not an ideal solution as in order to add headers I would need to add them to the HttpClient itself and this code lives on an IIS server and I don't want to make a new client for every request
So, with SendAsync the value of the Host header of the request is determined by the uri parameter... However it is possible to override the Host header through the Headers property of the request.
It's highly likely that the NameValueCollection headers that you are blindly injecting into the request's headers contains an entry for Host which is different to that which you supplied in the Uri.
As an aside, this behaviour can be useful, if (for instance) you were to discover that the DNS performance of HttpWebRequest (the business end of HttpClient on Windows) is sub-standard. I've bypassed .Net/Windows DNS by using a third party library to look up the IP of the host, rewriting the Uri to use the IP address in place of the host name, then setting the Host header on the outgoing request back to the original host name.
I'm trying to create a https client in C#.
I had HTTP client which worked fine and I changed it to work with HTTPS. But unfortunately there is something wrong with the authorization (the server uses OAuth 2).
My program sends a request to a server and gets the token. But it can't get or send any data from the server.
The server works fine with other clients, so it's not its fault.
This is a piece of code which causes the problem. I know that, because when I comment authorization on the server, the data is send (everything is fine).
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue(
"Basic", Convert.ToBase64String(System.Text.ASCIIEncoding.ASCII.GetBytes(
string.Format("Authorization: {0}", token))));
This is the whole function, which should send data:
WebRequestHandler handler = new WebRequestHandler();
X509Certificate certificate = GetMyX509Certificate();
handler.ClientCertificates.Add(certificate);
var client = new HttpClient(handler);
string uri = "https://192.168.0.10:8443/data";
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue(
"Basic", Convert.ToBase64String(System.Text.ASCIIEncoding.ASCII.GetBytes(
string.Format("Authorization: {0}", token))));
client.BaseAddress = new Uri(uri);
var parameters = new Dictionary<string, string>();
parameters["name"] = name;
parameters["surname"] = surname;
JavaScriptSerializer serializer = new JavaScriptSerializer();
var json = serializer.Serialize(parameters);
System.Net.ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
var response = client.PostAsync(uri, new StringContent(json, System.Text.Encoding.UTF8, "application/json")).Result;
Console.WriteLine((response.StatusCode.ToString()));
string resultContent = response.Content.ReadAsStringAsync().Result;
Console.WriteLine(resultContent);
I guess I'm missing something in the header but can't find any information in the documentation about that.
It's a difficult issue so any advice will be very appreciated.
You shouldn't be including the HTTP header name ("Authorization: ") in the parameter of the AuthenticationHeaderValue. Setting the Authorization property will add the header to the request.
Additionally for OAuth 2, you probably want to be using "Bearer" as the scheme and not encoding token with base64.
Something like this should therefore work:
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
I am creating a Metro App that makes a HTTP Post request to a webserver to obtain JSON data. Initially I use the same code to logon to the web server and the HTTP Post request returns fine. It is only later that I run into an issue where the code hangs when I call the SendAsync() method.
I used wireshark to view the network traffic and I did see the server returning a response. So I am not sure why the call is not completing. Anyone have any ideas?
Here is the code I am using:
var httpHandler = new HttpClientHandler();
httpHandler.CookieContainer = __CookieJar;
var httpClient = new HttpClient(httpHandler);
UserAgentDetails userAgent = UserAgentDetails.GetInstance();
httpClient.DefaultRequestHeaders.UserAgent.ParseAdd(userAgent.UserAgentString);
foreach (string key in __colHeaders)
httpClient.DefaultRequestHeaders.Add(key, __colHeaders[key]);
var content = new StringContent(postData);
if (contentType != null && contentType.Length > 0)
content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue(contentType);
var requestMsg = new HttpRequestMessage(HttpMethod.Post, new Uri(url));
requestMsg.Content = content;
requestMsg.Headers.TransferEncodingChunked = true;
var responseMsg = await httpClient.SendAsync(requestMsg);
// no return from method after logon