Develop Reference

Why HttpClient is not allowed for auto redirect?

I am trying to crawl some domains with different user-agents. My crawler works fins, the problem happens when a domain does not have an SSL certificate and is insecure, in that case, I do not get any response with HttpClient. To skip that I use HttpHandler and set the certificate myself.
With this solution I get 301 for all those domains, it feels like my AllowAutoRedirect is false however it is not. I tried and assigned MaxAutomaticRedirections to 5, that did not work as well.
Here is my code:
public Task<int> Crawl(string userAgent, string url)
{
var handler = new HttpClientHandler();
handler.ClientCertificateOptions = ClientCertificateOption.Manual;
handler.ServerCertificateCustomValidationCallback =
(httpRequestMessage, cert, cetChain, policyErrors) =>
{
return true;
};
var httpClient = new HttpClient(handler);
httpClient.DefaultRequestHeaders.Add("User-Agent", userAgent);
var statusCode = (int)(await httpClient.SendAsync(new HttpRequestMessage(HttpMethod.Get, URL))).StatusCode;
return statusCode;
}

Domains that I was trying to crawl did not have any SSL certificates and HttpClient was redirected to the HTTP version. My guess is HttpClient did not have any clue where it was redirected to, so just did not continue.
My problem got solved by crawling HTTP version of domains, for example: http://example.com

Passing Credentials works for WebRequest but not for HttpClient

I'm trying to pass credentials back to a web service using HttpClient.
However, I keep getting an Unauthorized request.
However, when I try using a WebRequest it authenticates?
HttpClient:
var handler = new NativeMessageHandler
{
UseDefaultCredentials = true,
Credentials = credential
};
var client = new HttpClient(handler);
var content = _httpClientHelper.Serialize(data);
var response = await _client.PostAsync($"{_baseurl}/api/foos/List", content);
WebRequest:
HttpResponseMessage response = null;
try
{
var data = JsonConvert.SerializeObject(new
{
ViewTitle = "New",
PageCount = 60
});
var content = _httpClientHelper.Serialize(data);
using (var client = new WebClient { UseDefaultCredentials = true, Credentials = credentials })
{
client.Headers.Add(HttpRequestHeader.ContentType, "application/json; charset=utf-8");
client.UploadData("$"{baseurl}/api/foos/List", "POST", Encoding.UTF8.GetBytes(content));
}
I cannot figure out why one works and the other does not.
Any help or insight on this would be greatly appreciated

As noted here and here this behavior of HttpClient could be because of how HttpClientHandler is implemented.
"[..] the StartRequest method is executed in new thread with the credentials of the asp.net process (not the credentials of the impersonated user) [..]"
You might be seeing the difference in behavior of HttpClient and WebClient because
"HttpClient creates new threads
via the Task Factory. WebClient on the other hand, runs synchronously
on the same thread thereby forwarding its
credentials (i.e. the credentials of the impersonated user) ."

Xamarin.Mac - How to pass current credentials to REST endpoint

Unable to use the System.Net API to authenticate current user to a REST endpoint. Example below returns 401 unauthorized
using (HttpClient client = new HttpClient())
{
using (HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, "http://rest/api/endpoint")
{
using (httpResponseMessage response = await client.SendAsync(request))
{
if (response.IsSuccessStatusCode)
//do something
}
}
}
If I use NSUrlConnection I am able to authenticate not problem. So, NSUrlConnection must be passing the credentials some how. Below is a snippet of that code:
var request = new NSMutableUrlRequest(new NSUrl("http://rest/api/endpoint"), NSUrlRequestCachePolicy.ReloadIgnoringCacheData, 0);
request["Accept"] = "application/json";
NSUrlConnection.SendAsynchronousRequest(request, NSOperationQueue.MainQueue, delegate(NSUrlResponse, response, NSData data, NSError error)
{
// successfully authenticated and do something with response
});
I would like to wrap my service code in a PCL to share with other platforms. Therefore, I would like to get this all working within the System.Net api. Is this possible?
UPDATE:
I've tried using an HttpClientHandler and using default credentials as well as CredentialCache.DefaultNetworkCredentials. The only way to get this to work is to hardcode the credentials, which I do not want. It appears the System.Net stack does not surface the credentials from the OS.

I assume it's using default authentication. You could do that with HttpClient with an HttpClientHandler, for example:
using (var handler = new HttpClientHandler {
UseDefaultCredentials = true
})
using (var client = new HttpClient(handler))
{
var result = await client.SendAsync(...);
}

HttpClient access url with '#' (at symbol)

I'm integrating a service that returns a key when I a GET request to a URL that is in the following format:
https://username:password#service.com/refresh.key
When I access the URL in my browser, it returns the new key as expected, by when I do a GET request using HttpClient I get a 401.
HttpClient _client = new HttpClient();
var response = await _client.GetAsync(#"https://username:password#service.com/refresh.key"); // Returns a 401
I think it has something to do with the '#' in the URL, but I'm not sure how to fix it, I tried replacing it with '%40', but when I do that I get a UriFormatException.
Does anyone know how to do this?

You should modify Authorization header of HttpClient, can you try the code below;
HttpClient _client = new HttpClient();
byte[] usernamePasswordBytes = Encoding.ASCII.GetBytes("user:pass");
_client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", Convert.ToBase64String(usernamePasswordBytes));
var response = await _client.GetAsync(#"https://service.com/refresh.key");
PS: Such username:pass#domain.com requests are BasicAuthentication request so in fact you try to make basic authentication request.
Hope this works for you

You don't need to provide credentials in url. Instead you can do:
using (var handler = new HttpClientHandler {Credentials = new NetworkCredential("username", "password")}) {
using (HttpClient _client = new HttpClient(handler)) {
var response = await _client.GetAsync(#"https://service.com/refresh.key");
}
}

How can I tell if HttpClient is connecting via HTTPS

I am using HttpClient to talk to an API. The server will automatically redirect http:// requests to https:// if enabled. So, in order to protect the users API key I want to create a test connection to the website to see if I am redirected before sending over the API Key.
The HttpClient redirects properly, but I cannot seem to find a decent way to find out if the client is using HTTPS or not. I know I could test whether https:// exists within response.RequestMessage.RequestUri but this seems a little flaky
public void DetectTransportMethod()
{
using (HttpClient client = new HttpClient())
{
client.Timeout = TimeSpan.FromSeconds(20);
using (HttpResponseMessage response = client.GetAsync(this.HttpHostname).Result)
{
using (HttpContent content = response.Content)
{
// check if the method is HTTPS or not
}
}
}
}

The documentation of HttpClient (MSDN) and HttpResponseMessage (MSDN) do not contain any methods that can be used to determine whether a request has been made over https. Even though checking the URI of the HttpResponseMessage does indeed sound flaky I'm afraid it's the easiest and most readable option. Implementing this as a extension method for HttpResponseMessage probably is the most readable. To ensure that the HttpClient you are using can be redirected make sure that the WebRequestHandler (MSDN) passed into the HttpClient has the AllowAutoRedirect (MSDN) property set to true.
See the following extension method:
static class Extensions
{
public static bool IsSecure(this HttpResponseMessage message)
{
rreturn message.RequestMessage.RequestUri.Scheme == "https";
}
}
And the following Console application that demonstrates it works. For this to work however, the HTTP server has to upgrade the connection to https (Like Facebook does) and the AllowAutoRedirect property of the WebRequestHandler has to be set to true.
static void Main(string[] args)
{
using (var client = new HttpClient(new HttpClientHandler { AllowAutoRedirect = true}))
{
client.Timeout = TimeSpan.FromSeconds(20);
using (var response = client.GetAsync("http://www.geenstijl.nl").Result)
{
Console.WriteLine(response.IsSecure());
}
using (var response = client.GetAsync("http://www.facebook.com").Result)
{
Console.WriteLine(response.IsSecure());
}
}
Console.ReadLine();
}