I am using HttpClient to talk to an API. The server will automatically redirect http:// requests to https:// if enabled. So, in order to protect the users API key I want to create a test connection to the website to see if I am redirected before sending over the API Key.
The HttpClient redirects properly, but I cannot seem to find a decent way to find out if the client is using HTTPS or not. I know I could test whether https:// exists within response.RequestMessage.RequestUri but this seems a little flaky
public void DetectTransportMethod()
{
using (HttpClient client = new HttpClient())
{
client.Timeout = TimeSpan.FromSeconds(20);
using (HttpResponseMessage response = client.GetAsync(this.HttpHostname).Result)
{
using (HttpContent content = response.Content)
{
// check if the method is HTTPS or not
}
}
}
}
The documentation of HttpClient (MSDN) and HttpResponseMessage (MSDN) do not contain any methods that can be used to determine whether a request has been made over https. Even though checking the URI of the HttpResponseMessage does indeed sound flaky I'm afraid it's the easiest and most readable option. Implementing this as a extension method for HttpResponseMessage probably is the most readable. To ensure that the HttpClient you are using can be redirected make sure that the WebRequestHandler (MSDN) passed into the HttpClient has the AllowAutoRedirect (MSDN) property set to true.
See the following extension method:
static class Extensions
{
public static bool IsSecure(this HttpResponseMessage message)
{
rreturn message.RequestMessage.RequestUri.Scheme == "https";
}
}
And the following Console application that demonstrates it works. For this to work however, the HTTP server has to upgrade the connection to https (Like Facebook does) and the AllowAutoRedirect property of the WebRequestHandler has to be set to true.
static void Main(string[] args)
{
using (var client = new HttpClient(new HttpClientHandler { AllowAutoRedirect = true}))
{
client.Timeout = TimeSpan.FromSeconds(20);
using (var response = client.GetAsync("http://www.geenstijl.nl").Result)
{
Console.WriteLine(response.IsSecure());
}
using (var response = client.GetAsync("http://www.facebook.com").Result)
{
Console.WriteLine(response.IsSecure());
}
}
Console.ReadLine();
}
Related
I have some code that is making a Server2Server call using an HttpClient. Here is some quick code
Code to make a request
private HttpRequestMessage GetRequest(Uri uri, NameValueCollection headers)
{
var method = HttpMethod.Get;
var request = new HttpRequestMessage(method, uri);
foreach (string v in headers)
{
var success = request.Headers.TryAddWithoutValidation(v, headers[v]);
if (!success)
{
// log something ( no logs are appearing )
}
}
return request;
}
Code to make the request
private void AsyncCallUrl(Uri url, NameValueCollection headers)
{
object result1 = null;
var handler = new HttpClientHandler() { AllowAutoRedirect = false };
using (HttpClient client = new HttpClient(handler))
{
var request = GetRequest(url, headers);
using (HttpResponseMessage response = client.SendAsync(request).Result) // when this line is executed the request object's domain is changed to something else
{
using (HttpContent content = response.Content)
{
result1 = content.ReadAsStringAsync().Result;
}
}
}
I've verified that the request object is created appropriately with the proper domain. I've also verified that the network traffic is going to the wrong domain, even the request object shows the new bad domain. What I don't is why this is happening. I've even set the AllowAutoRedirect to false
NOTE: As a note I notice that if I use GetAsync instead of SendAsync the domain change doesn't happen. However this is not an ideal solution as in order to add headers I would need to add them to the HttpClient itself and this code lives on an IIS server and I don't want to make a new client for every request
So, with SendAsync the value of the Host header of the request is determined by the uri parameter... However it is possible to override the Host header through the Headers property of the request.
It's highly likely that the NameValueCollection headers that you are blindly injecting into the request's headers contains an entry for Host which is different to that which you supplied in the Uri.
As an aside, this behaviour can be useful, if (for instance) you were to discover that the DNS performance of HttpWebRequest (the business end of HttpClient on Windows) is sub-standard. I've bypassed .Net/Windows DNS by using a third party library to look up the IP of the host, rewriting the Uri to use the IP address in place of the host name, then setting the Host header on the outgoing request back to the original host name.
Unable to use the System.Net API to authenticate current user to a REST endpoint. Example below returns 401 unauthorized
using (HttpClient client = new HttpClient())
{
using (HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, "http://rest/api/endpoint")
{
using (httpResponseMessage response = await client.SendAsync(request))
{
if (response.IsSuccessStatusCode)
//do something
}
}
}
If I use NSUrlConnection I am able to authenticate not problem. So, NSUrlConnection must be passing the credentials some how. Below is a snippet of that code:
var request = new NSMutableUrlRequest(new NSUrl("http://rest/api/endpoint"), NSUrlRequestCachePolicy.ReloadIgnoringCacheData, 0);
request["Accept"] = "application/json";
NSUrlConnection.SendAsynchronousRequest(request, NSOperationQueue.MainQueue, delegate(NSUrlResponse, response, NSData data, NSError error)
{
// successfully authenticated and do something with response
});
I would like to wrap my service code in a PCL to share with other platforms. Therefore, I would like to get this all working within the System.Net api. Is this possible?
UPDATE:
I've tried using an HttpClientHandler and using default credentials as well as CredentialCache.DefaultNetworkCredentials. The only way to get this to work is to hardcode the credentials, which I do not want. It appears the System.Net stack does not surface the credentials from the OS.
I assume it's using default authentication. You could do that with HttpClient with an HttpClientHandler, for example:
using (var handler = new HttpClientHandler {
UseDefaultCredentials = true
})
using (var client = new HttpClient(handler))
{
var result = await client.SendAsync(...);
}
My team maintains a tool that is responsible for doing rapid verification of over 1000 different client websites. The tool is a Windows Service (.NET 4.5.2, C#) that reads requests from a queue, and executes a 'health check' for each request. It usually handles over 500 requests a minute, but can be responsible for more. Each request takes a second or two to execute.
A request contains a Uri and credentials needed for doing the health check. A health check is a POST against the AUTH page with the credentials (the app has custom auth, it's not header based auth), and then a GET to the home page, with a quick verification that it's the home page we expect. It then goes to a status page in the application, and does some quick checks against that. The GET requests have to use the cookies from the Set-Cookie header in the auth post.
We've been having performance problems with the tool as it scales. It currently creates a new HttpWebRequest object for each post and get in the process. There is a shared CookieContainer that is populated by the first post, so that we can get to the home page and then the status page.
I want to change this service to use the HttpClient object available in .NET 4.5. The problem is everywhere I read online says you want to avoid rapid creation and destruction of HttpClients. You'd rather keep one instance alive for the lifetime of the application. The problem I have is that HttpClient seems to work really well against one endpoint, not many.
I have looked into several options, and am not sure which is best to proceed:
Create a new HttpClient for each request, and use it for the duration of that request. That means it will live for a couple seconds, and be used for 3 calls. This would not be easy to implement, but I'm concerned about the overhead of creating and destroying hundreds of HttpClients a minute.
Figure out if it's possible to use one HttpClient instance for different endpoints by avoiding usage of a BaseAddress, and using the client to pass HttpRequestMessages using SendAsync. I haven't been able to figure out cookies with this method yet. To avoid having the HttpClient store the cookies, I set UseCookies to false in the HttpClientHandler, and tried managing cookies via headers in the HttpRequest/ResponseMessages themselves, but it looks like HttpClient simply strips cookies when UseCookies is set to false, so I was unable to pass cookies between request. edit: cookies work fine because they are stored per domain.
Store several hundred different HttpClient instances in some sort of dictionary, and pull the appropriate one for each Uri as the requests come in. I'm not sure about the memory overhead on this though. Also each unique Uri is only verified once every 5 minutes, so I'm not sure if having an HttpClient used once every 5 minutes keeps an unnecessary number of ports open.
Keep using HttpWebRequests. Maybe this older method still performs better in this situation.
If anyone has faced a similar issue, I'd love some input on where to proceed on this.
Thanks!
The problem with creating new HttpClients for each request is that HttpClientHandler will close the underlying TCP/IP connection. However, if you are using each HttpClient for the 3 requests to one host and then hitting a different host, then keeping the connection open doesn't help when you move to a new host. So, you probably will not see perf problem with one client per host. HttpClient itself is a very lightweight object. It isn't going to cost much to create one.
However, HttpClient simply delegates the real work to HttpClientHandler which uses HttpWebRequest under the covers, therefore will be unlikely to have any better performance than directly using HttpWebRequest.
If you are looking for better performance, then I suggest looking into replacing HttpClientHandler with the new WinHttpHandler which bypasses HttpWebRequest and goes directly to the Win32 API to make calls.
The full source is available for WinHttpHandler on GitHub so you can see exactly how it handles cookies and credentials.
And I would be really interested to hear if you do get much better perf with WinHttpHandler.
To start with, what part of this would you need modified to suit your needs?
var urisToCheck = new List<Uri>(); //get these somehow
//basic auth work?
var credentials = new NetworkCredential("user", "pass");
var handler = new HttpClientHandler { Credentials = credentials };
var client = new HttpClient(handler);
Parallel.ForEach(urisToCheck,
async uri =>
{
var response = await client.GetAsync(uri.AbsoluteUri);
//check for whatever you want here
}
);
here's my basic API client that uses the same HttpClient object for every request.
using System;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Threading.Tasks;
//You need to install package Newtonsoft.Json > https://www.nuget.org/packages/Newtonsoft.Json/
using Newtonsoft.Json;
using Newtonsoft.Json.Serialization;
public class MyApiClient : IDisposable
{
private readonly TimeSpan _timeout;
private HttpClient _httpClient;
private HttpClientHandler _httpClientHandler;
private readonly string _baseUrl;
private const string ClientUserAgent = "my-api-client-v1";
private const string MediaTypeJson = "application/json";
public MyApiClient(string baseUrl, TimeSpan? timeout = null)
{
_baseUrl = NormalizeBaseUrl(baseUrl);
_timeout = timeout ?? TimeSpan.FromSeconds(90);
}
public async Task<string> PostAsync(string url, object input)
{
EnsureHttpClientCreated();
using (var requestContent = new StringContent(ConvertToJsonString(input), Encoding.UTF8, MediaTypeJson))
{
using (var response = await _httpClient.PostAsync(url, requestContent))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
}
public async Task<TResult> PostAsync<TResult>(string url, object input) where TResult : class, new()
{
var strResponse = await PostAsync(url, input);
return JsonConvert.DeserializeObject<TResult>(strResponse, new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver()
});
}
public async Task<TResult> GetAsync<TResult>(string url) where TResult : class, new()
{
var strResponse = await GetAsync(url);
return JsonConvert.DeserializeObject<TResult>(strResponse, new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver()
});
}
public async Task<string> GetAsync(string url)
{
EnsureHttpClientCreated();
using (var response = await _httpClient.GetAsync(url))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
public async Task<string> PutAsync(string url, object input)
{
return await PutAsync(url, new StringContent(JsonConvert.SerializeObject(input), Encoding.UTF8, MediaTypeJson));
}
public async Task<string> PutAsync(string url, HttpContent content)
{
EnsureHttpClientCreated();
using (var response = await _httpClient.PutAsync(url, content))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
public async Task<string> DeleteAsync(string url)
{
EnsureHttpClientCreated();
using (var response = await _httpClient.DeleteAsync(url))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
public void Dispose()
{
_httpClientHandler?.Dispose();
_httpClient?.Dispose();
}
private void CreateHttpClient()
{
_httpClientHandler = new HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip
};
_httpClient = new HttpClient(_httpClientHandler, false)
{
Timeout = _timeout
};
_httpClient.DefaultRequestHeaders.UserAgent.ParseAdd(ClientUserAgent);
if (!string.IsNullOrWhiteSpace(_baseUrl))
{
_httpClient.BaseAddress = new Uri(_baseUrl);
}
_httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(MediaTypeJson));
}
private void EnsureHttpClientCreated()
{
if (_httpClient == null)
{
CreateHttpClient();
}
}
private static string ConvertToJsonString(object obj)
{
if (obj == null)
{
return string.Empty;
}
return JsonConvert.SerializeObject(obj, new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver()
});
}
private static string NormalizeBaseUrl(string url)
{
return url.EndsWith("/") ? url : url + "/";
}
}
Usage
using ( var client = new MyApiClient("http://localhost:8080"))
{
var response = client.GetAsync("api/users/findByUsername?username=alper").Result;
var userResponse = client.GetAsync<MyUser>("api/users/findByUsername?username=alper").Result;
}
Register this object as singleton to your dependency injection library. It's safe to reuse because it's stateless.
Do NOT recreate HTTPClient for each request.
Reuse Httpclient as much as possible
I have gone through the forum and tried those techniques but still Fiddler is not able capture my traffic. Any help would help.
Following works with Fiddler, so my WebAPI server is working. My C# client returns OK.
http://localhost:49305/api/Employee/12345
.
Host file
#localhost name resolution is handled within DNS itself.
#127.0.0.1 localhost
#::1 localhost
.
static async Task GoAny()
{
HttpClientHandler clntHand = new HttpClientHandler()
{
CookieContainer = new CookieContainer(),
Proxy = new WebProxy("http://localhost:8888", false),
UseProxy = true,
UseDefaultCredentials = false
};
HttpClient clnt = new HttpClient(clntHand)
{
BaseAddress = new Uri("http://localhost:49305")
};
clnt.DefaultRequestHeaders.Accept.Clear();
clnt.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
HttpResponseMessage resp = await clnt.GetAsync("api/Employee/12345");
if (resp.StatusCode == System.Net.HttpStatusCode.OK)
{
string c = resp.Content.ToString();
}
}
This is a known issue when using localhost.
Your url is http://localhost:49305 and you need to change it to include .fiddler after localhost: http://localhost.fiddler:49305.
Once you have done this, the request from HttpClient should then appear in Fiddler.
Please see this SO question: How can I trace the HttpClient request using fiddler or any other tool?
Here is my code
internal static void ValidateUrl(string url)
{
Uri validUri;
if(Uri.TryCreate(url,UriKind.Absolute,out validUri))
{
using (HttpClient client = new HttpClient())
{
try
{
HttpResponseMessage response = client.Get(url);
response.EnsureStatusIsSuccessful();
}
catch (Exception ex)
{
//exception handler goes here
}
}
}
}
This code when i run it produces this result.
ProxyAuthenticationRequired (407) is not one of the following:
OK (200), Created (201), Accepted (202), NonAuthoritativeInformation
(203), NoContent (204), ResetContent (205), PartialContent (206).
All i want to do is make this code validate whether a given website is up and running.
Any ideas?
This basically means exactly what it says: That you are trying to access the service via a proxy that you are not authenticated to use.
I guess that means your server was reached from the Web Service, but that it was not permitted to access the URL it tried to reach, since it tried to access it through a proxy it was not authenticated for.
It's what EnsureStatusIsSuccessful() does, it throws an exception if status code (returned from web server) is not one of that ones.
What you can do, to simply check without throwing an exception is to use IsSuccessStatusCode property. Like this:
HttpResponseMessage response = client.Get(url);
bool isValidAndAccessible = response.IsSuccessStatusCode;
Please note that it simply checks if StatusCode is within the success range.
In your case status code (407) means that you're accessing that web site through a proxy that requires authentication then request failed. You can do following:
Provide settings for Proxy (in case defaults one doesn't work) with WebProxy class.
Do not download page but just try to ping web server. You won't know if it's a web page or not but you'll be sure it's accessible and it's a valid URL. If applicable or not depends on context but it may be useful if HTTP requests fails.
Example from MSDN using WebProxy with WebRequest (base class for HttpWebRequest):
var request = WebRequest.Create("http://www.contoso.com");
request.Proxy = new WebProxy("http://proxyserver:80/",true);
var response = (HttpWebResponse)request.GetResponse();
int statusCode = (int)response.StatusCode;
bool isValidAndAccessible = statusCode >= 200 && statusCode <= 299;
You are invoking EnsureStatusIsSuccessful() which rightfully complains that the request was not successful because there's a proxy server between you and the host which requires authentication.
If you are on framework 4.5, I've included a slightly enhanced version below.
internal static async Task<bool> ValidateUrl(string url)
{
Uri validUri;
if(Uri.TryCreate(url,UriKind.Absolute,out validUri))
{
var client = new HttpClient();
var response = await client.GetAsync(validUri, HttpCompletionOption.ResponseHeadersRead);
return response.IsSuccessStatusCode;
}
return false;
}