Develop Reference

AgilityPack - Maintaining a 'connection' for several HTTP Requests

I have a system that needs to make several calls to an API as a single 'job'.
I connect to a proxy via a hostname to carry out the job so it doesn't assign me an IP address I can see. I essentially hand over the task.
At the moment I'm using HtmlWeb to make the call to my URL list like so:
var webProxy = new WebProxy(Address: [MyProxyHost]);
var web = new HtmlWeb();
web.UseCookies = false;
web.UserAgent = "[MyUserAgent]";
var htmlDoc = web.Load("[URL]","GET", webProxy, webCreds);
//...do stuff with results...
The problem: as it stands, every request I make will rotate the proxy (as every 'call' using the proxy host randomly changes the IP their side - I have no control over this).
What I want to be able to do is re-use the http request/connection over several calls (I appreciate I probably won't be able to use HtmlWeb for this).
So I would essentially say:
Create HTTP Request Object (w/ Proxy host)
Open/Initialize
Make Request 1
...
Make Request N
Close & Dispose
Is this possible?

Ok, I've got the following in place and so far it seems to be ok:
var webProxy = new WebProxy()
{
Address = new Uri([ServerHost] + ":" + [ServerPort]),
BypassProxyOnLocal = false,
UseDefaultCredentials = false,
Credentials = new NetworkCredential(
userName: [Username],
password: [Password])
};
var httpClientHandler = new HttpClientHandler()
{
Proxy = webProxy,
};
var _httpClient = new HttpClient(httpClientHandler);
...
var response = await _httpClient.GetAsync([Url]);
var pageContents = await response.Content.ReadAsStringAsync();
var htmlDoc.LoadHtml(pageContents);
...
This makes the request to each URL using the proxy and converts the page returned to a HtmlDocument so I can do the AgilityPack stuff I need to.

CookieAuthentication in .NET Core expires while using MVC App which uses that external API

I have an API in .NET Core which source code is available here. Now I'm building an MVC application which uses that API. The problem is about CookieAuthentication system in the API.
I have this code in my Configure method in Startup.cs:
app.UseCookieAuthentication(new CookieAuthenticationOptions()
{
AuthenticationScheme = "CookieAuthentication",
LoginPath = new PathString("/users/login"),
AutomaticAuthenticate = true,
AutomaticChallenge = true
});
In my login method (HttpPost) I set my cookie like this:
var claims = new List<Claim>
{
new Claim(ClaimTypes.Name, request.Email)
};
var userIdentity = new ClaimsIdentity(claims, "login");
ClaimsPrincipal principal = new ClaimsPrincipal(userIdentity);
await HttpContext.Authentication.SignInAsync("CookieAuthentication", principal);
return StatusCode(302);
Then in every method which requires authorization I check if user is logged in like this:
if (email != await GetLoggedUserEmail())
{
throw new ServiceException(ErrorCodes.UserNotFound, "You can only edit information on your account.");
}
// do some logic
...
public async Task<string> GetLoggedUserEmail()
{
if (HttpContext.User.Identity.Name == null)
{
throw new ServiceException(ErrorCodes.UserNotFound, "There is no logged in user.");
}
return await Task.FromResult(HttpContext.User.Identity.Name);
}
The problem is that when I log in by using for example Fiddler for requests and then try to delete some information by the HttpDelete request the cookie authentication works correct (I can only edit or delete informations about myself. I can't edit or delete not mine information because it will check that I'm the wrong logged user to do that. That's great.).
But when I try to do it in my MVC application strange thing happens.
When I login in the API like this:
using (var client = new HttpClient())
{
var content = new StringContent(JsonConvert.SerializeObject(values), Encoding.UTF8, "application/json");
var response = await client.PostAsync("http://www.pets.pawelkowalewicz.pl/users/login", content);
}
Then my response.StatusCode is 302 (which is correct according to the API). And then when I try to delete some information on the other page using this request:
using (var client = new HttpClient())
{
var response = await client.DeleteAsync("http://www.pets.pawelkowalewicz.pl/users/kowalewicz.pawel#gmail.com/Animals/aaa");
}
Then response.StatusCode is 400! Seems like the cookie that I'm logged in expired or something...
But when I try to execute this request right after I login like this:
var response = await client.PostAsync("http://www.pets.pawelkowalewicz.pl/users/login", content);
var response2 = await client.DeleteAsync("http://www.pets.pawelkowalewicz.pl/users/kowalewicz.pawel#gmail.com/Animals/aaa");
Response2.StatusCode is correct and the information is deleted. I really don't know why it is happening.
Seems like this is some problem that occurres while I'm redirecting throught few pages before I try to execute delete request. When I do it right after login request in the same method it works fine.
Please help me with this problem.

The problem is that HttpClientHandler instance is storing cookies inside CookieContainer and you are not sharing the container.
You may want to create shared CookieContainer instance.
private static readonly cookieContainer = new CookieContainer();
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer })
{
using (var client = new HttpClient(handler))
{
// Do your work here
}
}
Or after successful login extract cookies from handler.
using (var client = new HttpClient())
{
var content = new StringContent(JsonConvert.SerializeObject(values), Encoding.UTF8, "application/json");
var response = await client.PostAsync("http://www.pets.pawelkowalewicz.pl/users/login", content);
CookieCollection cookies = client.Handler.CookieContainer.GetCookies("http://www.pets.pawelkowalewicz.pl/");
// Store cookies collection somewhere it suits you
}
Then add to new instance of CookieContainer each time you instantiate new HttpClient.
CookieContainer cookieContainer = new CookieContainer();
CookieCollection cookies = LoadCookiesFromSomewhereYouStoredIt();
cookieContainer.Add("http://www.pets.pawelkowalewicz.pl/", cookies);
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer })
{
using (var client = new HttpClient(handler))
{
// Do your work here
}
}

Xamarin.Mac - How to pass current credentials to REST endpoint

Unable to use the System.Net API to authenticate current user to a REST endpoint. Example below returns 401 unauthorized
using (HttpClient client = new HttpClient())
{
using (HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, "http://rest/api/endpoint")
{
using (httpResponseMessage response = await client.SendAsync(request))
{
if (response.IsSuccessStatusCode)
//do something
}
}
}
If I use NSUrlConnection I am able to authenticate not problem. So, NSUrlConnection must be passing the credentials some how. Below is a snippet of that code:
var request = new NSMutableUrlRequest(new NSUrl("http://rest/api/endpoint"), NSUrlRequestCachePolicy.ReloadIgnoringCacheData, 0);
request["Accept"] = "application/json";
NSUrlConnection.SendAsynchronousRequest(request, NSOperationQueue.MainQueue, delegate(NSUrlResponse, response, NSData data, NSError error)
{
// successfully authenticated and do something with response
});
I would like to wrap my service code in a PCL to share with other platforms. Therefore, I would like to get this all working within the System.Net api. Is this possible?
UPDATE:
I've tried using an HttpClientHandler and using default credentials as well as CredentialCache.DefaultNetworkCredentials. The only way to get this to work is to hardcode the credentials, which I do not want. It appears the System.Net stack does not surface the credentials from the OS.

I assume it's using default authentication. You could do that with HttpClient with an HttpClientHandler, for example:
using (var handler = new HttpClientHandler {
UseDefaultCredentials = true
})
using (var client = new HttpClient(handler))
{
var result = await client.SendAsync(...);
}

HttpClient access url with '#' (at symbol)

I'm integrating a service that returns a key when I a GET request to a URL that is in the following format:
https://username:password#service.com/refresh.key
When I access the URL in my browser, it returns the new key as expected, by when I do a GET request using HttpClient I get a 401.
HttpClient _client = new HttpClient();
var response = await _client.GetAsync(#"https://username:password#service.com/refresh.key"); // Returns a 401
I think it has something to do with the '#' in the URL, but I'm not sure how to fix it, I tried replacing it with '%40', but when I do that I get a UriFormatException.
Does anyone know how to do this?

You should modify Authorization header of HttpClient, can you try the code below;
HttpClient _client = new HttpClient();
byte[] usernamePasswordBytes = Encoding.ASCII.GetBytes("user:pass");
_client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", Convert.ToBase64String(usernamePasswordBytes));
var response = await _client.GetAsync(#"https://service.com/refresh.key");
PS: Such username:pass#domain.com requests are BasicAuthentication request so in fact you try to make basic authentication request.
Hope this works for you

You don't need to provide credentials in url. Instead you can do:
using (var handler = new HttpClientHandler {Credentials = new NetworkCredential("username", "password")}) {
using (HttpClient _client = new HttpClient(handler)) {
var response = await _client.GetAsync(#"https://service.com/refresh.key");
}
}

HttpClient does not work with Fiddler

I have gone through the forum and tried those techniques but still Fiddler is not able capture my traffic. Any help would help.
Following works with Fiddler, so my WebAPI server is working. My C# client returns OK.
http://localhost:49305/api/Employee/12345
.
Host file
#localhost name resolution is handled within DNS itself.
#127.0.0.1 localhost
#::1 localhost
.
static async Task GoAny()
{
HttpClientHandler clntHand = new HttpClientHandler()
{
CookieContainer = new CookieContainer(),
Proxy = new WebProxy("http://localhost:8888", false),
UseProxy = true,
UseDefaultCredentials = false
};
HttpClient clnt = new HttpClient(clntHand)
{
BaseAddress = new Uri("http://localhost:49305")
};
clnt.DefaultRequestHeaders.Accept.Clear();
clnt.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
HttpResponseMessage resp = await clnt.GetAsync("api/Employee/12345");
if (resp.StatusCode == System.Net.HttpStatusCode.OK)
{
string c = resp.Content.ToString();
}
}

This is a known issue when using localhost.
Your url is http://localhost:49305 and you need to change it to include .fiddler after localhost: http://localhost.fiddler:49305.
Once you have done this, the request from HttpClient should then appear in Fiddler.
Please see this SO question: How can I trace the HttpClient request using fiddler or any other tool?