I'm making a POST request to a web site to send username/login information in order to get a cooking containing an authentication token. I build, test, and debug my process running in a console application, then I set it to run as part of a Windows Service.
When I am running in the console app, there are 2 cookies being returned from the post request: the JSESSIONID and the AuthToken. After I deploy and run it in the Windows Service, I only see the JSESSIONID.
I set up Fiddler to watch the windows service, and I can see the AuthToken being passed back in the response to my post request, but I am unable to get it from the cookie container.
Additionally, if I install the process on my local machine, and run through my Fiddler proxy, I am able to get the AuthToken, but if I disable the proxy, no AuthToken is contained in the cookies.GetCookies result.
I'm at a loss as to why it will operate fine when running as a console app on my local machine, but the AuthToken fails to return when run as a service on a remote machine.
My local machine is running .net 4.7, and the server where the service is installed is running 4.5.2.
Here's the code I'm using:
public string SubmitPost(Uri uri, string action, string contentPost, bool putRequest)
{
CookieContainer cookies = new CookieContainer();
WebRequestHandler handler = new WebRequestHandler();
handler.Proxy = new WebProxy("http://<FIDDLER PROXY>:8888", false, new string[] {});
X509Certificate cert = X509Certificate2.CreateFromSignedFile(m_CertPath);
handler.ClientCertificates.Add(cert);
handler.CookieContainer = cookies;
string resultContent = "";
using (var client = new HttpClient(handler))
{
AddClientHeadersForPost(uri, client);
cookies.Add(uri, m_jar);
var content = new StringContent(contentPost, Encoding.UTF8);
content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
content.Headers.ContentLength = contentPost.Length;
HttpResponseMessage result = client.PostAsync(action, content).Result;
resultContent = result.Content.ReadAsStringAsync().Result;
IEnumerable<Cookie> responseCookies = cookies.GetCookies(uri).Cast<Cookie>();
Logger.InfoFormat("{0} cookies", responseCookies.Count());
foreach (Cookie item in responseCookies)
{
Logger.InfoFormat("Cookie: {0}", item.Name);
if (item.Name.Contains("auth"))
{
Logger.InfoFormat("Auth Token: {0}", item.Value);
}
m_jar.Add(item);
}
}
return (resultContent);
}
protected virtual void AddClientHeadersForPost(Uri uri, HttpClient client)
{
client.BaseAddress = uri;
client.DefaultRequestHeaders.TryAddWithoutValidation("Host", "<HOST URL>");
client.DefaultRequestHeaders.TryAddWithoutValidation("Origin", "<ORIGIN URL>");
client.DefaultRequestHeaders.TryAddWithoutValidation("X-Requested-With", "XMLHttpRequest");
client.DefaultRequestHeaders.TryAddWithoutValidation("Accept", "*/*");
client.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate, br");
client.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Language", "en-US,en;q=0.8");
client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
client.DefaultRequestHeaders.TryAddWithoutValidation("DNT", "1");
client.DefaultRequestHeaders.TryAddWithoutValidation("Referer", "<REFERER URL>");
client.DefaultRequestHeaders.TryAddWithoutValidation("X-CSRF-TOKEN", m_csrf);
client.DefaultRequestHeaders.TryAddWithoutValidation("X-2-CSRF-TOKEN", m_sppCSRF);
client.DefaultRequestHeaders.TryAddWithoutValidation("Connection", "keep-alive");
client.DefaultRequestHeaders.ExpectContinue = false;
}
Related
I have this simple code to instantiate an HttpClient object, and send a few webrequests, but am running into a few problems that I will explain shortly:
var client = WebHelper.CreateGzipHttpClient(new WebProxy("127.0.0.1", 8888));
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36");
client.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3");
client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.9");
client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, br");
client.DefaultRequestHeaders.Add("Sec-Fetch-Mode", "navigate");
client.DefaultRequestHeaders.Add("Sec-Fetch-Site", "none");
client.DefaultRequestHeaders.Add("Sec-Fetch-User", "?1");
client.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1");
await client.GetAsync("https://www.example.com");
await client.GetAsync("https://www.bestbuy.com");
await client.GetAsync("https://www.costco.com");
If I remove the request to example.com, the subsequent requests fail (504 Gateway Timeout on bestbuy.com). Doesn't make any sense to me, so was wondering if someone on SO could enlighten me as to why that is.
Furthermore, if I remove the WebProxy from the HttpClient, only the request to example.com will succeed, and the other 2 will fail.
What is going on and how can I fix it?
public static HttpClient CreateGzipHttpClient(WebProxy proxy = null)
{
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
Proxy = proxy
};
return new HttpClient(handler);
}
Fixed by removing the Fiddler related SSL certificates within Internet Explorer's Internet Options. These weren't being removed even after an uninstall.
I would like to post a http request in order to login to a website using HttpClient and FormUrlEncodedContent. I can track how the response is supposed to look on my chrome browser, but when i recreate the post request made by my browser, I don't get the anticipated response. The website I'm trying to log into is https://www.lectio.dk/lectio/31/login.aspx.
I guess I'm in doubt as to what I should supply the FormUrlEncodedContent with (using the network tab in chrome I can see request headers and form data, but how do I select what I should supply?). Currently my code looks like this.
CookieContainer container = new CookieContainer();
HttpClientHandler handler = new HttpClientHandler();
handler.CookieContainer = container;
HttpClient client = new HttpClient(handler);
Console.WriteLine(client.DefaultRequestHeaders + "\n" + "--------------------------");
Dictionary<string, string> vals = new Dictionary<string, string>
{
{"user-agent","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"},
{"accept-language","en-GB,en-AS;q=0.9,en-DK;q=0.8,en;q=0.7,da-DK;q=0.6,da;q=0.5,en-US;q=0.4"},
{"accept-encoding","gzip, deflate, br"},
{"accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"},
{"m$Content$username2","username"},
{"m$Content$passwordHidden","password"},
{"__EVENTTARGET","m$Content$submitbtn2"},
};
FormUrlEncodedContent content = new FormUrlEncodedContent(vals);
var response = client.PostAsync("https://www.lectio.dk/lectio/31/login.aspx", content);
var responseString = response.Result;
Console.WriteLine(responseString);
handler.Dispose();
client.Dispose();
The idea is to be logged into the website(I guess my Cookiecontainer will take care of that?) so that I can scrape som data.
There's an ASP.NET website from a third party that requires one to log on. I need to get some data from the website and parse it, so I figured I'd use HttpClient to post the necessary credentials to the website, same as the browser would do it. Then, after that POST request, I figured I'd be able to use the cookie values I received to make further request to the (authorization-only) urls.
I'm down to the point where I can successfully POST the credentials to the login url and receive three cookies: ASP.NET_SessionId, .ASPXAUTH, and a custom value used by the website itself, each with their own values. I figured that since the HttpClient I set up is using an HttpHandler that is using a CookieContainer, the cookies would be sent along with each further request, and I'd remain logged in.
However, this does not appear to be working. If I use the same HttpClient instance to then request one of the secured areas of the website, I'm just getting the login form again.
The code:
const string loginUri = "https://some.website/login";
var cookieContainer = new CookieContainer();
var clientHandler = new HttpClientHandler() { CookieContainer = cookieContainer, AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate };
var client = new HttpClient(clientHandler);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
var loginRequest = new HttpRequestMessage(HttpMethod.Post, loginUri);
// These form values correspond with the values posted by the browser
var formContent = new FormUrlEncodedContent(new[]
{
new KeyValuePair<string, string>("customercode", "password"),
new KeyValuePair<string, string>("customerid", "username"),
new KeyValuePair<string, string>("HandleForm", "Login")
});
loginRequest.Content = formContent;
loginRequest.Headers.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393");
loginRequest.Headers.Referrer = new Uri("https://some.website/Login?ReturnUrl=%2f");
loginRequest.Headers.Host = "some.website";
loginRequest.Headers.Connection.Add("Keep-Alive");
loginRequest.Headers.CacheControl = new System.Net.Http.Headers.CacheControlHeaderValue() { NoCache = true };
loginRequest.Headers.AcceptLanguage.ParseAdd("nl-NL");
loginRequest.Headers.AcceptEncoding.ParseAdd("gzip, deflate");
loginRequest.Headers.Accept.ParseAdd("text/html, application/xhtml+xml, image/jxr, */*");
var response = await client.SendAsync(loginRequest);
var responseString = await response.Content.ReadAsStringAsync();
var cookies = cookieContainer.GetCookies(new Uri(loginUri));
When using the proper credentials, cookies contains three items, including a .ASPXAUTH cookie and a session id, which suggests that the login succeeded. However:
var text = await client.GetStringAsync("https://some.website/secureaction");
...this just returns the login form again, and not the content I get when I log in using the browser and navigate to /secureaction.
What am I missing?
EDIT: here's the complete request my application is making and the request chrome is making. They are identical, save for the cookie values. I ran them through windiff: the lines marked <! are the lines sent by my application, the ones marked !> are sent by Chrome.
GET https://some.website/secureaction
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36
Accept-Encoding: gzip, deflate, sdch, br
Upgrade-Insecure-Requests: 1
Host: some.website
Accept-Language:nl-NL,
>> nl;q=0.8,en-US;q=0.6,en;q=0.4
Accept: text/html,
>> application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Cookie:
<! customCookie=7CF190C0;
<! .ASPXAUTH=37D61E47(shortened for readability);
<! ASP.NET_SessionId=oqwmfwahpvf0qzpiextx0wtb
!> ASP.NET_SessionId=kn4t4rmeu2lfrgozjjga0z2j;
!> customCookie=8D43E263;
!> .ASPXAUTH=C2477BA1(shortened for readability)
The HttpClient application get a 302 referral to /login, Chrome gets a 200 response containing the requested page.
As requested, here's how I eventually made it work. I had to do a simple GET request to /login first, and then do a POST with the login credentials. I don't recall what value exactly is being set by that GET (I assume a cookie with some encoded value the server wants), but the HttpClient takes care of the cookies anyway, so it just works. Here's the final, working code:
const string loginUri = "https://some.website/login";
var cookieContainer = new CookieContainer();
var clientHandler = new HttpClientHandler()
{
CookieContainer = cookieContainer,
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
var client = new HttpClient(clientHandler);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
// First do a GET to the login page, allowing the server to set certain
// required cookie values.
var initialGetRequest = new HttpRequestMessage(HttpMethod.GET, loginUri);
await client.SendAsync(initialGetRequest);
var loginRequest = new HttpRequestMessage(HttpMethod.Post, loginUri);
// These form values correspond with the values posted by the browser
var formContent = new FormUrlEncodedContent(new[]
{
new KeyValuePair<string, string>("customercode", "password"),
new KeyValuePair<string, string>("customerid", "username"),
new KeyValuePair<string, string>("HandleForm", "Login")
});
loginRequest.Content = formContent;
loginRequest.Headers.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393");
loginRequest.Headers.Referrer = new Uri("https://some.website/Login?ReturnUrl=%2f");
loginRequest.Headers.Host = "some.website";
loginRequest.Headers.Connection.Add("Keep-Alive");
loginRequest.Headers.CacheControl = new System.Net.Http.Headers.CacheControlHeaderValue() { NoCache = true };
loginRequest.Headers.AcceptLanguage.ParseAdd("nl-NL");
loginRequest.Headers.AcceptEncoding.ParseAdd("gzip, deflate");
loginRequest.Headers.Accept.ParseAdd("text/html, application/xhtml+xml, image/jxr, */*");
var response = await client.SendAsync(loginRequest);
var responseString = await response.Content.ReadAsStringAsync();
var cookies = cookieContainer.GetCookies(new Uri(loginUri));
Dearl All
I am new to WebAPI2. I want to consume data from a function in JSON format but throwing error StatusCode: 403, ReasonPhrase: 'ModSecurity Action'.
I can consume directly from browser but can not from HttpClient. No security implemented.
Working perfect on local server but above error throws on remote shared server.
APIURL. http://api.owncircles.com/api/Circles/Education/Questions/getAns/2012460157
API Function Code.
[HttpGet()]
[AllowAnonymous]
[Route("~/api/Circles/Education/Questions/getAns/{quesID}")]
public IHttpActionResult getAns(string quesID)
{
IQuestions objQuestion = Questions.getInatance();
var qtn = objQuestion.getAns(quesID);
return Json(qtn);
}
Client Side
[AllowAnonymous]
public async Task<ActionResult> checkAns(string id)
{
string url = common.apiURL + "Circles/Education/Questions/getAns/"+id;
//HttpClient client = new HttpClient(new HttpClientHandler() {UseDefaultCredentials = true });
HttpClient client = new HttpClient();
// client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(Constants.));
// client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("");
// client.DefaultRequestHeaders.Authorization = null;
client.BaseAddress = new Uri(url);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
ent_QuestionsDetails Questions = null;
var response = await client.GetAsync(url);
if (response.IsSuccessStatusCode)
{
Questions = response.Content.ReadAsAsync<ent_QuestionsDetails>().Result;
}
OC.Models.mod_Questions objModel = new OC.Models.mod_Questions();
objModel.questionID = Questions.questionID;
objModel.questions = Questions.questions;
objModel.questionOptions = Questions.questionOptions;
return View(objModel);
}
It seems its your user agent, when this is omitted the request is rejected. If you mimic the chrome user agent value in the header the request will succeed. Here is a self contained working example:
Note that this example does not use await/async as it was tested in a console app
using (HttpClient client = new HttpClient())
{
client.BaseAddress = new Uri("http://api.owncircles.com/");
client.DefaultRequestHeaders.Clear();
client.DefaultRequestHeaders.Accept.ParseAdd("application/json");
client.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36");
var result = client.GetAsync("api/Circles/Education/Questions/getAns/2012460157").Result;
if(result.IsSuccessStatusCode)
Console.Write(result.Content.ReadAsStringAsync().Result);
else
Console.Write("fail");
}
That being said I do not know what kind of check you have API server side on the user agent that it would reject a request.
Somehow I Iam not able to download the html content of https://{YourName}.visualstudio.com/Defaultcollection/ via HttpWebRequest/WebRequest or WebClient.
It always results a HTML-Page with following error Message:
Microsoft Internet Explorer's Enhanced Security Configuration is currently enabled on your environment. This enhanced level of security prevents our web integration experiences from displaying or performing correctly. To continue with your operation please disable this configuration or contact your administrator.
I have tried alot of ways to get to my needed result. I tried using OAuth2 and also setup Alternate authentication credentials. I even disabled Microsoft Internet Explorer's Enhanced Security.
Here are 2 of my x methods which doesnt seem to work. Both give the same result (see error msg above):
private static void Test()
{
WebClient client = new WebClient();
client.UseDefaultCredentials = true;
client.Credentials = new NetworkCredential(UserName,Password);
//Pretend to be a browser
client.Headers.Add("user-agent", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 4.0.20506)");
var HTML = client.DownloadString("https://<YourName>.visualstudio.com/Defaultcollection/");
Console.WriteLine(HTML);
}
private static void Test2()
{
CookieContainer cookies = new CookieContainer();
HttpWebRequest authRequest = (HttpWebRequest)HttpWebRequest.Create("https://<YourName>.visualstudio.com/Defaultcollection/");
//Set Header
authRequest.UserAgent = "Mozilla/5.0 (Windows NT 5.1; rv:2.0b8) Gecko/20100101 Firefox/4.0b8";
authRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
authRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
authRequest.Headers.Add("Accept-Language", "de,en;q=0.5");
authRequest.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
//authRequest.Headers.Add("Keep-Alive", "30000");
authRequest.Headers.Add(HttpRequestHeader.Authorization, SetAuthHeaderValue());
//Something
authRequest.ContentLength = 0;
authRequest.ContentType = "application/soap+xml; charset=utf-8";
authRequest.Host = "<YourName>.visualstudio.com";
//Set Cookies
authRequest.CookieContainer = cookies;
HttpWebResponse response = (HttpWebResponse)authRequest.GetResponse();
StreamReader readStream = new StreamReader(response.GetResponseStream());
string HTML = readStream.ReadToEnd();
Console.WriteLine(HTML);
readStream.Close();
}
private static string SetAuthHeaderValue()
{
//string _auth = string.Format("{0}:{1}",UserName,Password);
//string _enc = Convert.ToBase64String(Encoding.ASCII.GetBytes(_auth));
String encoded = System.Convert.ToBase64String(System.Text.Encoding.GetEncoding("ISO-8859-1").GetBytes(UserName + ":" + Password));
string _cred = string.Format("{1}", "Basic", encoded);
return _cred;
}
I picked the Header-Values you see here, by tracing the connection with fiddler.
Is somebody able to authenticated,connect and download the html-content from https://{YourName}.visualstudio.com/Defaultcollection/?
Would be awesome, thanks :)!