Get content of response with StatusCode 401 - c#

I'm trying to establish a connection to a server that sends 401 Authentication Error for all my requests along with the normal html response. e.g.
However, I also want to read the HTML response that is sent alongwith so that I can parse that. An example header exchange captured using LiveHTTPHeaders:
Clearly, content-length is non-zero. Firefox shows it to be javascript.
https://172.31.1.251:1003/fgtauth?73e285357b2dc5cc
Request:
GET /fgtauth?73e285357b2dc5cc HTTP/1.1
Host: 172.31.1.251:1003
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Response:
HTTP/1.x 401 Unauthorized
WWW-Authenticate: Fortigate
Content-Length: 1091
Connection: Keep-Alive
Cache-Control: no-cache
Content-Type: text/html
At this point, a form opens in firefox that asks me to enter my username and password.
https://172.31.1.251:1003/
Request:
POST / HTTP/1.1
Host: 172.31.1.251:1003
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: https://172.31.1.251:1003/
Content-Type: application/x-www-form-urlencoded
Content-Length: 93
magic=73e285357b2dc5cc&username=uuxx&password=xxuu&4Tredir=http%3A%2F%2Fwww.google.com%2F
Response:
HTTP/1.x 401 Unauthorized
WWW-Authenticate: Fortigate
Content-Length: 924
Connection: Keep-Alive
Cache-Control: no-cache
Content-Type: text/html
At this point I am redirected to another URL. However, the problem that I have is how to get the content of lenght 924 that is sent along with the 401 Unauthorized, because that content will help me in doing what I want to do further. But the very line:
WebResponse loginResponse = loginRequest.GetResponse();
throws an exception.
I will be grateful for any suggestions to help me get to the actual content.
Thanks.

You need to read the WebException's Response property, like this:
WebResponse loginResponse;
try {
loginResponse = loginRequest.GetResponse();
} catch(WebException ex) {
if (ex.Status == WebExceptionStatus.ProtocolError) {
loginResponse = ex.Response;
} else
throw;
}
//Do something with the response, and remember to dispose it.

Related

Unable to get request data in asp.net web forms

in asp.net pageload method I am trying to get request data using
Request.Form["keyname"]
Request.Query
but all are empty but when I run fiddler in inspector>>raw I am getting this
POST http://localhost:50844/Success.aspx HTTP/1.1
Host: localhost:50844
Connection: keep-alive
Content-Length: 15583
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
Origin: null
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: ASP.NET_SessionId=ax0e4j1ucaghsarowzmvgoep
valuestr=abc&randomnum=99389
I need to get values of valuestr and randomnum,how can I do this
You can use the below code as per the values in your URL.
if your URl is = http://localhost:3039/test.aspx?valuestr=abc&randomnum=99389
then you can use the below code
string valuestr = Request.QueryString["valuestr"];
string randomnum= Request.QueryString["valuestr"];
Response.Write(valuestr );
Response.Write(randomnum);
do let me know in case you require any help

How can I fake a cookie in a web request that's generated dynamically?

I need to get some public data from a link, but this is a second time I encounter a Cookie that I cannot replicate as some values in it are dynamic. How can I get this cookie before doing an actual request to get my data?
Example link:
https://www.blackrock.com/ca/individual/en/products/239832/ishares-sptsx-60-index-etf/1464253357804.ajax?tab=chart
Example headers:
Host: www.blackrock.com
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Accept: */*
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: https://www.blackrock.com/ca/individual/en/products/239832/ishares-sptsx-60-index-etf
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
Cookie: JSESSION_amer-retail01=EA5A098E120F4E3FAD2C994A64F292E4.03; ts-ca-one-locale=en_CA; FIRSTPAGE_ca_ca-one=page-class-home; blkUserType-ca-one=individual; ipe_s=0d9c807a-04cc-ae5b-e459-d89bddd2faab; ca-one-recent-funds=239832; __utmt=1; __utma=17023466.874636303.1499433731.1499433731.1499433731.1; __utmb=17023466.5.10.1499433731; __utmc=17023466; __utmz=17023466.1499433731.1.1.utmcsr=google|utmgclid=CJjax_ah99QCFZxMDQodKQ4NxQ|utmgclsrc=aw.ds|utmccn=(not%20set)|utmcmd=(not%20set)|utmctr=(not%20provided); _bizo_bzid=0688d0c2-d203-488b-b128-94c0406b5568; _bizo_cksm=B16C1965883E71EB; _bizo_np_stats=14%3D243%2C; ipe.20668.pageViewedCount=3; utag_main=v_id:015d1d3702f6001b054da00951140407300ad06b00bd0$_sn:1$_ss:0$_pn:6%3Bexp-session$_st:1499436924463$ses_id:1499433730806%3Bexp-session; s_pers=%20s_fid%3D5295C92DBD9DC781-0177F2F787EC9066%7C1562507124793%3B%20gpv%3Dca-one%257Cca%257Cfund%257Cishares%2520s%2526p%252Ftsx%252060%2520index%2520etf%257Cxiu%7C1499436924795%3B%20s_nr%3D1499435124796-New%7C1530971124796%3B%20s_pers_eVar17%3Dprospect%7C1502027124797%3B%20s_pers_prop19%3Danonymous%7C1502027124799%3B; s_sess=%20s_campaign%3Dppc%253Ablackrock_ca%253Agoogle%253Ablackrock%3B%20s_cc%3Dtrue%3B%20s_sq%3D%3B; s_vi=[CS]v1|2CAFC5818507C33B-6000011540002202[CE]
You can instantiate a CookieContainer and set it to the CookieContainer property of your HttpWebRequest. Then make an initial request and after you GetResponse the cookie container will be populated with any cookies returned.
Then reuse the CookieContainer in subsequent requests and all the previous returned cookies will be sent.

Simulate Upload in Chrome Postman WebApi

I am trying to simulate a FileUpload with Chrome Postman to my ASP.NET WebApi method . To do this I perform the following steps in Postman.
1.Open postman and choose POST method from address bar.
2.Enter valid URL.
3.Go to Body section below address bar.
4.From Body section choose Binary option.
5.Hit the "Chose Files" button to select a file to upload.
6.Choose a file
7.Ensure the "Post" option is selected
8.Hit "Send"
I then test for MultiPartContent, however when I step through my method the Request Object doesn't appear to have the info that I need . The exception below will get hit.
public List<Premium.Model.BureauModUpdate> Post() {
HttpRequestMessage request = this.Request;
if (!request.Content.IsMimeMultipartContent()) {
throw new HttpResponseException(HttpStatusCode.UnsupportedMediaType);
}
}
It doesn't appear as if any info concerning the file upload is in the request. A review of the request content yields this:
{Method: POST, RequestUri: 'http://localhost/WCAPI/Lookup/BureauModUpdate/CreateNyModUpdates', Version: 1.1, Content: System.Web.Http.WebHost.HttpControllerHandler+LazyStreamContent, Headers:
{
Cache-Control: no-cache
Connection: keep-alive
Accept: */*
Accept-Encoding: gzip
Accept-Encoding: deflate
Accept-Encoding: br
Accept-Language: en-US
Accept-Language: en; q=0.8
Host: localhost
User-Agent: Mozilla/5.0
User-Agent: (Windows NT 6.1; WOW64)
User-Agent: AppleWebKit/537.36
User-Agent: (KHTML, like Gecko)
User-Agent: Chrome/55.0.2883.87
User-Agent: Safari/537.36
Postman-Token: 23f64855-3b60-3f87-3aa1-ba7d0a841273
Origin: chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop
Content-Length: 200284
Content-Type: application/x-www-form-urlencoded
}}
Is there something else I need to be doing ? If anybody has any suggestions or can point me in the right direction on this it would be greatly appreciated.
I believe your content type is incorrect. It should be multipart/form-data when uploading files.
application/x-www-form-urlencoded is used only when posting a form with no binary data.

Send POST request through WebClient with boundary

I have a problem with sending a POST request through WebClient in C# API.
The POST request is:
POST http://aaa.com/login.php HTTP/1.1
Host: www.aaa.com
Connection: keep-alive
Content-Length: 346
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Origin: http://www.aaa.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.152 Safari/537.22
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryK0J3zdRYpjgldAFy
Referer: http://www.turbotrafficbooster.com/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: bg-BG,bg;q=0.8
Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.3
Cookie: PHPSESSID=b703525853495c9257b3f3ec579c937a
------WebKitFormBoundaryK0J3zdRYpjgldAFy
Content-Disposition: form-data; name="navaction"
login
------WebKitFormBoundaryK0J3zdRYpjgldAFy
Content-Disposition: form-data; name="UserID"
test
------WebKitFormBoundaryK0J3zdRYpjgldAFy
Content-Disposition: form-data; name="Password"
test
------WebKitFormBoundaryK0J3zdRYpjgldAFy--
I know what to do with the headers like user-agent and etc., but I have a problem with the boundaries. I don't know how to put them and where to do it..
See this answer and the examples in the linked specification here. Example:
Content-Type: multipart/form-data; boundary=AaB03x
--AaB03x
Content-Disposition: form-data; name="submit-name"
Larry
--AaB03x
WebKitFormBoundaryK0J3zdRYpjgldAFy is just a convention. It could be MyFoobaredBoundry if you want, provided that that you declare and close it off appropriately. Different browsers use different strings, so nothing on the receiving end should differentiate because of it.
However, this is probably not something you have to do yourself...

C# - Connection: keep-alive Header is Not Being Sent During HttpWebRequest

I'm trying to send to send the following header with my HttpWebRequest:
Connection: keep-alive
However, the header is never sent. Fiddler2 is showing that whenever I request the page in Google Chrome, the header is sent. However, my application refuses to send this header for some reason.
I have set the KeepAlive property to true (it's true by default anyway), yet the header still does not get sent.
I am trying to send this header with multiple HttpWebRequests, but they all basically look like this:
HttpWebRequest logIn6 = (HttpWebRequest)WebRequest.Create(new Uri(responseFromLogIn5));
logIn6.CookieContainer = cookies;
logIn6.KeepAlive = true;
logIn6.Referer = "https://login.yahoo.com/config/login?.src=spt&.intl=us&.lang=en-US&.done=http://football.fantasysports.yahoo.com/";
logIn6.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.220 Safari/535.1";
logIn6.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
logIn6.Headers.Add("Accept-Encoding:gzip,deflate,sdch");
logIn6.Headers.Add("Accept-Language:en-US,en;q=0.8");
logIn6.Headers.Add("Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3");
logIn6.AllowAutoRedirect = false;
HttpWebResponse logIn6Response = (HttpWebResponse)logIn6.GetResponse();
string responseFromLogIn6 = logIn6Response.GetResponseHeader("Location");
cookies.Add(logIn6Response.Cookies);
logIn6Response.Close();
Does anyone know what I have to do to make sure this header is sent?
Fiddler2 Raw From Chrome:
GET xxx HTTP/1.1
Host: accounts.google.com
Connection: keep-alive
Referer: https://login.yahoo.com/config/login?.src=spt&.intl=us&.lang=en-US&.done=http://football.fantasysports.yahoo.com/
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.220 Safari/535.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: xxx
HTTP/1.1 302 Moved Temporarily
Set-Cookie: xxx
Set-Cookie: xxx
Location: xxx
Content-Type: text/html; charset=UTF-8
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Date: Sat, 17 Sep 2011 22:27:09 GMT
Expires: Sat, 17 Sep 2011 22:27:09 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Content-Length: 2176
Server: GSE
Fiddler2 Raw From My Application:
GET xxx HTTP/1.1
Referer: https://login.yahoo.com/config/login?.src=spt&.intl=us&.lang=en-US&.done=http://football.fantasysports.yahoo.com/
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.220 Safari/535.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Host: accounts.google.com
HTTP/1.1 302 Moved Temporarily
Location: xxx
Content-Type: text/html; charset=UTF-8
Date: Sun, 18 Sep 2011 00:05:40 GMT
Expires: Sun, 18 Sep 2011 00:05:40 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Content-Length: 573
Server: GSE
I'm trying to get the second Fiddler2 raw information to look like the first Fiddler2 raw information.
I've had the same issue: The Connection: Keep-Alive header is not sent except the first request, and the server I accessed won't give me the correct response if it is missing. So, here are my workarounds to this issue:
First is set the ProtocolVersion property of HttpWebRequest instance to HttpVersion.Version10. Except the http command will become GET xxx HTTP/1.0, it works and uses only the public API.
The second way uses the reflection to modify the internal property ServicePoint.HttpBehaviour of HttpWebRequest instance, like this:
var req = (HttpWebRequest)WebRequest.Create(someUrl);
var sp = req.ServicePoint;
var prop = sp.GetType().GetProperty("HttpBehaviour",
BindingFlags.Instance | BindingFlags.NonPublic);
prop.SetValue(sp, (byte)0, null);
req.GetResponse().Close();
Hope this helps.
I struggled with this problem for half a day! And dear old Fiddler (my guardian angel) was inadvertantly part of the problem:
Whenever I tested my HTTP POSTs using Fiddler monitoring ON - the problem DIDN'T appear
Whenever I tested my HTTP POSTs with Fiddler monitoring OFF - the problem DID appear
My POSTS were sent with protocol 1.1 and the Keep-Alive was ignored/redundant/why after the initial connection. i.e. I could see it in the header of the first POST (via Fiddler!), but not in subsequent POSTs despite using the same code. Hey ho ...
But the remote server would only respond if Keep-Alive was sent. Now I can't prove this, but I suspect that Fiddler monitoring the connection caused the remote server to think or believe that the connection was still active (despite no Keep-Alives sent after my first POST) and responded correctly. As I said, the second I turned Fiddler off, the absence of Keep-Alives caused the remote server to timeout on me..
I implemented the 1.0 solution described above and my POSTS now work, with or without Fiddler on or off. Hope this helps somebody else stuck somewhere ...
You doing it right. The code should result in following header added:
Connection: Keep-Alive
Post the code that you use for sending request and Raw output from Fiddler if you don't see this header. You may also ignore this because HTTP 1.1 connection is keep-alive by default.
Update: it looks like .NET only sets Keep-Alive explicitly for the first (!) request. Further requests to the same host/url will not have this header presumably because underlying tcp connection is already being reused.
After downloading HttpWebRequest source code, noticed that every property checks some known headers for HeaderCollection. To get rid of that doing some reflection stuff on that collection make it work
var webRequest = (HttpWebRequest) WebRequest.Create(url);
webRequest.Headers.GetType().InvokeMember("ChangeInternal",
BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.InvokeMethod,
Type.DefaultBinder, webRequest.Headers, new object[] {name, value}
);
I know the answer for this as I had the same problem and managed to solve it by inheriting the webclient and overriding it's Get Web Request method.
See the code below:
public class CookieAwareWebClient : WebClient
{
public CookieContainer CookieContainer { get; set; }
public CookieAwareWebClient()
: this(new CookieContainer())
{ }
public CookieAwareWebClient(CookieContainer c)
{
this.CookieContainer = c;
}
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
var castRequest = request as HttpWebRequest;
if (castRequest != null)
{
castRequest.KeepAlive = true; //<-- this what you want! The rest you don't need.
castRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
castRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36";
castRequest.Referer = "https://www.jobserve.com/gb/en/Candidate/Login.aspx?url=48BB4C724EA6A1F2CADF4243A0D73C13225717A29AE8DAD6913D";
castRequest.Headers.Add("Accept-Encoding", "gzip,deflate,sdch");
castRequest.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");
castRequest.CookieContainer = this.CookieContainer;
}
return request;
}
}
As you can see I am not only enabling keep-alive but I am utilizing cookies and other headers also!
I hope that helps!
Kiran

Categories

Resources