I've recently run into an issue which I believe is related with the CookieContainer.
It is a basic HttpWebRequest to a url that was previously working fine (2 days ago) but is now having issues.
Below is the code I am using.
string url = "http://www.footywire.com/afl/footy/ft_match_statistics?mid=5827";
var cookieJar = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.CookieContainer = cookieJar;
request.AllowAutoRedirect = false;
request.Method = "GET";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
WebHeaderCollection myWebHeaderCollection = request.Headers;
myWebHeaderCollection.Add("Accept-Encoding", "gzip, deflate");
myWebHeaderCollection.Add("Accept-Language", "en-US,en;q=0.5");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
int cookieCount = cookieJar.Count;
response.Close();
I am getting a (500) Internal Server Error during the GetResponse() call.
The URL works fine in the browser, but fails if I use an incognito/private window which makes me think it has to be cookie related.
I've monitored the request/responses using fiddler and can see the request is not handling cookies at all.
Here is a working request through the browser with cookies enabled.
GET http://www.footywire.com/afl/footy/ft_match_statistics?mid=5827 HTTP/1.1
Host: www.footywire.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: __utma=123692316.616048912.1368166198.1400452319.1400457786.25; __gads=ID=c95b71c78d8a7516:T=1368166203:S=ALNI_MbHrLnKztzLrfUEkZNoQfcimetGNw; _em_vt=3938ffa289939f5bd80fd60798185181c3303dddd6-9262208553794a3e; __utmz=123692316.1395356585.4.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmc=123692316; JSESSIONID=62B598BD99DC2D228C4FC1E1549ED6FB; __utmb=123692316.1.10.1400457786; _em_v=5cc514d9ac92155cd445ac583ffd53794a3e7088d0-4657961653794a3e
Connection: keep-alive
This is the request the code is sending. The cookie is clearly missing.
GET http://www.footywire.com/afl/footy/ft_match_statistics?mid=5827 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.5
Host: www.footywire.com
Connection: Keep-Alive
I'm at a bit of a loss as to why its not working now, but it was 2 days ago. I'm also not sure of what to do next.
Any help would be appreciated. Thanks.
Related
I try to get a specific value from a specific site...
the site periodically updating the value using an Ajax call to
https://www.plus500.co.il/api/LiveData/FeedUpdate?instrumentId=19
(you can Navigate to the address and see you get the XML response.)
using Postman:
sending
GET /api/LiveData/FeedUpdate?instrumentId=19 HTTP/1.1
Host: www.plus500.co.il
Cache-Control: no-cache
Postman-Token: f823c87d-3edc-68ce-e1e7-02a8fc68be7a
I get a valid Json Response...
Though, when i try it from C#:
var webRequest = WebRequest.CreateHttp(#"https://www.plus500.co.il/api/LiveData/FeedUpdate?instrumentId=19");
webRequest.Method = "GET";
using (var response = webRequest.GetResponse())
{...}
The request Fails with Error-Code 403 (Forbidden)
when adding:
webRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 Safari/537.36";
The request Fails with Error-Code 500 (Internal Server Error)
Addition (Edit)
I also initiate with
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 |
SecurityProtocolType.Tls11 |
SecurityProtocolType.Tls |
SecurityProtocolType.Ssl3;
Also, I Tried Setting a CookieContainer, but the result is the same 500.
Why is Postman/Chrome Successfuly querying this API while C# Webrequest do not?
What is the difference?
So, the reason that this is failing is because of the headers being included in the client request from postman by default, though not from the C# request.
Using a program like Fiddler (https://www.telerik.com/fiddler) you can watch the request to see that the headers from the postman request are:
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.84 Safari/537.36
Yet from C# are just
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
Filling in the extra client request headers like this allows it to go through fine:
webRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
webRequest.Headers.Add("Accept-Encoding", "gzip deflate,br");
webRequest.Headers.Add("Accept-Language", "en-US,en;q=0.9");
I need to get some public data from a link, but this is a second time I encounter a Cookie that I cannot replicate as some values in it are dynamic. How can I get this cookie before doing an actual request to get my data?
Example link:
https://www.blackrock.com/ca/individual/en/products/239832/ishares-sptsx-60-index-etf/1464253357804.ajax?tab=chart
Example headers:
Host: www.blackrock.com
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Accept: */*
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: https://www.blackrock.com/ca/individual/en/products/239832/ishares-sptsx-60-index-etf
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
Cookie: JSESSION_amer-retail01=EA5A098E120F4E3FAD2C994A64F292E4.03; ts-ca-one-locale=en_CA; FIRSTPAGE_ca_ca-one=page-class-home; blkUserType-ca-one=individual; ipe_s=0d9c807a-04cc-ae5b-e459-d89bddd2faab; ca-one-recent-funds=239832; __utmt=1; __utma=17023466.874636303.1499433731.1499433731.1499433731.1; __utmb=17023466.5.10.1499433731; __utmc=17023466; __utmz=17023466.1499433731.1.1.utmcsr=google|utmgclid=CJjax_ah99QCFZxMDQodKQ4NxQ|utmgclsrc=aw.ds|utmccn=(not%20set)|utmcmd=(not%20set)|utmctr=(not%20provided); _bizo_bzid=0688d0c2-d203-488b-b128-94c0406b5568; _bizo_cksm=B16C1965883E71EB; _bizo_np_stats=14%3D243%2C; ipe.20668.pageViewedCount=3; utag_main=v_id:015d1d3702f6001b054da00951140407300ad06b00bd0$_sn:1$_ss:0$_pn:6%3Bexp-session$_st:1499436924463$ses_id:1499433730806%3Bexp-session; s_pers=%20s_fid%3D5295C92DBD9DC781-0177F2F787EC9066%7C1562507124793%3B%20gpv%3Dca-one%257Cca%257Cfund%257Cishares%2520s%2526p%252Ftsx%252060%2520index%2520etf%257Cxiu%7C1499436924795%3B%20s_nr%3D1499435124796-New%7C1530971124796%3B%20s_pers_eVar17%3Dprospect%7C1502027124797%3B%20s_pers_prop19%3Danonymous%7C1502027124799%3B; s_sess=%20s_campaign%3Dppc%253Ablackrock_ca%253Agoogle%253Ablackrock%3B%20s_cc%3Dtrue%3B%20s_sq%3D%3B; s_vi=[CS]v1|2CAFC5818507C33B-6000011540002202[CE]
You can instantiate a CookieContainer and set it to the CookieContainer property of your HttpWebRequest. Then make an initial request and after you GetResponse the cookie container will be populated with any cookies returned.
Then reuse the CookieContainer in subsequent requests and all the previous returned cookies will be sent.
I am trying to make a web service call and include and authorization cookie in the header.
I see another site do this, using fiddler and the call looks like -
GET url/services/events HTTP/1.1
Host: url
Connection: keep-alive
Accept: application/json
X-Auth-Token: 33d131bb-37b4-4d87-b646-6edccab4f294
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36
Content-Type: application/json
Referer: url/api-docs/index.html
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Cookie: knownUser=true
How do I code in "X-Auth-Token: 33d131bb-37b4-4d87-b646-6edccab4f294".
Here is my C# code.
HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Add("X-Auth-Token", "33d131bb-37b4-4d87-b646-6edccab4f294");
HttpResponseMessage response = client.PostAsync(new Uri("url?keywords=Taylor%20Swift&eventType=CONCERT&city=Mexico%20City"), new StringContent(JsonConvert.SerializeObject(sp), Encoding.UTF8, "application/json")).Result;
But I keep getting a 500 error. (StatusCode: 500, ReasonPhrase: 'Internal Server Error')
What is the ocrrect syntax to get that token passed intot he web service call?
I'm trying to log into a website to download data through my account. This is the raw Fiddler Request for the POST login form.
POST login/login.jsp HTTP/1.1
Host: server.com
Connection: keep-alive
Content-Length: 73
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Origin: https://server.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: https://server.com/login/login.jsp
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: __utma=109610308.114257620.1370889472.1373479499.1371761934.3; __utmc=109613338; __utmz=109610308.1373249472.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); _bcvm_vid_424161365915852877=4393500994580715020; _bcvm_vrid_424161365915852877=4393492275825713189; WT_FPC=id=199.234.233.42-2645888112.30303753:lv=1371356395815:ss=1371758333825; JSESSIONID=RGJGy4yQ2WCXRPbnhxCTKGb2rZh39b67d8g8PktTQLqfsBQTlTlYLTD!1154156211; BIGipServeresuite_prod_pool=295635768.2713643.0000
It then responds with:
HTTP/1.1 302 Moved Temporarily
Date: Fri, 21 Jun 2013 12:39:46 GMT
Location: https://server.com/login/redirect.jsp?APPLICATION=0
Content-Type: text/html
Set-Cookie: SECURITY_SESSION_ID=383826514*198399234219875960; domain=.server.com; path=/
Connection: Close
Set-Cookie: BIGipServeresuite_prod_pool=294168768.27163.0000; expires=Fri, 21-Jun-2013 13:09:47 GMT; path=/
Content-Length: 3669
That SECURITY_SESSION_ID is what's needed to do anything on the site.
To mimic it I coded wrote this:
//GET the Login page - I preform a quick get to pick up the first two important cookies
HttpWebRequest GETLoginRequest = (HttpWebRequest)HttpWebRequest.Create("https://server.com/login/login.jsp");
GETLoginRequest.Method = "GET";
GETLoginRequest.Accept = "application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*";
GETLoginRequest.AllowAutoRedirect = false;
GETLoginRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)";
GETLoginRequest.CookieContainer = cookieJar;
HttpWebResponse GETLoginResponse = (HttpWebResponse)GETLoginRequest.GetResponse(); //Gets the JSession and BIGipServer cookies
Console.Write(" \n 3rd count after GETLoginResponse : " + cookieJar.Count + "\n");
//POST Login
HttpWebRequest POSTLoginRequest = (HttpWebRequest)HttpWebRequest.Create("https://server.com/login/login.jsp");
POSTLoginRequest.Method = "POST";
WebHeaderCollection myWebHeaderCollection = POSTLoginRequest.Headers;
POSTLoginRequest.AllowAutoRedirect = true;
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
///Cache
POSTLoginRequest.Headers.Add(HttpRequestHeader.CacheControl, "max-age=0");
//Client
POSTLoginRequest.Accept = "application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*";
POSTLoginRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate,sdch");
myWebHeaderCollection.Add("Accept-Language:en-US");
POSTLoginRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)";
//Cookies/Login
POSTLoginRequest.CookieContainer = cookieJar; //The cookie jar contains mainly added
//Entity
POSTLoginRequest.ContentLength = bytes.Length;
POSTLoginRequest.ContentType = "Content-Type: application/x-www-form-urlencoded";
//Miscellanous
POSTLoginRequest.Headers.Add("Origin: https://server.com");
POSTLoginRequest.Referer = "https://server.com/login/login.jsp";
//Transport
//Fix I found to allow Connection: Keep-Alive
var sp = POSTLoginRequest.ServicePoint;
var prop = sp.GetType().GetProperty("HttpBehaviour", BindingFlags.Instance | BindingFlags.NonPublic);
prop.SetValue(sp, (byte)0, null);
ServicePointManager.Expect100Continue = false;
POSTLoginRequest.Host = "server.com";
using (Stream os = POSTLoginRequest.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
HttpWebResponse POSTLoginResponse = (HttpWebResponse)POSTLoginRequest.GetResponse();
Console.Write(" \n 4th count after POSTLoginResponse : " + cookieJar.Count + "\n");
at the end of the day my request in fiddler looks like this:
POST /login/login.jsp HTTP/1.1
Cache-Control: max-age=0
Accept: application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)
Content-Type: Content-Type: application/x-www-form-urlencoded
Origin: https://server.com
Referer: https://server.com/login/login.jsp
Host: esuite.pjm.com
Cookie: __utma=1096103034.235016339.1371048460.1371048460.1371048460.1; __utmz=109610428.1371048460.1.1.utmcsr=bing|utmccn=(organic)|utmcmd=organic|utmctr=pjm; _bcvm_vrid_424161365915852877=4393493862784729423; WT_FPC=id=199.234.233.42-3603288592.30304123:lv=1371044861062:ss=1371044859892; JSESSIONID=RGbYQd7JnPdNkTvtGCzQ9NLyFgfBnnyLFzbvKPg2Y0gLnhL2hp8F!-1770592471; BIGipServeresuite_prod_pool=327723200.27163.0000
Content-Length: 73
Connection: Keep-Alive
which is pretty much the same and yet I get this as a response:
HTTP/1.1 200 OK
Date: Fri, 21 Jun 2013 13:58:17 GMT
Content-Length: 3356
Content-Type: text/html
Set-Cookie: BIGipServeresuite_prod_pool=327723200.27163.0000; expires=Fri, 21-Jun-2013 14:28:17 GMT; path=/
I think one thing has to do with that in the browser the reply is a 302 Moved Temporarily but I really don't know. The two important cookies seem to be JSession and BIGIp because
those are set by the site. The other cookies I manually added in and they seem to be google analytic cookies and I don't think they matter all to much. Anyway the headers are nearly
identicaly but it's still not responding with the SECURITY_SESSION_ID that I'm looking for. Does anyone have any idea on what I'm doing wrong?
Your 'accept' headers are different, and your Content-Type part of the submit is wrong:
POSTLoginRequest.ContentType = "Content-Type: application/x-www-form-urlencoded";
Should be
POSTLoginRequest.ContentType = "application/x-www-form-urlencoded";
Make sure the post parameters are the same as in fiddler example:
So do this first before converting them to bytes.
HttpUtility.Urlencode(forumparameters)
See if that works.
As you are missing counts in ContentLength too, so must be cause of this.
I 'd like to get captcha images.
I have captured request to get images using fiddler.
this is raw text of request.
GET http://api.minteye.com/slider/image.ashx?CaptchaId=4162&PublicKey=76dd4b3f-92d2-4d29-9cb8-740ccfca4fe4&w=300&h=250&dumm=635086078045177660&reqid=e112b222-0f38-4309-9ab4-791aa11590d8-2eb9&img=1 HTTP/1.1
Host: api.minteye.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36
Accept: /
Referer: http://abc-site.com/?param=value
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US;q=0.8,en-US;q=0.6,en;q=0.4
Cookie: __utma=219856005.121997545.1372326418.1372834638.1372992381.5; __utmz=219856005.1372992381.5.2.utmcsr=api.minteye.com|utmccn=(referral)|utmcmd=referral|utmcct=/; ASP.NET_SessionId=oh543ydsa2z033ybv2cg0wwi; ExchangeAds=10959011
After that, i have made request header and send it to getresponse.
but i can't get any image.(can't see anything even in fiddler)
this is my code.
CookieContainer cookie = new CookieContainer();
request = (HttpWebRequest)HttpWebRequest.Create("http://api.minteye.com/slider/image.ashx?CaptchaId=4162&PublicKey=76dd4b3f-92d2-4d29-9cb8-740ccfca4fe4&w=300&h=250&dumm=4211089392&reqid=8075f74e-7878-4a9a-a5a0-3c21b5fbfb79-5f5f&img=5");
request.KeepAlive = true;
request.Accept = "*/*";
request.UserAgent = "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36";
request.Referer = "http://abc-site.com/?param=value";
request.Headers.Add("Accept-Language: en-US;q=0.8,en-US;q=0.6,en;q=0.4");
request.CookieContainer = cookie;
response = (HttpWebResponse)request.GetResponse();
While running this code, fiddler captures requests and its response, but doesn't get any image.
What's wrong?
If you give me your sources, i can look how you became the requestID,
with this requestID can you make the URL for this images.
With this URL, you can download it with a simple WebClient ;)