Develop Reference

HttpWebRequest Operation Timeout with CookieContainer

When I am using HttpWebRequest and add this:
CookieContainer container = new CookieContainer();
request.CookieContainer = container;
it will throw an Operation Timeout exception in
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
If I don't add the CookieContainer the program runs without errors. Why doesn't this work?

I had this problem too. when I am requesting a Web Page and I put the HttpWebRequest in the Page_Load function and add cookie from current page into CookieContainer but it just hang in there till to timeout
the problem is when I first requesting the web page asp dot net just locked the session for protecting session being writing simultaneously from different place. when I try to call HttpWebRequest to request the same domain same Session it will be block till the first page end it's session lock, meanwhile, the first page is waiting for HttpWebRequest ends it job
the two request just waiting for each other till to timeout.
the good news is I found a way to solve this problem, I just set EnableSessionState="ReadOnly" in the first page, it makes first page that I call do not lock the session.
hope this can help someone.

How are you creating Request object??
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(args[0]);
request.CookieContainer = new CookieContainer();
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
CookieContainer
Try to set myWebRequest.Timeout =xxx
If you are running multiple concurrent requests try setting connections limit
System.Net.ServicePointManager.DefaultConnectionLimit
ServicePoint.ConnectionLimit
Check these links for more information:
HttpWebRequest "operation has timed out"
I need help setting .NET HttpWebRequest timeout
Adjusting HttpWebRequest Connection Timeout in C#
here are many reasons why GetResponse() might hang. Some are:
1) You have reached the connection limit on the client ( 2 conns per http/1.1 server, 4 connections per http/1.0 server, or custom limit set on the ServicePoint), and no connection is free to send a request to the server. In other words, you might have 2 requests outstanding (eg: to a 1.1 server) which are taking more than 2 minutes to complete, and then issue another GetResponse() to the same server. The last getresponse() might time out.
2) You have not closed the response stream of an earlier httpwebresponse. So, even if a connection is free, the request wont be sent.
3) The server is taking too long to send the response.

Situation 1:
There can be several reasons beind it. I wrote the following lines after reading about your problem and this code works,
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://google.com");
req.Method = "GET";
req.Timeout = 282;
CookieContainer cont = new CookieContainer();
req.CookieContainer = cont;
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
using (StreamReader reader = new StreamReader(resp.GetResponseStream()))
{
Console.Write(reader.ReadToEnd());
}
resp.Close();
req.Abort();
Console.ReadLine();
I wrotereq.Timeout = 282;because i tested for several values and http://google.com does takes 282 milisecond from my computer to respond. From a slower internet connection, this code may return timeout.
Please be sure to set the time out high enough.
Situation 2:
May be the server you are connecting to takes little bit longer if it realize that cookie is enabled. When you don't set anything to req.CookieContainer then cookie is disabled. So please be sure about this fact. :) hope it will work.

HttpWebRequest type "GET" returning error 400

It seems to be occurring only one machine and none of the other machines.
HttpWebRequest myRequest =(HttpWebRequest)WebRequest.Create("https://connect.zystemsgo.com/auto/");
myRequest.Method = "GET";
SetCertificatePolicy();
Application.DoEvents();
WebResponse myResponse = myRequest.GetResponse();
StreamReader sr = new StreamReader(myResponse.GetResponseStream(),System.Text.Encoding.UTF8);
string result = sr.ReadToEnd();
I tried searching other 400 request errors, but it is not clear. How do I go about debugging this?

HTTP Error 400 means Bad Request. This is being returned by the server.
Usually, when I'm debugging HTTP requests, I use Fiddler to monitor the requests and responses and find out what's going on. It never fails.

(Not really an answer, but too big for comment)
For what it's worth, I ran the following Python code (too lazy to spin up C# :), and it worked fine:
import httplib
conn = httplib.HTTPSConnection('connect.zystemsgo.com')
conn.request('GET', '/auto/')
resp = conn.getresponse()
data = resp.read()
print data # expected ouput, just like visiting in a browser
print resp.status # 200
Are you sure you are showing us the URL that is actually failing, or is your code a more general example?
Perhaps the server certificate is not installed on that machine? I wouldn't expect a HTTP 400 in that case, but it's the only thing I can think of so far...

it is a bad request error .Are there no parameters in the request?
Can you post the response message,it will give some idea of what is going wrong.

The code that i supplied in the comment above works.
WebClient webClient = new WebClient();
webClient.DownloadFile("Your complete url for the file", #"c:\myfile.txt");
you need to have permission to write in the directory of your choice.
You could also try and use the async download if you want.I am not getting why it would not work on a certain machine.

Fast HTTP call ASP.Net

I'm sending an HTTPWebRequest to a 3rd party with the code below. The response takes between 2 and 22 seconds to come back. The 3rd party claims that once they receive it, they are sending back a response immediately, and that none of their other partners are reporting any delays (but I'm not sure I believe them -- they've lied before).
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://www.example.com");
request.Timeout = 38000;
request.Method = "POST";
request.ContentType = "text/xml";
StreamWriter streamOut = new StreamWriter(request.GetRequestStream(), System.Text.Encoding.ASCII);
streamOut.Write(XMLToSend); // XMLToSend is just a string that is maybe 1kb in size
streamOut.Close();
HttpWebResponse resp = null;
resp = (HttpWebResponse)request.GetResponse(); // This line takes between 2 and 22 seconds to return.
StreamReader responseReader = new StreamReader(resp.GetResponseStream(), Encoding.UTF8);
Response = responseReader.ReadToEnd(); // Response is merely a string to hold the response.
Is there any reason that the code above would just...pause? The code is running in a very solid hosting provider (Rackspace Intensive Segment), and the machine it is on isn't being used for anything else. I'm merely testing some code that we are about to put into production. So, it's not that the machine is taxed, and given that it is Rackspace and we are paying a boatload, I doubt it is their network either.
I'm just trying to make sure that my code is as fast as possible, and that I'm not doing anything stupid, because in a few weeks, this code will be ramped up to run 20,000 requests to this 3rd part every hour.

Try doing a flush before you close.
streamOut.Flush();
streamOut.Close();
Also download microsoft network monitor to see for certain if the hold up is you or them, you can download it here...
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=983b941d-06cb-4658-b7f6-3088333d062f&displaylang=en

There is a few things that I would do:
I would profile the code above and get some definitive timings.
Implement the using statements in order to dispose of resources correctly.
Write the code in an async style there's going to be an awful lot of IO wait once its ramped.

Can you hit the URL in a regular ole browser? How fast is that?
Can you hit other URL's (not your partner's) in this code? How fast is that?
It is entirely possible you're getting bitten by the 'latency bug' where even an instant response from your partner results in unpredictable delays from your perspective.
Another thought: I noticed the https in your URL. Is it any faster with http?

vexed! POST returns a 302 found object moved error in HttpWebRequest

Someone please help - been struggling with this lousy problem!
What I'm doing - I have an ASPX page from which I originate a GET and then a POST to a HTTPS page with a view to login to it. I have spent quite a bit of time comparing my GET and POST construction to a browser GET/POST using fiddler (protocol analyzer) and my requests are fine.
However, when I try login through the browser, everything works fine and it logs in. When I run my page, I can see the correct GET and POST, but I get a 302 found 'object moved error'
Originally I thought this was a cookie issue, but after much experimentation I'm pretty sure this has nothing to do with cookies. I have disabled cookies AND javascript on the browser and tried, and the pages work fine without either. I then simulated the exact GET/POST.
This is my situation:
My GET and the browsers GET are EXACTLY THE SAME
The 200 OK response from the site is EXACTLY the same EXCEPT three VIEWSTATE variables which have slightly different lengths (why? why different even if GET is same?)
My POST and the browsers POST are EXACTLY the same EXCEPT the 3 Viewstate variables (I fill it correctly from the GET)
And yet, the browser logs in, while I get a 302 found / object moved errror.
A couple of other things -
a) I copied the POST response from a recent browser POST and replaced my POST params with this browser POST and that got me the right response! This indicates that
- my headers are just fine
- my coding setup / environment etc. are fine
- something fishy in the VIEWSTATE values, which can only be because the browser sent it to me in the first place (there is no corruption in my parsing the GET VIEWSTATE variables and using it in POST, it's perfectly fine)
update I have also tried WebClient just to check - no difference, same 302.
update The object moved basically points to a error page which says 'a serious error occurred blah blah' - the POST is causing a error at the server, and the ONLY difference between the good POST (of the browser) and my POST are the Viewstate variables.
So - WHAT AM I DOING WRONG? Why is this cruel world tormenting me?!!
(PS - one other difference in the browser sequence, not sure how it matters)
Browser:
CONNECT
GET
GET (for a favicon, which returns an error)
CONNECT
POST (success)
Me:
CONNECT
GET
POST (flaming failure, 302 - page moved)
and for those who care, my POST header construction code
HttpWebRequest myRequest = (HttpWebRequest)WebRequest.Create(URL);
myRequest.UserAgent = chromeUserAgent;
//myRequest.CookieContainer = cCookies;
myRequest.ContentType = "application/x-www-form-urlencoded";
myRequest.Accept = chromeAccept;
myRequest.Referer = url;
myRequest.AllowAutoRedirect = false;
myRequest.Host = "thesitethatskillingme.com";
myRequest.Headers.Add("Origin", "https://thesitethatskillingme.com");
myRequest.Headers.Add("Accept-Encoding", chromeAcceptEncoding);
myRequest.Headers.Add("Accept-Language", chromeAcceptLanguage);
myRequest.Headers.Add("Accept-Charset", chromeAcceptCharset);
myRequest.Headers.Add("Cache-Control", "max-age=0");
myRequest.ServicePoint.Expect100Continue = false;
myRequest.Method = "POST";
myRequest.KeepAlive = true;
ASCIIEncoding ascii = new ASCIIEncoding();
byte[] bData = ascii.GetBytes(data);
myRequest.ContentLength = bData.Length;
using (Stream oStream = myRequest.GetRequestStream())
oStream.Write(bData, 0, bData.Length);
...and then read stream etc. no cookies.

I finally figured it out - and hopefully someone else who chances upon the same problem does not have to go through this again. It's possible that most HTTP gurus and people familiar with WWW development would never hit it, but a newbie quite well could.
So what was the problem? I had narrowed down the problem to VIEWSTATE which I always suspected (see my post above...). It turns out that all I had to do was to Server.UrlEncode the parsed VIEWSTATE values before putting them onto POST - that's it. It took me all day to get to that.
SO, as a learning to other newcomers
If you are trying to POST to a page through code and need to send it VIEWSTATE variables that you parsed from GET, then first Server.UrlEncode it before creating the parameters - for e.g.
do GET
get the response stream into a string
parse the string (I use HtmlAgilityPack- fabulous)
param1 = name +"="+Server.UrlEncode(value)+"&"
POST param = param1+param2+...
-send this in POST - voila, it works
because I have never, ever programmed with HttpWebRequest etc., I started by narrowing down my problem by eliminating cookies, javascript, GET construction, POST construction one-by-one using fiddler (great analyzer tool, free) and then finally did byte-comparison using BeyondCompare, and that's when I caught the VIEWSTATE variable modifications.
I learnt a lesson on URL encoding, and hopefully you won't have to!

System.Net.WebException: The operation has timed out

I have a big problem: I need to send 200 objects at once and avoid timeouts.
while (true)
{
NameValueCollection data = new NameValueCollection();
data.Add("mode", nat);
using (var client = new WebClient())
{
byte[] response = client.UploadValues(serverA, data);
responseData = Encoding.ASCII.GetString(response);
string[] split = Javab.Split(new[] { '!' }, StringSplitOptions.RemoveEmptyEntries);
string command = split[0];
string server = split[1];
string requestCountStr = split[2];
switch (command)
{
case "check":
int requestCount = Convert.ToInt32(requestCountStr);
for (int i = 0; i < requestCount; i++)
{
Uri myUri = new Uri(server);
WebRequest request = WebRequest.Create(myUri);
request.Timeout = 200000;
WebResponse myWebResponse = request.GetResponse();
}
break;
}
}
}
This produces the error:
Unhandled Exception: System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at vir_fu.Program.Main(String[] args)
The requestCount loop works fine outside my base code but when I add it to my project I get this error. I have tried setting request.Timeout = 200; but it didn't help.

It means what it says. The operation took too long to complete.
BTW, look at WebRequest.Timeout and you'll see that you've set your timeout for 1/5 second.

Close/dispose your WebResponse object.

I'm not sure about your first code sample where you use WebClient.UploadValues, it's not really enough to go on, could you paste more of your surrounding code? Regarding your WebRequest code, there are two things at play here:
You're only requesting the headers of the response**, you never read the body of the response by opening and reading (to its end) the ResponseStream. Because of this, the WebRequest client helpfully leaves the connection open, expecting you to request the body at any moment. Until you either read the response body to completion (which will automatically close the stream for you), clean up and close the stream (or the WebRequest instance) or wait for the GC to do its thing, your connection will remain open.
You have a default maximum amount of active connections to the same host of 2. This means you use up your first two connections and then never dispose of them so your client isn't given the chance to complete the next request before it reaches its timeout (which is milliseconds, btw, so you've set it to 0.2 seconds - the default should be fine).
If you don't want the body of the response (or you've just uploaded or POSTed something and aren't expecting a response), simply close the stream, or the client, which will close the stream for you.
The easiest way to fix this is to make sure you use using blocks on disposable objects:
for (int i = 0; i < ops1; i++)
{
Uri myUri = new Uri(site);
WebRequest myWebRequest = WebRequest.Create(myUri);
//myWebRequest.Timeout = 200;
using (WebResponse myWebResponse = myWebRequest.GetResponse())
{
// Do what you want with myWebResponse.Headers.
} // Your response will be disposed of here
}
Another solution is to allow 200 concurrent connections to the same host. However, unless you're planning to multi-thread this operation so you'd need multiple, concurrent connections, this won't really help you:
ServicePointManager.DefaultConnectionLimit = 200;
When you're getting timeouts within code, the best thing to do is try to recreate that timeout outside of your code. If you can't, the problem probably lies with your code. I usually use cURL for that, or just a web browser if it's a simple GET request.
** In reality, you're actually requesting the first chunk of data from the response, which contains the HTTP headers, and also the start of the body. This is why it's possible to read HTTP header info (such as Content-Encoding, Set-Cookie etc) before reading from the output stream. As you read the stream, further data is retrieved from the server. WebRequest's connection to the server is kept open until you reach the end of this stream (effectively closing it as it's not seekable), manually close it yourself or it is disposed of. There's more about this here.

proxy issue can cause this. IIS webconfig put this in
<defaultProxy useDefaultCredentials="true" enabled="true">
<proxy usesystemdefault="True" />
</defaultProxy>

I remember I had the same problem a while back using WCF due the quantity of the data I was passing. I remember I changed timeouts everywhere but the problem persisted. What I finally did was open the connection as stream request, I needed to change the client and the server side, but it work that way. Since it was a stream connection, the server kept reading until the stream ended.

I encountered the same error than adding
Task.Delay(2000);
in each request solved the problem