I'm sending an HTTPWebRequest to a 3rd party with the code below. The response takes between 2 and 22 seconds to come back. The 3rd party claims that once they receive it, they are sending back a response immediately, and that none of their other partners are reporting any delays (but I'm not sure I believe them -- they've lied before).
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://www.example.com");
request.Timeout = 38000;
request.Method = "POST";
request.ContentType = "text/xml";
StreamWriter streamOut = new StreamWriter(request.GetRequestStream(), System.Text.Encoding.ASCII);
streamOut.Write(XMLToSend); // XMLToSend is just a string that is maybe 1kb in size
streamOut.Close();
HttpWebResponse resp = null;
resp = (HttpWebResponse)request.GetResponse(); // This line takes between 2 and 22 seconds to return.
StreamReader responseReader = new StreamReader(resp.GetResponseStream(), Encoding.UTF8);
Response = responseReader.ReadToEnd(); // Response is merely a string to hold the response.
Is there any reason that the code above would just...pause? The code is running in a very solid hosting provider (Rackspace Intensive Segment), and the machine it is on isn't being used for anything else. I'm merely testing some code that we are about to put into production. So, it's not that the machine is taxed, and given that it is Rackspace and we are paying a boatload, I doubt it is their network either.
I'm just trying to make sure that my code is as fast as possible, and that I'm not doing anything stupid, because in a few weeks, this code will be ramped up to run 20,000 requests to this 3rd part every hour.
Try doing a flush before you close.
streamOut.Flush();
streamOut.Close();
Also download microsoft network monitor to see for certain if the hold up is you or them, you can download it here...
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=983b941d-06cb-4658-b7f6-3088333d062f&displaylang=en
There is a few things that I would do:
I would profile the code above and get some definitive timings.
Implement the using statements in order to dispose of resources correctly.
Write the code in an async style there's going to be an awful lot of IO wait once its ramped.
Can you hit the URL in a regular ole browser? How fast is that?
Can you hit other URL's (not your partner's) in this code? How fast is that?
It is entirely possible you're getting bitten by the 'latency bug' where even an instant response from your partner results in unpredictable delays from your perspective.
Another thought: I noticed the https in your URL. Is it any faster with http?
Related
My problem is when using WebClient or WebRequest/HttpRequest that the first request often takes a bit more than 40 seconds and all subsequent ones only around 250 milliseconds regardless of size (if it is serveral k characters or only 40). Im only trying to get JSON data from some endpoint.
with WebClient I use wc.DownloadString("some address");
with WeRequest I use (HttpWebRequest) WebRequest.Create("some address") and webRequest.GetResponse(); with according stream handling afterwards
I already looked around for a Solution, and found:
WebRequest.DefaultWebProxy = GlobalProxySelection.GetEmptyWebProxy();
WebRequest.DefaultWebProxy = null;
webclient.Proxy = null;
webclient.Proxy = GlobalProxySelection.GetEmptyWebProxy();
The above mentionen options are NOT reliable working for me.
Are there any other ways to reliable fix this huge delay for first Request? I would prefere a solution without external libs if possible.
As a note I use .Net Framework 4.5.
I would like to know how Reusing HttpWebRequests works to avoid SSL handshake process everytime.
I use the keep alive headr in the request and the first handshake is successfull but i would like to reuse the request in order to avoid future handshakes against the same certificate.
Think is i dont know if i had to reuse the HttpWebRequest object instance or even if i create a new request object it will use the same connection since the keep alive is already on place and working.
Should i store the existing request object lets say at class level and reuse it? or i can safely dispose the object and next time i create a request it will be under the effect of the keep alive connection?
I am asking this cause i need to lower the timings in an application, and worst part is always ssl handshake, that can take over 3seconds in a phone with medium signal from carrier.
I am using C# to develop.
I have tried to look for this kind of information but all i read over internet is how to set up the SSL Server and enabling certain settings but not how to make the client work with these features.
EDIT: FINDINGS RESULTS
I created a sample program in .NET C# whith the following code:
Stopwatch sw = new Stopwatch();
sw.Start();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri(#"https:\\www.gmail.com"));
request.KeepAlive = true;
request.Method = "GET";
request.ContentType = "application/json";
request.ContentLength = 0;
request.ConnectionGroupName = "test";
//request.UnsafeAuthenticatedConnectionSharing = true;
//request.PreAuthenticate = true;
var response = request.GetResponse();
//response.Close();
request.Abort();
sw.Stop();
listBox1.Items.Add("Connection in : " + sw.Elapsed.ToString());
sw.Reset();
sw.Start();
HttpWebRequest request2 = (HttpWebRequest)WebRequest.Create(new Uri(#"https:\\www.gmail.com"));
request2.KeepAlive = true;
request2.Method = "GET";
//request2.UnsafeAuthenticatedConnectionSharing = true;
//request2.PreAuthenticate = true;
request2.ContentType = "application/json";
request2.ContentLength = 0;
request2.ConnectionGroupName = "test";
var response2 = request2.GetResponse();
//response2.Close();
request2.Abort();
sw.Stop();
listBox1.Items.Add("Connection 2 in : " + sw.Elapsed.ToString());
Results was that the first connection triggered the CertificatevalidationCallback 3 times (one for each certificate) and then the second connection only once, but when i CLOSED THE RESPONSE before performing the next request, no callback was triggered.
I suppose that keeping a response open keeps the socket open and thats why the partial handshake takes place (not the full certificate chain).
Sorry if I sound kind of noob in this matter, SSL and timings was coded by a work mate and the code was not clear to follow. But i think i have the answer. Thanks Poupou for your tremendous help
This is already built-in the the SSL/TLS stack shipped with Xamarin.iOS (i.e. at a lower level than HttpWebRequest). There's nothing to set up to enabled this. In fact you would need extra code if you wanted to disable it.
If the server supports it then subsequent handshake will already faster because a Session ID cache will be used (see TLS 1.0 RFC page 30).
However the server does not have to honor the session id (given to it). In such case a full handshake will need be done again. IOW you cannot force this from the client (only offer).
You can verify this using a network analyzer, e.g. wireshark, by looking at the exchanges (and comparing them to the RFC).
When I am using HttpWebRequest and add this:
CookieContainer container = new CookieContainer();
request.CookieContainer = container;
it will throw an Operation Timeout exception in
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
If I don't add the CookieContainer the program runs without errors. Why doesn't this work?
I had this problem too. when I am requesting a Web Page and I put the HttpWebRequest in the Page_Load function and add cookie from current page into CookieContainer but it just hang in there till to timeout
the problem is when I first requesting the web page asp dot net just locked the session for protecting session being writing simultaneously from different place. when I try to call HttpWebRequest to request the same domain same Session it will be block till the first page end it's session lock, meanwhile, the first page is waiting for HttpWebRequest ends it job
the two request just waiting for each other till to timeout.
the good news is I found a way to solve this problem, I just set EnableSessionState="ReadOnly" in the first page, it makes first page that I call do not lock the session.
hope this can help someone.
How are you creating Request object??
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(args[0]);
request.CookieContainer = new CookieContainer();
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
CookieContainer
Try to set myWebRequest.Timeout =xxx
If you are running multiple concurrent requests try setting connections limit
System.Net.ServicePointManager.DefaultConnectionLimit
ServicePoint.ConnectionLimit
Check these links for more information:
HttpWebRequest "operation has timed out"
I need help setting .NET HttpWebRequest timeout
Adjusting HttpWebRequest Connection Timeout in C#
here are many reasons why GetResponse() might hang. Some are:
1) You have reached the connection limit on the client ( 2 conns per http/1.1 server, 4 connections per http/1.0 server, or custom limit set on the ServicePoint), and no connection is free to send a request to the server. In other words, you might have 2 requests outstanding (eg: to a 1.1 server) which are taking more than 2 minutes to complete, and then issue another GetResponse() to the same server. The last getresponse() might time out.
2) You have not closed the response stream of an earlier httpwebresponse. So, even if a connection is free, the request wont be sent.
3) The server is taking too long to send the response.
Situation 1:
There can be several reasons beind it. I wrote the following lines after reading about your problem and this code works,
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://google.com");
req.Method = "GET";
req.Timeout = 282;
CookieContainer cont = new CookieContainer();
req.CookieContainer = cont;
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
using (StreamReader reader = new StreamReader(resp.GetResponseStream()))
{
Console.Write(reader.ReadToEnd());
}
resp.Close();
req.Abort();
Console.ReadLine();
I wrotereq.Timeout = 282;because i tested for several values and http://google.com does takes 282 milisecond from my computer to respond. From a slower internet connection, this code may return timeout.
Please be sure to set the time out high enough.
Situation 2:
May be the server you are connecting to takes little bit longer if it realize that cookie is enabled. When you don't set anything to req.CookieContainer then cookie is disabled. So please be sure about this fact. :) hope it will work.
It seems to be occurring only one machine and none of the other machines.
HttpWebRequest myRequest =(HttpWebRequest)WebRequest.Create("https://connect.zystemsgo.com/auto/");
myRequest.Method = "GET";
SetCertificatePolicy();
Application.DoEvents();
WebResponse myResponse = myRequest.GetResponse();
StreamReader sr = new StreamReader(myResponse.GetResponseStream(),System.Text.Encoding.UTF8);
string result = sr.ReadToEnd();
I tried searching other 400 request errors, but it is not clear. How do I go about debugging this?
HTTP Error 400 means Bad Request. This is being returned by the server.
Usually, when I'm debugging HTTP requests, I use Fiddler to monitor the requests and responses and find out what's going on. It never fails.
(Not really an answer, but too big for comment)
For what it's worth, I ran the following Python code (too lazy to spin up C# :), and it worked fine:
import httplib
conn = httplib.HTTPSConnection('connect.zystemsgo.com')
conn.request('GET', '/auto/')
resp = conn.getresponse()
data = resp.read()
print data # expected ouput, just like visiting in a browser
print resp.status # 200
Are you sure you are showing us the URL that is actually failing, or is your code a more general example?
Perhaps the server certificate is not installed on that machine? I wouldn't expect a HTTP 400 in that case, but it's the only thing I can think of so far...
it is a bad request error .Are there no parameters in the request?
Can you post the response message,it will give some idea of what is going wrong.
The code that i supplied in the comment above works.
WebClient webClient = new WebClient();
webClient.DownloadFile("Your complete url for the file", #"c:\myfile.txt");
you need to have permission to write in the directory of your choice.
You could also try and use the async download if you want.I am not getting why it would not work on a certain machine.
I need to test around 300 URLs to verify if they lead to actual pages or redirect to some other page. I wrote a simple application in .NET 2.0 to check it, using HttpWebRequest. Here's the code snippet:
System.Net.HttpWebRequest wr = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create( url );
System.Net.HttpWebResponse resp = (System.Net.HttpWebResponse)wr.GetResponse();
code = resp.StatusDescription;
Code ran fast and wrote to file that all my urls return status 200 OK. Then I realized that by default GetResponse() follows redirects. Silly me! So I added one line to make it work properly:
System.Net.HttpWebRequest wr = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create( url );
wr.AllowAutoRedirect = false;
System.Net.HttpWebResponse resp = (System.Net.HttpWebResponse)wr.GetResponse();
code = resp.StatusDescription;
I ran the program again and waited... waited... waited... It turned out that for each url I was getting a System.Net.WebException "The operation has timed out". Surprised, I checked the URL manually - works fine... I commented out AllowAutoRedirect = false line - and it works fine again. Uncommented this line - timeout. Any ideas what might cause this problem and how to work around?
Often timeouts are due to web responses not being disposed. You should have a using statement for your HttpWebResponse:
using (HttpWebResponse resp = (HttpWebResponse)wr.GetResponse())
{
code = resp.StatusDescription;
// ...
}
We'd need to do more analysis to predict whether that's definitely the problem... or you could just try it :)
The reason is that .NET has a connection pool, and if you don't close the response, the connection isn't returned to the pool (at least until the GC finalizes the response). That leads to a hang while the request is waiting for a connection.