Develop Reference

HttpWebRequest random 'request timed out' exception

I am currently developing in Unity (in particular using C#) and I'm stuck with HttpWebRequest - HttpWebResponse random timeouts.
I have some methods that send a POST request to a server I host on my local machine (XAMPP) to use various php scripts which are going to fetch informations from MySQL Database (hosted with XAMPP) and give back those info in JSON format.
Then I handle these JSON informations with my C# scripts.
The problem is that when I run the first test all is good:I can get the JSON data from my Server and show it in the Debug Console.
When I run the second test,a WebException is raised with error:
WebException - The request timed out
After that second test,if I run again and again,the problem keeps presenting in a random way.
I followed all the guidelines I found on the internet on how to setup a webrequest - webresponse properly,in particular I tried to use ServicePoint.DefaultConnectionLimit and ServicePoint.MaxServicePointIdleTime,without any result.
The general structure of my methods (regarding the web request/response part) is something like that:
public void WebMethod(){
string post_url = "http://localhost/service.php?someparam=1&someparam=2";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(post_url);
request.Method = "POST";
request.KeepAlive = false;
request.Timeout = 5000;
request.Proxy = null;
string Response = "";
try
{
using (HttpWebResponse resp = request.GetResponse() as HttpWebResponse)
{
using (Stream objStream = resp.GetResponseStream())
{
using (StreamReader objReader = new StreamReader(objStream, Encoding.UTF8))
{
Response = objReader.ReadToEnd();
objReader.Close();
}
objStream.Flush();
objStream.Close();
}
resp.Close();
}
}catch(WebException e)
{
Debug.Log(e.Message);
}
finally
{
request.Abort();
}
//tried this one after reading some related answers here on StackOverflow,without results
//GC.Collect();
Debug.Log("SERVER RESPONSE:" + Response);
//Response Handling
}
I know that it may be something related to a wrong abort on the HttpWebRequest / Response or maybe related to the HTTP 1.1 connections limit,but I can't figure out any solution at the moment.
Any help is appreciated.

HttpWebRequest .GetResponse throws WebException 'The operation has timed out'

I've been working on a project which makes use of an RTC API and forms authentication. I've hit a bit of bizarre behaviour and I just can't figure this one out.
The scenario that has played out to date is that I can successfully run this project locally end to end. That is, this specific piece of code can:
Contact the remote server and successfully authenticate
After authentication I'm able to pass XML to update a ticket in RTC
The problem starts when I publish to our IIS (7.5) server. All works fine right up until the last .GetResponse call which uses a PUT method to pass my XML to update the ticket in RTC. I keep getting 'The operation has timed out'.
I've spent literally days trying to figure this one out doing all manner of things but nothing has proved useful.
As a test I changed the PUT method on the second call to a GET. And it works! If I used a PUT with the .AllowAutoRedirect = false it works in that I get a response back, but then nothing happens on the RTC side so the request is clearly being ignored. I also noticed that the status being returned is marked as 'Found' instead of 'OK'.
Some people thought at this stage perhaps it was a lack of connectivity between the remote server and the web server. This wouldn't be the case as authentication works and this happens against the same server. I have also manually passed the XML / PUT call using the RESTClient on the web server which was accepted fine.
I just can't understand why it works end to end when running locally, but plays up once deployed to IIS?
I tried using log tracing and I'm not entirely sure if I'm getting anything useful from it. It might be totally unrelated but I can see this in the log that is generated on the IIS server:
<EventData>
<Data Name="ContextId">{00000000-0000-0000-12AF-0080000000F8}</Data>
<Data Name="ModuleName">ManagedPipelineHandler</Data>
<Data Name="Notification">128</Data>
<Data Name="HttpStatus">500</Data>
<Data Name="HttpReason">Internal Server Error</Data>
<Data Name="HttpSubStatus">0</Data>
<Data Name="ErrorCode">0</Data>
<Data Name="ConfigExceptionInfo"></Data>
</EventData>
As I say, I'm not sure if this is even related to the problem I'm having, but rather than ignore it I thought I'd share.
Code that forms the call (excuse the standard of coding, it's work in progress and got messy trying out different things to fix this problem)
//Setup webrequest
CookieContainer _cookies = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(getPath);
var test44 = test4.ToString();
request.CookieContainer = _cookies;
request.ContentType = "application/rdf+xml";
request.Accept = "application/rdf+xml";
request.Method = "PUT";
request.AllowAutoRedirect = true;
request.AllowWriteStreamBuffering = true;
request.Timeout = 40000;
byte[] bytes = Encoding.ASCII.GetBytes(test44);
request.ContentLength = bytes.Length;
Stream dataStream = request.GetRequestStream();
dataStream.Write(bytes, 0, bytes.Length);
dataStream.Close();
//Pass request
logger.Info("Made it up to start of RTC request for secure document.");
using (HttpWebResponse getrespn = requestSecureDocument(request, "https://myserver:9100/jazz", "username", "pass", test44))
{
//Stream ReceiveStream = getrespn.GetResponseStream();
// Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
//StreamReader readStream = new StreamReader(ReceiveStream);
//response = readStream.ReadToEnd();
getrespn.Close();
}
The segment of code which interacts with the RTC server (based on the example from: https://nkumar83.wordpress.com/2013/06/13/consuming-rtc-rational-team-concert-oslc-apis-using-c-post-1-authentication/ with my own tweaks):
public static HttpWebResponse requestSecureDocument(HttpWebRequest _requestItem, string _rtcServerURL, string _userName, string _password, string passXml)
{
try
{
//FormBasedAuth Step 1: Request the resource
HttpWebRequest _request = (HttpWebRequest)WebRequest.Create(_requestItem.RequestUri);
_request.CookieContainer = _requestItem.CookieContainer;
//store the response in _docResponse variable
HttpWebResponse _docResponse = (HttpWebResponse)_request.GetResponse();
//HttpStatusCode.OK indicates that the request succeeded
if (_docResponse.StatusCode == HttpStatusCode.OK)
{
//X-com-ibm-team... header signifies form based authentication is being used
string _rtcAuthHeader = _docResponse.Headers["X-com-ibm-team-repository-web-auth-msg"];
if ((_rtcAuthHeader != null) && _rtcAuthHeader.Equals("authrequired"))
{
_docResponse.GetResponseStream().Flush();
_docResponse.Close();
//Prepare form for authentication
HttpWebRequest _formPost = (HttpWebRequest)WebRequest.Create(_rtcServerURL + "/j_security_check");
_formPost.Method = "POST";
_formPost.Timeout = 30000;
_formPost.CookieContainer = _request.CookieContainer;
_formPost.Accept = "text/xml";
_formPost.ContentType = "application/x-www-form-urlencoded";
string _authString = "j_username=" + _userName + "&j_password=" + _password;
Byte[] _outBuffer = Encoding.UTF8.GetBytes(_authString);
_formPost.ContentLength = _outBuffer.Length;
Stream _str = _formPost.GetRequestStream();
_str.Write(_outBuffer, 0, _outBuffer.Length);
_str.Close();
//FormBasedAuth Step 2: Submit the login form and get response
HttpWebResponse _formResponse = (HttpWebResponse)_formPost.GetResponse();
_rtcAuthHeader = _formResponse.Headers["X-com.ibm-team.repository-web-auth-msg"];
//Check if auth failed
if ((_rtcAuthHeader != null) && _rtcAuthHeader.Equals("authfailed"))
{
//auth fialed
var fail = "";
}
else
{
//login successful
//FormBasedAuth Step 3: Resend the request for the protected resource
_formResponse.GetResponseStream().Flush();
_formResponse.Close();
using (HttpWebResponse getresp = (HttpWebResponse)_requestItem.GetResponse()) *** THIS IS TH LINE WHICH THROWS THE EXCEPTION ***
{
return getresp;
}
}
}
}
return _docResponse;
}
catch (WebException e)
{
var filePath = AppDomain.CurrentDomain.GetData("DataDirectory") + #"/trapA.xml";
using (StreamWriter writer = new StreamWriter(filePath, true))
{
writer.WriteLine("Message: Failed to trigger getresponse successfully: " + e);
}
}
return null;
}
Hope someone out there can help :o)

Well I'm pleased to say I've finally got to the bottom of this one. Turns out the problem wasn't anything to do with IIS and does actually work when published 'if' I'm not using the RTC client to make updates to a ticket.
The short story is that our RTC client uses a custom script to post out to our web api. However the RTC client appears to put a record lock on the ticket your trying to update which is persisted until a response from our API is provided. Of course this can't happen because part of the response is to confirm if the update was successful which can't happen due to the lock made by the RTC client.
The solution was to get the call in from RTC closed as quickly as possible. So the segment of code which authenticates and calls back out to RTC to make updates is now wrapped around with some new code to create a new thread. This has allowed the connection to be closed in about 5 seconds, all the while our app continues to make the necessary calls to complete the transaction.
Thread t = new Thread(() = > {
//code here
}

Parallel async HttpWebRequests with different proxies. How to maximize performance?

I'm writing an app that needs to do a lot of parallel httpwebrequests to bookmaker site from different proxies. The reason why i'm using different proxies is that bookmaker can ban ip if there are many requests from it. My goal is to get freshest site content as fast as possible.
Here is my code with all settings that I have:
ServicePointManager.DefaultConnectionLimit = 1000;
ServicePointManager.Expect100Continue = false;
ServicePointManager.UseNagleAlgorithm = false;
for (var i = 0; i < proxyCollection.Count; i++)
{
var proxyLocal = proxyCollection[i];
var iLocal = i;
Task.Run(async () =>
{
var httpWebRequest = (HttpWebRequest) WebRequest.Create(String.Format("https://bookmaker.com/page{0}", iLocal));
httpWebRequest.Proxy = proxyLocal;
httpWebRequest.PreAuthenticate = true;
httpWebRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
using (var httpWebResponse = (HttpWebResponse) await httpWebRequest.GetResponseAsync())
using (var responseStream = httpWebResponse.GetResponseStream())
using (var streamReader = new StreamReader(responseStream))
{
var stringContent = await streamReader.ReadToEndAsync();
//Here i'm processing new data. It works pretty fast, so this isn't a problem
ProcessStringContent(stringContent);
}
});
}
And this code works not so fast than I expected.
First problem is that in strange reason all requests don't start at one time. As I can see in task manager loading has two or more maximums. More over I have one realy fast proxy and proxyCollection contains it. But if I use await Task.Delay(5000) after code above, in some cases to the moment when 5000ms have passed the request with my fast proxy doesn't even start!
Second problem is that total time of all task execution is slow. I expected that if one request needs 200-300ms to execute, than 100 requests in parallel and async needs a little more time. But some times this "more" is 10-20 times. I have a suspicion, that something is wrong.
Third problem is that when I'm running this code it freezes UI (not full freeze, but UI lags). I read that WebRequest.Create is processing synchronously and can get some time (dns lookup, proxy settings e.t.c) and if I'm making a lot of requests they can simply fill all my threads (UI thread too) for creating WebRequests. But I tried to create requests to direct ip address (WebRequest.Create(String.Format("https://10.10.10.1/page{0}", iLocal)) - nothing changed, and i'm setting proxy (so auto detect proxy isn't needed), so I don't understand why can creating take so much time (and is problem with creating or maybe with smth else?).
Please can someone point me what i'm doing wrong? I'm lost in all ServicePointManager settings (all what I tried didn't help). Can .Net deal with such type of task? Or maybe I need to use nodejs for example for best performance?
P.S: Collection of proxies is not such big (50-100 different proxies).

Back to (Parallel) Basics: Don't Block Your Threads, Make Async I/O Work For You
Example:
Parallel.For(0, 10, delegate(int i) {
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(
new Uri("http://www.mysi.com"));
string dataToSend = "Data";
byte[] buffer = System.Text.Encoding.GetEncoding(1252).
GetBytes(dataToSend);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = buffer.Length;
request.Host = "www.mysite.com";
Stream requestStream = request.GetRequestStream();
requestStream.Write(buffer, 0, buffer.Length);
requestStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
});
Send multiple WebRequest in Parallel.For

C# HTTP web request keeps timing out

I am making a Http Webrequest to an available site that I can visit fine, but the HTTP Web request keeps timing out. Is there any reason why this code might allow it to timeout when it shouldn't?
I've tried upping the timeout setting, but it still continues to timeout.
Uri CameraUrl = new Uri("http://" + cfg_cameraIps[i]);
HttpWebRequest myRequest = (HttpWebRequest)WebRequest.Create(CameraUrl);
myRequest.Timeout = 5000;
myRequest.Method = "HEAD";
try
{
HttpWebResponse webresponse;
webresponse = (HttpWebResponse)myRequest.GetResponse();
if (webresponse.StatusCode.ToString() == "OK")
{
continue;
}

You're not closing your web response - if you find that the first couple of requests work, but ones after that don't, then that's the problem. It's trying to reuse the existing connection to the server, but it can't because you haven't closed the response.
Change your code to:
using (HttpWebResponse webresponse = (HttpWebResponse) myRequest.GetResponse())
{
if (webresponse.StatusCode == HttpStatusCode.OK)
{
continue;
}
...
}
and see if that helps.
If it's failing on the very first request to the server, then that's something different. In that case, use Wireshark to see what's going on at the network level.
Note that in the code above I've also removed the string conversion in favour of comparing the status codes directly.

how to receive server push data in c#?

I am writing a program. my program receive data from a server through HTTP protocol. the data will be pushed by server to my program.
I tried to use WebRequest, but only received one session of data.
How can i keep the connection alive, to receive the data from server continuosly,
Any help is appreciated.
the following is the SDK document:
Under the authorization of GUEST or ADMIN, it is possible to get the series of live images
(Server push). To get the images, send the request to “/liveimg.cgi?serverpush=1” as shown
in the Figure. 2-1-1.
When the camera receives the above request from the client, it sends the return as shown
in the Figure. 2-2.
Each JPEG data is separated by “--myboundary”, and “image/jpeg” is returned as
“Content-Type” header, after “--myboundary”. For “Content-Length” header, it returns the
number of bytes in the --myboundary data (excluding “--myboundary”, each header, and
\r\n as delimiter). After the “Content-Length” header and “\r\n” (delimiter), the actual
data will be sent.
This data transmission will continue until the client stop the connection (disconnect), or
some network error occurs.
int len;
string uri = #"http://192.168.0.2/liveimg.cgi?serverpush=1";
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(uri);
req.Credentials = new NetworkCredential("admin", "admin");
req.KeepAlive = true;
string line = "";
HttpWebResponse reply = (HttpWebResponse)req.GetResponse();
Stream stream = reply.GetResponseStream();
System.Diagnostics.Debug.WriteLine(reply.ContentType);
StreamReader reader = new StreamReader(stream);
do
{
line = reader.ReadLine();
System.Diagnostics.Debug.WriteLine(line);
System.Threading.Thread.Sleep(300);
} while (line.Length>0);

You can keep an HTTP connection open for an extended period of time, if the server supports doing so. (As already mentioned, this will significantly limit the number of simultaneous users you can support.)
The server will need to be set Response.Buffer=false, and have an extended ScriptTimeout (I'm assuming your using ASP.NET on the server side). Once you do that, your page can keep sending Response.Write data as needed until whatever it is doing is done.
Your client will need to process the incoming Response before the connection is complete rather than blocking for the complete response.

You may want to take a look at StreamHub Push Server - its a popular Comet server and has an .NET Client SDK which allows you to receive real-time push updates in C# (or VB / C++).

If I'm understanding you correctly, your server is going to respond to some event by sending data to your client outside of the client making a request/response. Is this correct? If so, I wouldn't recommend trying to keep the connection open unless you have a very small number of clients -- there are a limited number of connections available, so keeping them open may rapidly result in an exception.
Probably the easiest solution would be to have the clients poll periodically for new data. This would allow you to use a simple server and you'd only have to code a thread on the client to request any changes or new work once every minute or thirty seconds or whatever your optimal time period is.
If you truly want to have the server notify the clients proactively, without them polling, then you'll have to do something other than a simple web server -- and you'll also have to code and configure the client to accept incoming requests. This may be difficult if your clients are running behind firewalls and such. If you go this route, WCF is probably your best choice, as it will allow you to configure server and client appropriately.

You need to get a cookie from IP cam and include that cookie in header of your next HttpWebRequest. Otherways it will always try to redirect you to "index.html".
Here is how you can do it...
BitmapObject is a class that serves as a container for Jpeg image, current date and eventual error text. Once a connection is established it will pool an image every 200 ms. Same should be applicable for continuous image stream obtained through "serverpush".
public void Connect()
{
try
{
request = (HttpWebRequest)WebRequest.Create("Http://192.168.0.2/index.html");
request.Credentials = new NetworkCredential(UserName,Password);
request.Method = "GET";
response = (HttpWebResponse)request.GetResponse();
WebHeaderCollection headers = response.Headers;
Cookie = headers["Set-Cookie"];//get cookie
GetImage(null);
}
catch (Exception ex)
{
BitmapObject bitmap = new BitmapObject(Properties.Resources.Off,DateTime.Now);
bitmap.Error = ex.Message;
onImageReady(bitmap);
}
}
private Stream GetStream()
{
Stream s = null;
try
{
request = (HttpWebRequest)WebRequest.Create("http://192.168.0.2/liveimg.cgi");
if (!Anonimous)
request.Credentials = new NetworkCredential(UserName, Password);
request.Method = "GET";
request.KeepAlive = KeepAlive;
request.Headers.Add(HttpRequestHeader.Cookie, Cookie);
response = (HttpWebResponse)request.GetResponse();
s = response.GetResponseStream();
}
catch (Exception ex)
{
BitmapObject bitmap = new BitmapObject(Properties.Resources.Off,DateTime.Now);
bitmap.Error = ex.Message;
onImageReady(bitmap);
}
return s;
}
public void GetImage(Object o)
{
BitmapObject bitmap = null;
stream = GetStream();
DateTime CurrTime = DateTime.Now;
try
{
bitmap = new BitmapObject(new Bitmap(stream),CurrTime);
if (timer == null)//System.Threading.Timer
timer = new Timer(new TimerCallback(GetImage), null, 200, 200);
}
catch (Exception ex)
{
bitmap = new BitmapObject(Properties.Resources.Off, CurrTime);
bitmap.Error = ex.Message;
}
finally
{
stream.Flush();
stream.Close();
}
onImageReady(bitmap);
}

If you are using a standard web server, it will never push anything to you - your client will have to periodically pull from it instead.
To really get server push data you have to build such server yourself.