Hy,
I am using HttpWebRequest in 10 concurent threads to download a list of images. I sorted the images after the hostName so each of this threads are getting an image from the same Hostname.
HttpWebRequest myReq = (HttpWebRequest)WebRequest.Create(url);
myReq.ServicePoint.ConnectionLimit = 10;
myReq.Timeout = 2000;
myReq.KeepAlive = true;
HttpWebResponse myResp = (HttpWebResponse )myReq.GetResponse();
After the program is running for a while I keep getting timeout exception.
My thoughts are that I get exception because maybe the Host server has some limitation regarding the concurent connections from the same user.
So how is a connection reused in .net ?
In my program each thread is creating a new connection to a hostname, or is reusing the existing one because of the KeepAlive property ??
It seems that the problem was some servers that were using http/1.0.
HttpWebRequest is using 100 Continue behaviour but the server doesn't support it.
So i changed the property System.Net.ServicePointManager.Expect100Continue to false and then everything worked fine.
Related
I currently have a python server running that handles many different kinds of requests and I'm trying to do this using C#. My code is as follows:
try
{
ServicePointManager.DefaultConnectionLimit = 10;
System.Net.ServicePointManager.Expect100Continue = false;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Proxy = null;
request.Method = "GET";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
response.Close();
}
}
catch (WebException e)
{
Console.WriteLine(e);
}
My first get request is almost instant but after that, the time it takes for one request to go through is almost 30 seconds to 1 min. I have researched everywhere online and tried to change the settings to make it run faster but it can't seem to work. Are there anymore settings that I could change to make it faster?
By using my psychic debugging skills I guess your server only accepts one connection at the time. You do not close your request, so you connection is kept alive. Keep alive attribute. The server will accept new connection when you current one is closed, which is default 100000ms or the server timeout. In your case I guess 30 to 60 seconds. You can start by setting the keepalive attribe to false.
We have a ASP.NET web application (MVC3) that makes calls out to another web server using HttpWebRequest.
Anyhow, when the remote server goes down, we start getting timeout errors, which we would expect.
However, then when the remote server comes back up, we continue to get timeout errors. Refreshing the application pool solves the issue, but we really don't want to have to restart the app pool every time.
Is there pooling that is going on that could be storing the bad timed-out connections? My expectation would be that if the connection throws an error, it would get disposed, and then we'd get a new connection the next time, but that doesn't seem to be the case.
Here's what our code looks like:
var request = (HttpWebRequest)HttpWebRequest.Create(url);
int timeout = GetTimeout();
request.ReadWriteTimeout = timeout;
WebRequest.DefaultWebProxy = null;
var asyncResult = request.BeginGetResponse(null, null);
bool complete = asyncResult.AsyncWaitHandle.WaitOne(timeout);
if (!complete)
{
ThrowTimeoutError(url, timeout);
}
using (var webResponse = request.EndGetResponse(asyncResult))
{
using (var responseStream = webResponse.GetResponseStream())
{
using (var responseStreamReader = new StreamReader(responseStream))
{
responseXml = responseStreamReader.ReadToEnd();
}
}
}
Have you seen/tried the CloseConnectionGroup method of the ServicePoint class?
From the MSDN documentation:
Connection groups associate a set of requests with a particular
connection or set of connections. This method removes and closes all
connections that belong to the specified connection group.
Calling this method may reset your connections without having to restart the pool.
I have an program where I need to create some large number of folders to an external sharepoint site (external meaning I can't use the sharepoint object model). Web requests work well for this, but simply doing them one at a time (send request, wait for response, repeat) is rather slow. I decided to multithread the requests, to try and speed it up. The program has sped up considerably, but after some amount of time (between 1-2 minutes or so), concurrency exceptions start getting thrown.
Code is below, is this the best way to go about this?
Semaphore Lock = new Semaphore(10, 10);
List<string> folderPathList = new List<string>();
//folderPathList populated
foreach (string folderPath in folderPathList)
{
Lock.WaitOne();
new Thread(delegate()
{
WebRequest request = WebRequest.Create(folderPath);
request.Credentials = DefaultCredentials;
request.Method = "MKCOL";
WebResponse response = request.GetResponse();
response.Close();
Lock.Release();
}).Start();
}
for(int i = 1;i <= 10;i++)
{
Lock.WaitOne();
}
The exception is something along the lines of
Unhandled Exception: System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: Only one useage of each socket address is normally permitted 192.0.0.1:81
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddre
ss socketAddress)
at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state,
IAsyncResult asyncResult, Int32 timeout, Exception& exception)
You might create too many connections, thus using up all the local ports you can use.
There's a timeout period for when a port can be reused after you close it.
WebRequest hides all the low level socket handling for you, but I am guessing it eventually runs out of ports, or tries to (re)bind to a socket already in a TIME_WAIT state.
You should make sure you read the response stream, even if you don't care about the response. This should help not producing too many lingering connections.
WebResponse response = request.GetResponse();
new StreamReader(response.GetResponseStream()).ReadToEnd();
I'll paste some relevant info from here:
When a connection is closed, on the side that is closing the connection the 5 tuple
{ Protocol, Local IP, Local Port, Remote IP, Remote Port} goes into a TIME_WAIT state for 240 seconds by default.
In this case, the protocol is fixed - TCP
the local IP, remote IP and remote PORT are also typically fixed. So the variable is the local port.
What happens is that when you don't bind, a port in the range 1024-5000 is used.
So roughly you have 4000 ports. If you use all of them in 4 minutes - meaning roughly you
make 16 web service calls per second for 4 minutes you will exhaust all the ports. That is the cause of this exception.
OK now how can this be fixed?
One of the ways is to increase the dynamic port range. The max by default is 5000. You can set this up to 65534.
HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\MaxUserPort is the key to use.
The second thing you can do is once the connection does get into an TIME_WAIT state you can reduce the time it is
in that state, Default is 4 minutes, but you can set this to 30 seconds
HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\TCPTimedWaitDelay is the key to use.
Set this to 30 seconds
You're not closing the webrequest which might cause the connection to be open unecessarily long. This sounds like a perfect job for Parallel.Net's Parallel.Foreach, just be sure to indicate how many threads you want it running on
ParallelOptions parallelOptions = new ParallelOptions();
parallelOptions.MaxDegreeOfParallelism = 10;
Parallel.ForEach(folderPathList, parallelOptions, folderPathList =>
{
using(WebRequest request = WebRequest.Create(folderPath))
{
request.Credentials = DefaultCredentials;
request.Method = "MKCOL";
GetResponse request = WebRequest.Create(folderPath);
request.Credentials = DefaultCredentials;
request.Method = "MKCOL";
using (WebResponse response = request.GetResponse());
}
});
Another thing to keep in mind is maxConnections, be sure to set it in your app.config:
<configuration>
<system.net>
<connectionManagement>
<add address = "*" maxconnection = "100" />
</connectionManagement>
</system.net>
</configuration>
Of couse in a real-world scenario you would have to add try-catch to and retrying connections that might time out leading to more complicated code
For this kind of IO intensive tasks, asynchronous programming model is very useful. However, it is a little hard to use in C#.C# also has language level support for async now, you can try the CTP release.
try this
folderPathList.ToList().ForEach(p =>
{
ThreadPool.QueueUserWorkItem((o) =>
{
WebRequest request = WebRequest.Create(p);
request.Credentials = DefaultCredentials;
request.Method = "MKCOL";
WebResponse response = request.GetResponse();
response.Close();
});
EDIT - different webrequest approach
folderPathList.ToList().ForEach(p =>
{
ThreadPool.QueueUserWorkItem((o) =>
{
using (WebClient client = new WebClient())
{
client.Credentials = DefaultCredentials;
client.UploadString(p, "MKCOL", "");
}
});
});
If the website isn't responding after one second or so, it's probably safe to assume that it's a bad link and that I should move on to my next link in a set of "possible links."
How do I tell the WebClient to stop attempting to download after some predetermined amount of time?
I suppose I could use a thread, but if it's taking longer than one second to download, that's ok. I just want to make sure that it's connecting with the site.
Perhaps I should modify the WebClient headers, but I don't see what I should modify.
Perhaps I should use some other class in the System.Net namespace?
If you use the System.Net.WebRequest class you can set the Timeout property to be short and handle timeout exceptions.
try{
var request = WebRequest.Create("http://www.contoso.com");
request.Timeout = 5000; //set the timeout to 5 seconds
request.Method = "GET";
var response = request.GetResponse();
}
catch(WebException webEx){
//there was an error, likely a timeout, try another link here
}
I have an program where I need to create some large number of folders to an external sharepoint site (external meaning I can't use the sharepoint object model). Web requests work well for this, but simply doing them one at a time (send request, wait for response, repeat) is rather slow. I decided to multithread the requests, to try and speed it up. The program has sped up considerably, but after some amount of time (between 1-2 minutes or so), concurrency exceptions start getting thrown.
Code is below, is this the best way to go about this?
Semaphore Lock = new Semaphore(10, 10);
List<string> folderPathList = new List<string>();
//folderPathList populated
foreach (string folderPath in folderPathList)
{
Lock.WaitOne();
new Thread(delegate()
{
WebRequest request = WebRequest.Create(folderPath);
request.Credentials = DefaultCredentials;
request.Method = "MKCOL";
WebResponse response = request.GetResponse();
response.Close();
Lock.Release();
}).Start();
}
for(int i = 1;i <= 10;i++)
{
Lock.WaitOne();
}
The exception is something along the lines of
Unhandled Exception: System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: Only one useage of each socket address is normally permitted 192.0.0.1:81
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddre
ss socketAddress)
at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state,
IAsyncResult asyncResult, Int32 timeout, Exception& exception)
You might create too many connections, thus using up all the local ports you can use.
There's a timeout period for when a port can be reused after you close it.
WebRequest hides all the low level socket handling for you, but I am guessing it eventually runs out of ports, or tries to (re)bind to a socket already in a TIME_WAIT state.
You should make sure you read the response stream, even if you don't care about the response. This should help not producing too many lingering connections.
WebResponse response = request.GetResponse();
new StreamReader(response.GetResponseStream()).ReadToEnd();
I'll paste some relevant info from here:
When a connection is closed, on the side that is closing the connection the 5 tuple
{ Protocol, Local IP, Local Port, Remote IP, Remote Port} goes into a TIME_WAIT state for 240 seconds by default.
In this case, the protocol is fixed - TCP
the local IP, remote IP and remote PORT are also typically fixed. So the variable is the local port.
What happens is that when you don't bind, a port in the range 1024-5000 is used.
So roughly you have 4000 ports. If you use all of them in 4 minutes - meaning roughly you
make 16 web service calls per second for 4 minutes you will exhaust all the ports. That is the cause of this exception.
OK now how can this be fixed?
One of the ways is to increase the dynamic port range. The max by default is 5000. You can set this up to 65534.
HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\MaxUserPort is the key to use.
The second thing you can do is once the connection does get into an TIME_WAIT state you can reduce the time it is
in that state, Default is 4 minutes, but you can set this to 30 seconds
HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\TCPTimedWaitDelay is the key to use.
Set this to 30 seconds
You're not closing the webrequest which might cause the connection to be open unecessarily long. This sounds like a perfect job for Parallel.Net's Parallel.Foreach, just be sure to indicate how many threads you want it running on
ParallelOptions parallelOptions = new ParallelOptions();
parallelOptions.MaxDegreeOfParallelism = 10;
Parallel.ForEach(folderPathList, parallelOptions, folderPathList =>
{
using(WebRequest request = WebRequest.Create(folderPath))
{
request.Credentials = DefaultCredentials;
request.Method = "MKCOL";
GetResponse request = WebRequest.Create(folderPath);
request.Credentials = DefaultCredentials;
request.Method = "MKCOL";
using (WebResponse response = request.GetResponse());
}
});
Another thing to keep in mind is maxConnections, be sure to set it in your app.config:
<configuration>
<system.net>
<connectionManagement>
<add address = "*" maxconnection = "100" />
</connectionManagement>
</system.net>
</configuration>
Of couse in a real-world scenario you would have to add try-catch to and retrying connections that might time out leading to more complicated code
For this kind of IO intensive tasks, asynchronous programming model is very useful. However, it is a little hard to use in C#.C# also has language level support for async now, you can try the CTP release.
try this
folderPathList.ToList().ForEach(p =>
{
ThreadPool.QueueUserWorkItem((o) =>
{
WebRequest request = WebRequest.Create(p);
request.Credentials = DefaultCredentials;
request.Method = "MKCOL";
WebResponse response = request.GetResponse();
response.Close();
});
EDIT - different webrequest approach
folderPathList.ToList().ForEach(p =>
{
ThreadPool.QueueUserWorkItem((o) =>
{
using (WebClient client = new WebClient())
{
client.Credentials = DefaultCredentials;
client.UploadString(p, "MKCOL", "");
}
});
});