I want to run about 10,000 concurrent requests, using .net HttpWebRequest, not all of them going to the same host and some of them go through a pool of proxies.
I'm currently using Threads, which works fine up to 1000 concurrent, but then my CPU load 100%. How to fix it?
public string HttpGet(string url)
{
try {
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Timeout = 20000;
request.CookieContainer = Cookie;
request.AutomaticDecompression = DecompressionMethods.GZip;
request.KeepAlive = true;
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string tmp = reader.ReadToEnd();
reader.Close();
dataStream.Close();
response.Close();
return tmp;
} catch (Exception ex) {
return "";
}
}
I send request each time, so make 1000 requests for 1 minute.
Use async tasks instead of threads. I/O operations don't need thread tho wait for them, but OS spends time on switching between them.
Use using blocks instead of Close call to properly dispose resources even if request fails.
Don't forget to set correct maximum limit for concurrent requests.
catch (Exception ex) is an antipattern.
Related
Hi all since i am new here i hope this is the right place.
i am trying out HttpWebRequest to check out the status code (in this case response) from different websites.
On ok the response is very fast and no form freeze.
On a connect failure the form freezes and take's around 5 seconds
bypassing the freezing can be done with a background worker.
But is there a way to say, there is no response in 1 second, forget this one and go to the next.
try
{
string url;
url = (LoopUrlFromListboxOrStream ) ; //example
// Creates an HttpWebRequest for the specified URL.
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(url);
// Sends the HttpWebRequest and waits for a response.
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
if (myHttpWebResponse.StatusCode == HttpStatusCode.OK)
label1.Text = (myHttpWebResponse.StatusDescription);
// Releases the resources of the response.
myHttpWebResponse.Close();
}
catch (WebException ex)
{
label2.Text = (ex.Status.ToString());
}
catch (Exception ex)
{
label2.Text = (ex.Message.ToString());
}
Set HttpWebRequest.Timeout property to a low enough value that suits your needs.
In you case
myHttpWebRequest.Timeout = 1;
Before doing GetResponse()
I am sending a large number of simultaneous requests to a particular web service with different data. To achieve this, I have created a number of threads(around 50 in number). Total number of requests per minute may increase up to 10000.
The application in the form of a windows service runs fine for a few minutes and then a operation time out error is encountered.
I have tried the usual suspects such as increasing DefaultConnectionLimit, closing the web response object. Since the requests do not take much time on server, I have also set the request Timeout and ReadWriteTimeout to 5 seconds.
Below is the code snippet which is called repeatedly by different threads.
// Below line is executed at the start of application
ServicePointManager.DefaultConnectionLimit = 15000;
// Below code is executed at repeatedly by different threads
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Host = hostName;
request.Proxy = null;
request.UserAgent = "Windows Service";
byte[] bytes = new byte[0];
if (body != null)
{
bytes = System.Text.Encoding.ASCII.GetBytes(body);
request.ContentType = "text/xml; encoding='utf-8'";
request.ContentLength = bytes.Length;
}
request.Method = "POST";
request.Timeout = 5000;
request.ReadWriteTimeout = 5000;
request.Headers["Authorization"] = "Basic " + Convert.ToBase64String(Encoding.Default.GetBytes(username + ":" + password));
request.CookieContainer = this.cookieContainer;
if (body != null)
{
Stream requestStream = request.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Close();
}
HttpWebResponse httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
responseText = streamReader.ReadToEnd();
}
httpResponse.Close();
ServicePointManager.DefaultConnectionLimit limits the number of outgoing web requests to a given server. The default is generally 2 or 10.
If you are making 50 parallel calls to that web service, you should set ServicePointManager.DefaultConnectionLimit (at app startup) to a larger number (e.g. 40-50).
Additionally, you are not calling Close or Dispose on request. You should do this, or let using take care of it for you.
I'm using a proxy service which allows me to get a new ip on each request I make trough their proxy system. It's working fine, but I need to set the keepalive to false in my HttpWebRequest, else c# is going to use the connection which was created before again, which leads into using the same connection/proxy over and over again. I would love to send 3 requests over one "session/proxy" and then close it so the next HttpWebRequest won't use the same connection again, but I'm not too sure how I should realize that.
The code shown below will open a new connection to the proxy system each time I call the function:
Uri site = new Uri("https://randomsitehere/");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(site);
request.CookieContainer = cookies;
request.Proxy = new WebProxy(proxyURI, true, null, credentials);
request.UserAgent = "";
request.KeepAlive = false;
request.Timeout = 5000;
request.Method = "GET";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) {
Stream dataStream = response.GetResponseStream();
StreamReader streamreader = new StreamReader(dataStream, Encoding.UTF8);
string responsex = streamreader.ReadToEnd();
dataStream.Close();
response.Close();
dataStream.Close();
string tokens = responsex.Split(new[] { #"""ip"": """ }, StringSplitOptions.None)[1].Split('"')[0];
Console.WriteLine("Thread " + i +": "+tokens);
}
What I'm trying to archive looks like this:
request number 1 with a new proxy
request number 2 with the proxy used above / connection
request number 3 with the same proxy used above / connection
close the connection to the proxy
start again from the beginning
Is there any way to realize that?
I am sending a large number of simultaneous requests to a particular web service with different data. To achieve this, I have created a number of threads(around 50 in number). Total number of requests per minute may increase up to 10000.
The application in the form of a windows service runs fine for a few minutes and then a operation time out error is encountered.
I have tried the usual suspects such as increasing DefaultConnectionLimit, closing the web response object. Since the requests do not take much time on server, I have also set the request Timeout and ReadWriteTimeout to 5 seconds.
Below is the code snippet which is called repeatedly by different threads.
// Below line is executed at the start of application
ServicePointManager.DefaultConnectionLimit = 15000;
// Below code is executed at repeatedly by different threads
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Host = hostName;
request.Proxy = null;
request.UserAgent = "Windows Service";
byte[] bytes = new byte[0];
if (body != null)
{
bytes = System.Text.Encoding.ASCII.GetBytes(body);
request.ContentType = "text/xml; encoding='utf-8'";
request.ContentLength = bytes.Length;
}
request.Method = "POST";
request.Timeout = 5000;
request.ReadWriteTimeout = 5000;
request.Headers["Authorization"] = "Basic " + Convert.ToBase64String(Encoding.Default.GetBytes(username + ":" + password));
request.CookieContainer = this.cookieContainer;
if (body != null)
{
Stream requestStream = request.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Close();
}
HttpWebResponse httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
responseText = streamReader.ReadToEnd();
}
httpResponse.Close();
ServicePointManager.DefaultConnectionLimit limits the number of outgoing web requests to a given server. The default is generally 2 or 10.
If you are making 50 parallel calls to that web service, you should set ServicePointManager.DefaultConnectionLimit (at app startup) to a larger number (e.g. 40-50).
Additionally, you are not calling Close or Dispose on request. You should do this, or let using take care of it for you.
I'm trying to write an application that gets stock quotes from Google Finance and graphs them, in C#. My program uses the HttpWebRequest class to request the source code for the web page containing the stock quote, about every thirty seconds. After getting the source, I then extract the quotes and graph them.
My issue is that when the program goes about requesting every 30 seconds, about 50% of the time I invoke "objWebReq.GetResponse()", I get timeout exceptions! I checked my internet connection, and I'm absolutely sure that I'm connected when the timeout exceptions occur. Is there something I'm not doing correctly, that would cause Google to occasionally reject my request? Here is the portion of my program that makes the request for the source code.
//Create and configure object for request
HttpWebRequest objWebReq = (HttpWebRequest)WebRequest.Create(url); //Valid URL
objWebReq.Timeout = MSWAIT; //Set to 10 seconds
objWebReq.Method = "GET";
objWebReq.ContentType = "text/html";
objWebReq.KeepAlive = false;
objWebReq.Referer = "http://www.google.com/finance";
objWebReq.UserAgent =
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115
Firefox/3.6";
StreamReader objStream = null;
//Create Object for the response.
try
{
HttpWebResponse objWebResp = (HttpWebResponse)objWebReq.GetResponse();
objStream = new StreamReader(objWebResp.GetResponseStream(),
System.Text.Encoding.ASCII);
}
catch
{
//Do Nothing
}
return objStream;
}
You're not disposing of the web response, which can lead to timeouts while the connection pool waits for a connection. I would suggest that you read the complete response within your method and return it as a string, so you can close the connection:
using (HttpWebResponse response = (HttpWebResponse)objWebReq.GetResponse())
{
// Are you sure it's *really* ASCII, rather than (say) UTF-8?
using (TextReader reader = new StreamReader(response.GetResponseStream(),
Encoding.ASCII))
{
return reader.ReadToEnd();
}
}