I am sending a large number of simultaneous requests to a particular web service with different data. To achieve this, I have created a number of threads(around 50 in number). Total number of requests per minute may increase up to 10000.
The application in the form of a windows service runs fine for a few minutes and then a operation time out error is encountered.
I have tried the usual suspects such as increasing DefaultConnectionLimit, closing the web response object. Since the requests do not take much time on server, I have also set the request Timeout and ReadWriteTimeout to 5 seconds.
Below is the code snippet which is called repeatedly by different threads.
// Below line is executed at the start of application
ServicePointManager.DefaultConnectionLimit = 15000;
// Below code is executed at repeatedly by different threads
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Host = hostName;
request.Proxy = null;
request.UserAgent = "Windows Service";
byte[] bytes = new byte[0];
if (body != null)
{
bytes = System.Text.Encoding.ASCII.GetBytes(body);
request.ContentType = "text/xml; encoding='utf-8'";
request.ContentLength = bytes.Length;
}
request.Method = "POST";
request.Timeout = 5000;
request.ReadWriteTimeout = 5000;
request.Headers["Authorization"] = "Basic " + Convert.ToBase64String(Encoding.Default.GetBytes(username + ":" + password));
request.CookieContainer = this.cookieContainer;
if (body != null)
{
Stream requestStream = request.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Close();
}
HttpWebResponse httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
responseText = streamReader.ReadToEnd();
}
httpResponse.Close();
ServicePointManager.DefaultConnectionLimit limits the number of outgoing web requests to a given server. The default is generally 2 or 10.
If you are making 50 parallel calls to that web service, you should set ServicePointManager.DefaultConnectionLimit (at app startup) to a larger number (e.g. 40-50).
Additionally, you are not calling Close or Dispose on request. You should do this, or let using take care of it for you.
Related
I want to run about 10,000 concurrent requests, using .net HttpWebRequest, not all of them going to the same host and some of them go through a pool of proxies.
I'm currently using Threads, which works fine up to 1000 concurrent, but then my CPU load 100%. How to fix it?
public string HttpGet(string url)
{
try {
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Timeout = 20000;
request.CookieContainer = Cookie;
request.AutomaticDecompression = DecompressionMethods.GZip;
request.KeepAlive = true;
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string tmp = reader.ReadToEnd();
reader.Close();
dataStream.Close();
response.Close();
return tmp;
} catch (Exception ex) {
return "";
}
}
I send request each time, so make 1000 requests for 1 minute.
Use async tasks instead of threads. I/O operations don't need thread tho wait for them, but OS spends time on switching between them.
Use using blocks instead of Close call to properly dispose resources even if request fails.
Don't forget to set correct maximum limit for concurrent requests.
catch (Exception ex) is an antipattern.
I'm using a proxy service which allows me to get a new ip on each request I make trough their proxy system. It's working fine, but I need to set the keepalive to false in my HttpWebRequest, else c# is going to use the connection which was created before again, which leads into using the same connection/proxy over and over again. I would love to send 3 requests over one "session/proxy" and then close it so the next HttpWebRequest won't use the same connection again, but I'm not too sure how I should realize that.
The code shown below will open a new connection to the proxy system each time I call the function:
Uri site = new Uri("https://randomsitehere/");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(site);
request.CookieContainer = cookies;
request.Proxy = new WebProxy(proxyURI, true, null, credentials);
request.UserAgent = "";
request.KeepAlive = false;
request.Timeout = 5000;
request.Method = "GET";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) {
Stream dataStream = response.GetResponseStream();
StreamReader streamreader = new StreamReader(dataStream, Encoding.UTF8);
string responsex = streamreader.ReadToEnd();
dataStream.Close();
response.Close();
dataStream.Close();
string tokens = responsex.Split(new[] { #"""ip"": """ }, StringSplitOptions.None)[1].Split('"')[0];
Console.WriteLine("Thread " + i +": "+tokens);
}
What I'm trying to archive looks like this:
request number 1 with a new proxy
request number 2 with the proxy used above / connection
request number 3 with the same proxy used above / connection
close the connection to the proxy
start again from the beginning
Is there any way to realize that?
I am sending a large number of simultaneous requests to a particular web service with different data. To achieve this, I have created a number of threads(around 50 in number). Total number of requests per minute may increase up to 10000.
The application in the form of a windows service runs fine for a few minutes and then a operation time out error is encountered.
I have tried the usual suspects such as increasing DefaultConnectionLimit, closing the web response object. Since the requests do not take much time on server, I have also set the request Timeout and ReadWriteTimeout to 5 seconds.
Below is the code snippet which is called repeatedly by different threads.
// Below line is executed at the start of application
ServicePointManager.DefaultConnectionLimit = 15000;
// Below code is executed at repeatedly by different threads
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Host = hostName;
request.Proxy = null;
request.UserAgent = "Windows Service";
byte[] bytes = new byte[0];
if (body != null)
{
bytes = System.Text.Encoding.ASCII.GetBytes(body);
request.ContentType = "text/xml; encoding='utf-8'";
request.ContentLength = bytes.Length;
}
request.Method = "POST";
request.Timeout = 5000;
request.ReadWriteTimeout = 5000;
request.Headers["Authorization"] = "Basic " + Convert.ToBase64String(Encoding.Default.GetBytes(username + ":" + password));
request.CookieContainer = this.cookieContainer;
if (body != null)
{
Stream requestStream = request.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Close();
}
HttpWebResponse httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
responseText = streamReader.ReadToEnd();
}
httpResponse.Close();
ServicePointManager.DefaultConnectionLimit limits the number of outgoing web requests to a given server. The default is generally 2 or 10.
If you are making 50 parallel calls to that web service, you should set ServicePointManager.DefaultConnectionLimit (at app startup) to a larger number (e.g. 40-50).
Additionally, you are not calling Close or Dispose on request. You should do this, or let using take care of it for you.
I am writing an application to send text messages through HTTP posts to a Slooce Tech API. Because the application will have to send a high volume of text messages, I'm trying to optimize its speed.
The second piece of code below is the method that I am currently using to send the posts. I wrote the first piece of code and left out the HTTPWebResponse to try to make it faster.
The problem is that the new method is actually slower and rather than taking .25 seconds to execute, it takes a second or more and sometimes will get stuck.
Does anyone know why it would do that or any other tips for improving the speed of this application? I have added Request.Proxy=null and that speeds it up a little bit.
Thanks.
The modified code is:
public void QuickSend()
{
XML = "<message id=\"" + lMessageID + "\"><partnerpassword>" + PartnerPassword + "</partnerpassword><content>" + sMessage + "</content></message>";
URL = "http://sloocetech.net:****/spi-war/spi/" + PartnerID + "/" + sRecipient + "/" + Keyword + "/messages/mt";
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create(URL);
RequestBytes = System.Text.Encoding.ASCII.GetBytes(XML);
Request.Method = "POST";
Request.ContentType = "text/xml;charset=utf-8";
Request.ContentLength = RequestBytes.Length;
RequestStream = Request.GetRequestStream();
RequestStream.Write(RequestBytes, 0, RequestBytes.Length);
RequestStream.Close();
}
And here is the original code:
public XDocument SendSMS()
{
XML = "<message id=\""+ lMessageID +"\"><partnerpassword>" + PartnerPassword + "</partnerpassword><content>" + sMessage + "</content></message>";
URL = "http://sloocetech.net:****/spi-war/spi/" + PartnerID + "/" + sRecipient + "/" + Keyword + "/messages/mt";
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create(URL);
RequestBytes = System.Text.Encoding.ASCII.GetBytes(XML);
Request.Method = "POST";
Request.ContentType = "text/xml;charset=utf-8";
Request.ContentLength = RequestBytes.Length;
RequestStream = Request.GetRequestStream();
RequestStream.Write(RequestBytes, 0, RequestBytes.Length);
RequestStream.Close();
HttpWebResponse Resp = (HttpWebResponse)Request.GetResponse();
oReader = new StreamReader(Resp.GetResponseStream(), System.Text.Encoding.Default);
string backstr = oReader.ReadToEnd();
oReader.Close();
Resp.Close();
Doc = XDocument.Parse(backstr);
return Doc;
}
First of all, I'd be very skeptical of any claims that you're going to see massive improvements because you're crafting your HttpWebRequest in a special way. The bottleneck on single threaded requests like yours is going to be network latency as well as the response time of the server. (Perhaps they're doing a lot of server-side processing before responding to your request).
You're making a blocking request, which means your CPU is doing nothing while it waits for a response.
If you want to multithread your application, you could do something like the following:
var tasks = new Task[10];
for (int i = 0; i < 10; i++)
{
tasks[i] = Task.Factory.StartNew(() =>
{
int messages_sent_by_one_task = 0;
while(messages_sent_by_one_task < 10)
{
QuickSend();
messages_sent_by_one_task++;
}
});
}
while (tasks.Any(t => !t.IsCompleted)) { } //wait for tasks to complete
This will spawn 10 tasks that will each send 10 messages. If one response is taking a long time, the other 9 threads will continue chugging along happily.
I believe you could probably improve on this is you were to incorporate asynchronous requests and HTTPClient, so each of your 10 threads never blocked. However, I don't feel qualified to give an example as I've never tried this.
You might be tempted to crank the number of threads up to some ungodly number, but avoid the temptation. The overhead of creating and managing threads will soon catch up with you. I don't know what the ideal number is, but you're welcome to experiment.
I am trying to send contents of 1GB text file over the network. I modified the suggested code for basic authentication and kept it as follows :
WRequest = (HttpWebRequest)HttpWebRequest.Create(URL);
WRequest.Credentials = Credentials;
WRequest.PreAuthenticate = true;
WRequest.ContentType = "text/plain";
WRequest.Method = "POST";
WRequest.AllowWriteStreamBuffering = false;
WRequest.Timeout = 10000;
FileStream ReadIn = new FileStream(filename, FileMode.Open, FileAccess.Read);
ReadIn.Seek(0, SeekOrigin.Begin);
WRequest.ContentLength = ReadIn.Length;
Byte[] FileData = new Byte[ReadIn.Length];
int DataRead = 0;
Stream tempStream = WRequest.GetRequestStream();
do
{
DataRead = ReadIn.Read(FileData, 0, 2048);
if (DataRead > 0)
{
tempStream.Write(FileData, 0, DataRead);
Array.Clear(FileData, 0, 2048);
}
} while (DataRead > 0);
// The response
WResponse = (HttpWebResponse)WRequest.GetResponse();
However, now it gives me System.Net.ProtocolViolationException error : "You must write ContentLength bytes to the request stream before calling [Begin]GetResponse". I checked HttpWebRequest.BeginGetRequestResponse ... and found from debugging that the contentlength for WRequest is not -1. What else could be going wrong ? How should I get the response ?
Update :
The code which worked for small files is as followed :
WebRequest request = WebRequest.Create(url);
request.Method = "POST";
request.Credentials = Credentials;
using (StreamReader reader = new StreamReader(filename))
{
postData = reader.ReadToEnd();
}
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
request.ContentType = "text/plain";
request.ContentLength = byteArray.Length;
Stream dataStream = request.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();
// The response
WebResponse response = request.GetResponse();
Console.WriteLine(((HttpWebResponse)response).StatusDescription);
dataStream = response.GetResponseStream();
using (StreamReader reader = new StreamReader(dataStream))
{
responseFromServer = reader.ReadToEnd();
}
dataStream.Close();
response.Close();
The article you referenced says
If the Microsoft Internet Information Services (IIS) Web server is configured to use Basic authentication, and you must set the HttpWebRequest.AllowWriteStreamBuffering property to false, you must send a HEAD request to pre-authenticate the connection before you send the POST or PUT request.
EDIT - now with more clarification!
To restate the article, if you want to send a large file to a destination which requires basic authentication, you'll need to issue two separate requests. The key here is that you are setting PreAuthenticate = true. Read the statement literally -- by setting the property to true, you are saying that you will authenticate any requests that you make before you actually attempt them! The framework doesn't know how you want to accomplish this pre-authentication, so you need to perform that action yourself, by sending a HEAD request to the destination. Think of the HEAD HTTP method as being a prologue to the actual request - it describes (or requests information about) a particular resource.
So the process goes like this:
Make a HEAD request to http://someurl/aresource containing the credentials you want to use when making future requests from this client to that server for the listed resource
The server will respond (ideally) with "OK - you may proceed. You're authenticated"
The server immediately regrets its' decision to allow the operation as it finds itself saving a very large file :-)
I don't see you making that HEAD request anywhere in the code you posted - if it's not already there, add this at the beginning of your code (snipped from the sample article ref in OP):
//preAuth the request
// You can add logic so that you only pre-authenticate the very first request.
// You should not have to pre-authenticate each request.
WRequest = (HttpWebRequest)HttpWebRequest.Create(URL);
// Set the username and the password.
WRequest.Credentials = new NetworkCredential(user, password);
WRequest.PreAuthenticate = true;
WRequest.UserAgent = "Upload Test";
WRequest.Method = "HEAD";
WRequest.Timeout = 10000;
WResponse = (HttpWebResponse)WRequest.GetResponse();
WResponse.Close();
// Make the real request.