I'm writing an app that needs to do a lot of parallel httpwebrequests to bookmaker site from different proxies. The reason why i'm using different proxies is that bookmaker can ban ip if there are many requests from it. My goal is to get freshest site content as fast as possible.
Here is my code with all settings that I have:
ServicePointManager.DefaultConnectionLimit = 1000;
ServicePointManager.Expect100Continue = false;
ServicePointManager.UseNagleAlgorithm = false;
for (var i = 0; i < proxyCollection.Count; i++)
{
var proxyLocal = proxyCollection[i];
var iLocal = i;
Task.Run(async () =>
{
var httpWebRequest = (HttpWebRequest) WebRequest.Create(String.Format("https://bookmaker.com/page{0}", iLocal));
httpWebRequest.Proxy = proxyLocal;
httpWebRequest.PreAuthenticate = true;
httpWebRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
using (var httpWebResponse = (HttpWebResponse) await httpWebRequest.GetResponseAsync())
using (var responseStream = httpWebResponse.GetResponseStream())
using (var streamReader = new StreamReader(responseStream))
{
var stringContent = await streamReader.ReadToEndAsync();
//Here i'm processing new data. It works pretty fast, so this isn't a problem
ProcessStringContent(stringContent);
}
});
}
And this code works not so fast than I expected.
First problem is that in strange reason all requests don't start at one time. As I can see in task manager loading has two or more maximums. More over I have one realy fast proxy and proxyCollection contains it. But if I use await Task.Delay(5000) after code above, in some cases to the moment when 5000ms have passed the request with my fast proxy doesn't even start!
Second problem is that total time of all task execution is slow. I expected that if one request needs 200-300ms to execute, than 100 requests in parallel and async needs a little more time. But some times this "more" is 10-20 times. I have a suspicion, that something is wrong.
Third problem is that when I'm running this code it freezes UI (not full freeze, but UI lags). I read that WebRequest.Create is processing synchronously and can get some time (dns lookup, proxy settings e.t.c) and if I'm making a lot of requests they can simply fill all my threads (UI thread too) for creating WebRequests. But I tried to create requests to direct ip address (WebRequest.Create(String.Format("https://10.10.10.1/page{0}", iLocal)) - nothing changed, and i'm setting proxy (so auto detect proxy isn't needed), so I don't understand why can creating take so much time (and is problem with creating or maybe with smth else?).
Please can someone point me what i'm doing wrong? I'm lost in all ServicePointManager settings (all what I tried didn't help). Can .Net deal with such type of task? Or maybe I need to use nodejs for example for best performance?
P.S: Collection of proxies is not such big (50-100 different proxies).
Back to (Parallel) Basics: Don't Block Your Threads, Make Async I/O Work For You
Example:
Parallel.For(0, 10, delegate(int i) {
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(
new Uri("http://www.mysi.com"));
string dataToSend = "Data";
byte[] buffer = System.Text.Encoding.GetEncoding(1252).
GetBytes(dataToSend);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = buffer.Length;
request.Host = "www.mysite.com";
Stream requestStream = request.GetRequestStream();
requestStream.Write(buffer, 0, buffer.Length);
requestStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
});
Send multiple WebRequest in Parallel.For
Related
I am sending a large number of simultaneous requests to a particular web service with different data. To achieve this, I have created a number of threads(around 50 in number). Total number of requests per minute may increase up to 10000.
The application in the form of a windows service runs fine for a few minutes and then a operation time out error is encountered.
I have tried the usual suspects such as increasing DefaultConnectionLimit, closing the web response object. Since the requests do not take much time on server, I have also set the request Timeout and ReadWriteTimeout to 5 seconds.
Below is the code snippet which is called repeatedly by different threads.
// Below line is executed at the start of application
ServicePointManager.DefaultConnectionLimit = 15000;
// Below code is executed at repeatedly by different threads
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Host = hostName;
request.Proxy = null;
request.UserAgent = "Windows Service";
byte[] bytes = new byte[0];
if (body != null)
{
bytes = System.Text.Encoding.ASCII.GetBytes(body);
request.ContentType = "text/xml; encoding='utf-8'";
request.ContentLength = bytes.Length;
}
request.Method = "POST";
request.Timeout = 5000;
request.ReadWriteTimeout = 5000;
request.Headers["Authorization"] = "Basic " + Convert.ToBase64String(Encoding.Default.GetBytes(username + ":" + password));
request.CookieContainer = this.cookieContainer;
if (body != null)
{
Stream requestStream = request.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Close();
}
HttpWebResponse httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
responseText = streamReader.ReadToEnd();
}
httpResponse.Close();
ServicePointManager.DefaultConnectionLimit limits the number of outgoing web requests to a given server. The default is generally 2 or 10.
If you are making 50 parallel calls to that web service, you should set ServicePointManager.DefaultConnectionLimit (at app startup) to a larger number (e.g. 40-50).
Additionally, you are not calling Close or Dispose on request. You should do this, or let using take care of it for you.
I am sending a large number of simultaneous requests to a particular web service with different data. To achieve this, I have created a number of threads(around 50 in number). Total number of requests per minute may increase up to 10000.
The application in the form of a windows service runs fine for a few minutes and then a operation time out error is encountered.
I have tried the usual suspects such as increasing DefaultConnectionLimit, closing the web response object. Since the requests do not take much time on server, I have also set the request Timeout and ReadWriteTimeout to 5 seconds.
Below is the code snippet which is called repeatedly by different threads.
// Below line is executed at the start of application
ServicePointManager.DefaultConnectionLimit = 15000;
// Below code is executed at repeatedly by different threads
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Host = hostName;
request.Proxy = null;
request.UserAgent = "Windows Service";
byte[] bytes = new byte[0];
if (body != null)
{
bytes = System.Text.Encoding.ASCII.GetBytes(body);
request.ContentType = "text/xml; encoding='utf-8'";
request.ContentLength = bytes.Length;
}
request.Method = "POST";
request.Timeout = 5000;
request.ReadWriteTimeout = 5000;
request.Headers["Authorization"] = "Basic " + Convert.ToBase64String(Encoding.Default.GetBytes(username + ":" + password));
request.CookieContainer = this.cookieContainer;
if (body != null)
{
Stream requestStream = request.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Close();
}
HttpWebResponse httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
responseText = streamReader.ReadToEnd();
}
httpResponse.Close();
ServicePointManager.DefaultConnectionLimit limits the number of outgoing web requests to a given server. The default is generally 2 or 10.
If you are making 50 parallel calls to that web service, you should set ServicePointManager.DefaultConnectionLimit (at app startup) to a larger number (e.g. 40-50).
Additionally, you are not calling Close or Dispose on request. You should do this, or let using take care of it for you.
This question already has answers here:
How to perform a fast web request in C#
(4 answers)
Closed 7 years ago.
I've made an app that can access and control Onvif cameras which it does well enough. However this is my first time making any app that uses web requests like this (or at all) so I assume I'm probably using quite basic techniques.
The part of code I'm curious about is this:
Uri uri = new Uri(
String.Format("http://" + ipAddr + "/onvif/" + "{0}", Service));
WebRequest request = WebRequest.Create((uri));
request.Method = "POST";
byte[] b = Encoding.ASCII.GetBytes(PostData);
request.ContentLength = b.Length;
//request.Timeout = 1000;
Stream stream = request.GetRequestStream();
//Send Message
XmlDocument recData = new XmlDocument();
try
{
using (stream = request.GetRequestStream())
{
stream.Write(b, 0, b.Length);
}
//Store response
var response = (HttpWebResponse) request.GetResponse();
if (response.GetResponseStream() != null)
{
string responsestring = new
StreamReader(response.GetResponseStream())
.ReadToEnd();
recData.LoadXml(responsestring);
}
}
catch (SystemException e)
{
MessageBox.Show(e.Message);
}
return recData;
}
The code works fine, however using the writeline statements I've found that the first request takes about 400ms to complete whereas the subsequent ones only take between 10 - 20ms. Is there anything I can do to speed up the first request?
You're doing it just fine. The reason for the difference in time to complete may be due to HTTP Keep-Alive. By default, the same connection is reused for subsequent requests. So the first request has to establish the connection, which is probably why it takes longer. The rest of the requests use the same already-open connection.
Aside from potential network and server issues, the request itself matters. You can opt to reduce the size of the request or break it down and asynchronously load your files.
Web Servers out of the box will not take 400ms to complete a simple request.
I am writing an application to send text messages through HTTP posts to a Slooce Tech API. Because the application will have to send a high volume of text messages, I'm trying to optimize its speed.
The second piece of code below is the method that I am currently using to send the posts. I wrote the first piece of code and left out the HTTPWebResponse to try to make it faster.
The problem is that the new method is actually slower and rather than taking .25 seconds to execute, it takes a second or more and sometimes will get stuck.
Does anyone know why it would do that or any other tips for improving the speed of this application? I have added Request.Proxy=null and that speeds it up a little bit.
Thanks.
The modified code is:
public void QuickSend()
{
XML = "<message id=\"" + lMessageID + "\"><partnerpassword>" + PartnerPassword + "</partnerpassword><content>" + sMessage + "</content></message>";
URL = "http://sloocetech.net:****/spi-war/spi/" + PartnerID + "/" + sRecipient + "/" + Keyword + "/messages/mt";
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create(URL);
RequestBytes = System.Text.Encoding.ASCII.GetBytes(XML);
Request.Method = "POST";
Request.ContentType = "text/xml;charset=utf-8";
Request.ContentLength = RequestBytes.Length;
RequestStream = Request.GetRequestStream();
RequestStream.Write(RequestBytes, 0, RequestBytes.Length);
RequestStream.Close();
}
And here is the original code:
public XDocument SendSMS()
{
XML = "<message id=\""+ lMessageID +"\"><partnerpassword>" + PartnerPassword + "</partnerpassword><content>" + sMessage + "</content></message>";
URL = "http://sloocetech.net:****/spi-war/spi/" + PartnerID + "/" + sRecipient + "/" + Keyword + "/messages/mt";
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create(URL);
RequestBytes = System.Text.Encoding.ASCII.GetBytes(XML);
Request.Method = "POST";
Request.ContentType = "text/xml;charset=utf-8";
Request.ContentLength = RequestBytes.Length;
RequestStream = Request.GetRequestStream();
RequestStream.Write(RequestBytes, 0, RequestBytes.Length);
RequestStream.Close();
HttpWebResponse Resp = (HttpWebResponse)Request.GetResponse();
oReader = new StreamReader(Resp.GetResponseStream(), System.Text.Encoding.Default);
string backstr = oReader.ReadToEnd();
oReader.Close();
Resp.Close();
Doc = XDocument.Parse(backstr);
return Doc;
}
First of all, I'd be very skeptical of any claims that you're going to see massive improvements because you're crafting your HttpWebRequest in a special way. The bottleneck on single threaded requests like yours is going to be network latency as well as the response time of the server. (Perhaps they're doing a lot of server-side processing before responding to your request).
You're making a blocking request, which means your CPU is doing nothing while it waits for a response.
If you want to multithread your application, you could do something like the following:
var tasks = new Task[10];
for (int i = 0; i < 10; i++)
{
tasks[i] = Task.Factory.StartNew(() =>
{
int messages_sent_by_one_task = 0;
while(messages_sent_by_one_task < 10)
{
QuickSend();
messages_sent_by_one_task++;
}
});
}
while (tasks.Any(t => !t.IsCompleted)) { } //wait for tasks to complete
This will spawn 10 tasks that will each send 10 messages. If one response is taking a long time, the other 9 threads will continue chugging along happily.
I believe you could probably improve on this is you were to incorporate asynchronous requests and HTTPClient, so each of your 10 threads never blocked. However, I don't feel qualified to give an example as I've never tried this.
You might be tempted to crank the number of threads up to some ungodly number, but avoid the temptation. The overhead of creating and managing threads will soon catch up with you. I don't know what the ideal number is, but you're welcome to experiment.
I am having difficulty in consuming the reCaptcha Web Service using C#/.Net 3.5. Although I think the problem is with consuming web services in general.
String validate = String.Format("http://api-verify.recaptcha.net/verify?privatekey={0}&remoteip={1}&challenge={2}&response={3}", PrivateKey, UserIP, Challenge, Response);
WebClient serviceRequest = new WebClient();
serviceRequest.Headers.Add("ContentType","application/x-www-form-urlencoded")
String response = serviceRequest.DownloadString(new Uri(validate ));
It keeps telling me that the error is: nverify-params-incorrect. Which means:
The parameters to /verify were incorrect, make sure you are passing all the required parameters.
But it's correct. I am using the private key, the IP address (locally) is 127.0.0.1, and the challenge and response seem fine. However the error keeps occurring.
I am pretty sure this is a issue with how I am requesting the service as this is the first time I have actually used webservices and .Net.
I also tried this as it ensures the data is posted:
String queryString = String.Format("privatekey={0}&remoteip={1}&challenge={2}&response={3}",PrivateKey, UserIP, Challenge, Response);
String Validate = "http://api-verify.recaptcha.net/verify" + queryString;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri(Validate));
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = Validate.Length;
**HttpWebResponse captchaResponse = (HttpWebResponse)request.GetResponse();**
String response;
using (StreamReader reader = new StreamReader(captchaResponse.GetResponseStream()))
response = reader.ReadToEnd();
Seems to stall at the point where I get response.
Any advice?
Thanks in advance
Haven't worked with the recaptcha service previously, but I have two troubleshooting recommendations:
Use Fiddler or Firebug and watch what you're sending outbound. Verifying your parameters would help you with basic troubleshooting, i.e. invalid characters, etc.
The Recaptcha Wiki has an entry about dealing with development on Vista. It doesn't have to be limited to Vista, though; if you're system can handle IPv6, then your browser could be communicating in that format as a default. It appears as if Recaptcha deals with IPv4. Having Fiddler/Firebug working would tell you about those other parameters that could be causing you grief.
This may not help solve your problem but it might provide you with better troubleshooting info.
So got this working, for some reason I needed to write the request to a stream like so:
//Write data to request stream
using (Stream requestSteam = request.GetRequestStream())
requestSteam.Write(byteData, 0, byteData.Length);
Could anyone explain why this works. I didn't think I would need to do this, don't completely understand what's happening behind the scenes..
Damien's answer is correct of course, but just to be clear about the order of things (I was a little confused) and to have a complete code sample...
var uri = new Uri("http://api-verify.recaptcha.net/verify");
var queryString = string.Format(
"privatekey={0}&remoteip={1}&challenge={2}&response={3}",
privateKey,
userIP,
challenge,
response);
var request = (HttpWebRequest)HttpWebRequest.Create(uri);
request.Method = WebRequestMethods.Http.Post;
request.ContentLength = queryString.Length;
request.ContentType = "application/x-www-form-urlencoded";
using (var writer = new StreamWriter(request.GetRequestStream()))
{
writer.Write(queryString);
}
string result;
using (var webResponse = (HttpWebResponse)request.GetResponse())
{
var reader = new StreamReader(webResponse.GetResponseStream());
result = reader.ReadToEnd();
}
There's a slight difference in that I'm writing the post variables to the request, but the core of it is the same.