I have a question regarding the performance of the .Net HttpWebRequest client (or WebClient, gives similar results).
If I use HttpWebRequest to request an html page (in this case news.bbc.co.uk) and analyse the speed (using HttpAnalyzer) at which the response is read by the application, this is significantly slower than a browser (Firefox, Chrome, IE) requesting the same resource (all caches cleared etc). The .Net application takes approximately 1.7 seconds versus 0.2 - 0.3 seconds for a browser.
Is this purely down to the speed and efficiency of the code / application or are there any other factors to consider?
Code as follows:
HttpWebRequest request = null;
Uri uriTest = new Uri("http://news.bbc.co.uk");
request = (HttpWebRequest)WebRequest.Create(uriTest);
request.Method = "GET";
request.KeepAlive = true;
request.Headers["Accept-Encoding"] = "gzip, deflate";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
response.Close();
I'd jack Fiddler in the middle, run the browser request and the .NET request one after the other and make sure you're really getting what you think. It's possible there's redirection or something else hinky going on (maybe browser is pre-appending the '/' while .NET waits for the redir, etc) that isn't immediately visible. I've built huge apps on the .NET HTTP client with nothing like what you describe- something else must be going on.
What happens if you stick '/' on the end of the URL?
If you make two requests does the second one happen more quickly?
I have also notice speed disparities between browsers and WebClient or WebRequest. Even the raw speed of the response can be drastically different - but not all the time!
There are a few things this could be caused by:
It could be all the .Net bootstrapping that happens. .Net assemblies aren't loaded and JITted until they are used, therefore you can see significant speed degradation on the initial call to a piece of code even if the application itself has been running for ages. Okay - so the .Net framework itself is nGen'd - but there's still the bridge between your code and the .Net framework to build on the fly.
Just checking that you're running without the debugger attached and that you definitely don't have symbol server switched on - symbol server and VS interrupts programs as the symbols are downloaded, slowing them down bucket-loads. Sorry if this is an insult ;)
Browsers are coded to make efficient use of only a few underlying sockets; and they will be opened and primed as soon as the browser is there. 'Our' code that uses .Net WebClient/WebRequest is totally inefficient in comparison, as everything is initialised anew each time.
There are a lot of platform resources associated with networking, and whilst .Net makes it much easier to code with networks, it's still bound to the same platform resource issues. Ergo, the closer you are to the platform you are, the faster some code will be. IE and Firefox et al are native and therefore can thrown around system resources natively; .Net isn't and therefore some marshalling(=slow) is required to set things up. Obviously, once a port is opened and being used, however, .Net is still no slouch; but it almost would never be as fast as well-written non-marshalled native code.
Run the application with Ctrl+F5 instead of F5 (Debug mode). You will see a difference:
class Program
{
static void Main()
{
using (var client = new WebClient())
{
Stopwatch watch = Stopwatch.StartNew();
var data = client.DownloadData("http://news.bbc.co.uk");
watch.Start();
Console.WriteLine("{0} ms", watch.ElapsedMilliseconds);
}
}
}
Prints 880 ms on my PC.
What's the breakdown of that 1.7s? I suspect you are measuring the entire process?
Using this piece of code I get about 200ms in average:
var request = (HttpWebRequest)WebRequest.Create("http://www.bbc.co.uk/news/");
var stopwatch = new Stopwatch();
stopwatch.Start();
using (var response = (HttpWebResponse)request.GetResponse())
{
stopwatch.Stop();
Console.WriteLine("Elapsed: {0}ms", stopwatch.ElapsedMilliseconds);
var responseStream = response.GetResponseStream();
if (responseStream != null)
using (var sr = new StreamReader(responseStream))
Console.WriteLine("Title: {0}", Regex.Match(sr.ReadToEnd(), #"title>(.*)</title").Groups[1].Value);
}
Edit changed the code just to measure the actual HTTP request and tried again using Fiddler as well:
Program above: Elapsed: 78ms
Fiddler: Overall Elapsed: 00:00:00.0620000
The first time you request a page, .net tries to detect proxy settings. The solution is to pass in an empty WebProxy object. This way it just connects to remote server instead of autodetecting the proxy server.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uriTest);
request.Proxy = new WebProxy();
Have you watched the network while using the browser? Perhaps the browser is using cached resources?
Markos' answer worked perfectly for me for the same issue:
request.Proxy = new WebProxy();
Reduced a 16-second request to less than a second. Thanks!
It maybe that bbc.co.uk checks the User-Agent header that is being passed to it and handles the response based on that. So if it sees automated clients then it responds slowly, where as if it believes that there is a real person at the end of the line then it speeds up. If you really want to try it out just tell the HttpWebRequest to pass a different header.
Whenever you measure anything, you have to account for the startup costs. If your .net code is in a single process,and you are only measuring the single request, then your measurement will be tainted by first time costs of initializing assemblies, types, etc.
As Darin and others have suggested, you should make sure that:
1) You are not running the process under debuggger.
2) You account for startup costs.
One way you can do #2, is to make two requests and only measure the second one. Or you can make N requests, discard the 1st one, and get the average of last N-1 requests. Also make sure that you read the entity stream.
Related
I'm using C# to connect to a Webservice to grab data. However, I'm currently having problems getting the code to run on a remote server; when I say problems, I mean its running, but the connection speed between client and server is ridiculously slow (through no fault of mine - the client is providing a slow resultset via a webservice, and they have all timeouts turned off their side in order to do so.)
if ((endpointConfiguration == EndpointConfiguration.SFFService))
{
System.ServiceModel.BasicHttpBinding result = new System.ServiceModel.BasicHttpBinding();
result.MaxBufferSize = int.MaxValue;
result.ReaderQuotas = System.Xml.XmlDictionaryReaderQuotas.Max;
result.MaxReceivedMessageSize = int.MaxValue;
result.AllowCookies = true;
result.OpenTimeout = TimeSpan.MaxValue;
result.CloseTimeout = TimeSpan.MaxValue;
result.SendTimeout = TimeSpan.MaxValue;
return result;
}
So. Not a great start. Open Close and Send all set to maximum.
Anyway, I've matched their long timeouts my side, and a few of the smaller webservice requests finish and succeed ok on the server. The biggest, slowest one however just hangs indefinitely, probably because I've told it to never timeout.
However, I'm pretty sure there's some other problem happening, as I left it overnight and it just sat there. Locally, on my development machine, although slow, it works.
My question is, has anyone any idea on additional things to check about the environment that could potentially be in play here? I thought perhaps firewall, but given that the small requests succeed (and connect) it is very difficult to debug the slow requests as I've no idea how long to wait until accepting that the program isn't going to do anything.
FWIW I've tried connecting via a browser, and again, the browser just sits there waiting for the request to finish which it never does (most likely due to the timeout being turned off on the server). If there was any way to see even how much of the request was left to finish (like a percentage download) that may help give me some guidance as to if the code is doing anything other than waiting.
There's no way to get a progress of the remote call even when you are attached to the remote process. Try using a local Visual Studio on the server machine (preferably on a non-production VM) and attach to the local process rather than using the Remote Debugger.
I am not sure exactly what the question is but the first step I'd take while debugging a slow application would be to test a local connection (local client and local server) to eliminate the network from the equation. If that works well, try hosting the server on a different place (public cloud maybe?) and try again there, if it works well then there's definitely something en-route or on that server.
If you're interested in tracking how long web service calls take you could track it by placing the start time into the HttpContext.Current.Items or OperationContext.Current.Items on BeginRequest/EndRequest in Global.asax or in a MessageInspector if you use WCF (you can send the datetime between the two methods by returning it into the Before method and read it from the corelationState parameter in the After method).
Using HttpWebRequest, I'm trying to query a secured (negotiate) url behind a load balancing setup in round-robin mode (two IIS 7.5 servers). Seems simple enough, but I have some problems.
The first anonymous request goes on one server and the negotiate part goes on the other. The problem is that it takes about six seconds between these two requests, so it is way too long. Trying to diagnose the delay, I realized that, going through Fiddler's proxy, all the requests went on the same server, so it took less than one second total. If I disable Fiddlers option "reuse server connections", then my requests have the same behavior as without Fiddler and it takes forever.
Googling this, I ended up on this link: http://fiddler2.com/blog/blog/2013/02/28/help!-running-fiddler-fixes-my-app-
I know that Fiddler is using sockets and its own connection pool, but is there a way to reproduce the same behavior using .NET HttpWebRequest so that my requests (anonymous and negotiate) will reuse connections and end up on the same server?
Here is a quick test that takes about 70 seconds to complete without Fiddler, and about 2 seconds going through Fiddler...
Also, please note that it isn't a Proxy detection delay and that sticky session are disabled on the nlb.
public void Main(string[] args)
{
int i = 0;
while (i < 10)
{
HttpWebRequest wr = (HttpWebRequest)WebRequest.Create("http://nlb/service.asmx");
HttpWebResponse response;
wr.KeepAlive = true;
wr.UseDefaultCredentials = true;
response = (HttpWebResponse)wr.GetResponse();
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
Console.WriteLine(sr.ReadToEnd());
}
response.Close();
i++;
}
}
This is another proof that Fiddler is plain awesome!
Thanks for any advice.
Just a shot here, and maybe it seems too easy -
But the last line of your code is Response.Close(). The documentation prior to .NET 4.5 doesn't say much about this other than it "closes the existing socket connection."
In .NET 4.5 however, this is the documentation:
This method terminates the connection to the client in an abrupt
manner and is not intended for normal HTTP request processing.
http://msdn.microsoft.com/en-us/library/system.web.httpresponse.close(v=vs.110).aspx
I'll admit that I don't know some of the subtle differences between .NET 4.5 and the prior versions of HttpResponse; however, I do think that logically, Connection.Close() is not compatible with Keep-Alive; and you could be seeing the behavior of Fiddler intervening (maybe as a bug) to patch over this. Just a theory- needs testing.
I have an ASP.NET 3.5 server application written in C#. It makes outbound requests to a REST API using HttpWebRequest and HttpWebResponse.
I have setup a test application to send these requests on separate threads (to vaguely mimic concurrency against the server).
Please note this is more of a Mono/Environment question than a code question; so please keep in mind that the code below is not verbatim; just a cut/paste of the functional bits.
Here is some pseudo-code:
// threaded client piece
int numThreads = 1;
ManualResetEvent doneEvent;
using (doneEvent = new ManualResetEvent(false))
{
for (int i = 0; i < numThreads; i++)
{
ThreadPool.QueueUserWorkItem(new WaitCallback(Test), random_url_to_same_host);
}
doneEvent.WaitOne();
}
void Test(object some_url)
{
// setup service point here just to show what config settings Im using
ServicePoint lgsp = ServicePointManager.FindServicePoint(new Uri(some_url.ToString()));
// set these to optimal for MONO and .NET
lgsp.Expect100Continue = false;
lgsp.ConnectionLimit = 100;
lgsp.UseNagleAlgorithm = true;
lgsp.MaxIdleTime = 100000;
_request = (HttpWebRequest)WebRequest.Create(some_url);
using (HttpWebResponse _response = (HttpWebResponse)_request.GetResponse())
{
// do stuff
} // releases the response object
// close out threading stuff
if (Interlocked.Decrement(ref numThreads) == 0)
{
doneEvent.Set();
}
}
If I run the application on my local development machine (Windows 7) in the Visual Studio web server, I can up the numThreads and receive the same avg response time with minimal variation whether it's 1 "user" or 100.
Publishing and deploying the application to Apache2 on a Mono 2.10.2 environment, the response times scale almost linearly. (i.e, 1 thread = 300ms, 5 thread = 1500ms, 10 threads = 3000ms). This happens regardless of server endpoint (different hostname, different network, etc).
Using IPTRAF (and other network tools), it appears as though the application only opens 1 or 2 ports to route all connections through and the remaining responses have to wait.
We have built a similar PHP application and deployed in Mono with the same requests and the responses scale appropriately.
I have run through every single configuration setting I can think of for Mono and Apache and the ONLY setting that is different between the two environments (at least in code) is that sometimes the ServicePoint SupportsPipelining=false in Mono, while it is true from my machine.
It seems as though the ConnectionLimit (default of 2) is not being changed in Mono for some reason but I am setting it to a higher value both in code and the web.config for the specified host(s).
Either me and my team are overlooking something significant or this is some sort of bug in Mono.
I believe that you're hitting a bottleneck in the HttpWebRequest. The web requests each use a common service point infrastructure within the .NET framework. This appears to be intended to allow requests to the same host to be reused, but in my experience results in two bottlenecks.
First, the service points allow only two concurrent connections to a given host by default in order to be compliant to the HTTP specification. This can be overridden by setting the static property ServicePointManager.DefaultConnectionLimit to a higher value. See this MSDN page for more details. It looks as if you're already addressing this for the individual service point itself, but due to the concurrency locking scheme at the service point level, doing so may be contributing to the bottleneck.
Second, there appears to be an issue with lock granularity in the ServicePoint class itself. If you decompile and look at the source for the lock keyword, you'll find that it uses the instance itself to synchronize and does so in many places. With the service point instance being shared among web requests for a given host, in my experience this tends to bottleneck as more HttpWebRequests are opened and causes it to scale poorly. This second point is mostly personal observation and poking around the source, so take it with a grain of salt; I wouldn't consider it an authoritative source.
Unfortunately, I did not find a reasonable substitute at the time that I was working with it. Now that the ASP.NET Web API has been released, you may wish to give the HttpClient a look. Hope that helps.
I know this is pretty old but I'm putting this here in case it might help somebody else who runs into this issue. We ran into the same problem with parallel outbound HTTPS requests. There are a few issues at play.
The first issue is that ServicePointManager.DefaultConnectionLimit did not change the connection limit as far as I can tell. Setting this to 50, creating a new connection, and then checking the connection limit on the service point for the new connection says 2. Setting it on that service point to 50 once appears to work and persist for all connections that will end up going through that service point.
The second issue we ran into was with threading. The current implementation of the mono thread pool appears to create at most 2 new threads per second. This is an eternity if you are doing many parallel requests that start at exactly the same time. To counteract this, we tried setting ThreadPool.SetMinThreads to a higher number. It appears that Mono only creates up to 1 new thread when you make this call, regardless of the delta between the current number of threads and the desired number. We were able to work around this by calling SetMinThreads in a loop until the thread pool had the desired number of idle threads.
I opened a bug about the latter issue because that's the one I'm most confident is not working as intended: https://bugzilla.xamarin.com/show_bug.cgi?id=7055
If #jake-moshenko is right about ServicePointManager.DefaultConnectionLimit not having any effect if changed in Mono, please file this as a bug in http://bugzilla.xamarin.com/.
However I would try some things before discarding this completely as a Mono issue:
Try using the SGen garbage collector instead of the old boehm one, by passing --gc=sgen as a flag to mono.
If the above doesn't help, upgrade to Mono 3.2 (which BTW defaults to SGEN GC too), because there has been a lot of fixes since you asked the question.
If the above doesn't help, build your own Mono (master branch), as this important pull request about threading has been merged recently.
If the above doesn't help, build your own Mono with this pull request added. If it fixes your problem, please add a "+1" to the pull request. It might be a fix for bug 7055.
I have a strange situation in my .NET CF 3.5 Windows Mobile 6.5 application.
I have 2 threads.
In 1st thread I do following:
try
{
String url = "http://myserverurl";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
_currentRequest = request;
request.Timeout = 10000;
response = (HttpWebResponse)request.GetResponse();
ConnectionStatus connStatus = response.StatusCode == HttpStatusCode.OK;
response.Close();
}
catch (Exception e)
{
//log e
}
finally
{
}
In 2n thread a call a WebService through a SoapHttpClientProtocol based class generated by WebService reference.
soapClient.Url = "http://myserverurl";
soapClient.MethodOnWebService();
In both cases the url is the same. The 1st thread is used for connection checking purpose. It does the WebRequest periodically to check whetrher the server is available and displays the connection status (not shown in code). 2nd thread calls WebService on the same server (url). I observed, that when one thread is executing a WebRequest the 2nd one gets blocked or event timeouted while executing a Web Method. They both look as interfering each other. Why? I wonder if windows mobile network stack simply creates only one socket connection for both threads if it realizes that both goes to the same target IP:port? What about sessions? On desktop Windows I would expect 2 sessions being created and at least 2 sockets on client machine.
Does anybody have any hints on how Windows Mobile (or .NET CF) manages connections and socket reusage?
Regards
I would guess that there is a third session somewhere. What you're seeing is most likely due to a little-known (until it bites you, like now) recommended connection-limitation in the HTTP-protocol. Section 8.1.4 of RFC2068 says "A single-user client SHOULD maintain AT MOST 2 connections with any server or proxy". I've experienced the same limitation myself, most recently on Windows Phone 7.
The limit lies in the WebRequest and the solution is to increase the limit:
// set connection limit to 5
ServicePointManger.DefaultConnectionLimit = 5;
See e.g. this old blog entry from David Kline.
I have an application that needs to download several files in a row in succession (sometimes a few thousand). However, what ends up happening when several files need to be downloaded is I get an exception with an inner exception of type SocketException and the error code 10048 (WSAEADDRINUSE). I did some digging and basically it's because the server has run out of sockets (and they are all waiting for 240s or so before they become available again) - not coincidentally it starts happening around the 1024 file range. I would expect that the HttpWebRequest/ServicePointManager would be reusing my connection, but apparently it is not (and the files are https, so that may be part of it). I never saw this problem in the C++ code that this was ported from (but that doesn't mean it didn't ever happen - I'd be surprised if it was, though).
I am properly closing the WebRequest object and the HttpWebRequest object has KeepAlive set to true by default. Next my intent is to fiddle around with ServicePointManager.SetTcpKeepAlive(). However, I can't see how more people haven't run into this problem.
Has anyone else run into the problem, and if so, what did you do to get around it? Currently I have a retry scheme that detects this error and waits it out, but that doesn't seem like the right thing to do.
Here's some basic code to verify what I'm doing (just in case I'm missing closing something):
WebRequest webRequest = WebRequest.Create(uri);
webRequest.Method = "GET";
webRequest.Credentials = new NetworkCredential(username, password);
WebResponse webResponse = webRequest.GetResponse();
try
{
using(Stream stream = webResponse.GetResponseStream())
{
// read the stream
}
}
finally
{
webResponse.Close()
}
What kind of application is this? You mentioned that the server is running out of ports, but then you mentioned HttpWebRequest. Are you running this code in a webservice or ASP.NET page, which is trying to then download multiple files for the same incoming request from the client?
What kind of authentication is the page using? If it is using NTLM authentication, then the connections cannot be shared if the credentials being used are different for each request.
What I would suggest is to group your request per credential. So, for eg, all requests using username "John" would be grouped. You can specify the "ConnectionGroupName" property on the service point, so the system will try to reuse connections for the same credential and server.
If that also doesnt work, you will need to do one or more of the following:
1) Throttle your requests.
2) Increase the wildcard port range.
3) Use the BindIPConnectionCallback on ServicePoint to make it bind to a non-wildcard port (i.e a port in the range 1024-16384)
More digging seems to point to it possibly being due to authentication and the UnsafeAuthenticatedConnectionSharing property might alleviate this. However, I'm not sure that's the best thing, either.