Develop Reference

HttpWebRequest possibly slowing website

Using Visual studio 2012, C#.net 4.5 , SQL Server 2008, Feefo, Nopcommerce
Hey guys I have Recently implemented a new review service into a current site we have.
When the change went live the first day all worked fine.
Since then though the sending of sales to Feefo hasnt been working, There are no logs either of anything going wrong.
In the OrderProcessingService.cs in Nop Commerce's Service, i call a HttpWebrequest when an order has been confirmed as completed. Here is the code.
var email = HttpUtility.UrlEncode(order.Customer.Email.ToString());
var name = HttpUtility.UrlEncode(order.Customer.GetFullName().ToString());
var description = HttpUtility.UrlEncode(productVariant.ProductVariant.Product.MetaDescription != null ? productVariant.ProductVariant.Product.MetaDescription.ToString() : "product");
var orderRef = HttpUtility.UrlEncode(order.Id.ToString());
var productLink = HttpUtility.UrlEncode(string.Format("myurl/p/{0}/{1}", productVariant.ProductVariant.ProductId, productVariant.ProductVariant.Name.Replace(" ", "-")));
string itemRef = "";
try
{
itemRef = HttpUtility.UrlEncode(productVariant.ProductVariant.ProductId.ToString());
}
catch
{
itemRef = "0";
}
var url = string.Format("feefo Url",
login, password,email,name,description,orderRef,productLink,itemRef);
var request = (HttpWebRequest)WebRequest.Create(url);
request.KeepAlive = false;
request.Timeout = 5000;
request.Proxy = null;
using (var response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusDescription == "OK")
{
var stream = response.GetResponseStream();
if(stream != null)
{
using (var reader = new StreamReader(stream))
{
var content = reader.ReadToEnd();
}
}
}
}
So as you can see its a simple webrequest that is processed on an order, and all product variants are sent to feefo.
Now:
this hasnt been happening all week since the 15th (day of the
implementation)
the site has been grinding to a halt recently.
The stream and reader in the the var content is there for debugging.
Im wondering does the code redflag anything to you that could relate to the process of website?
Also note i have run some SQL statements to see if there is any deadlocks or large escalations, so far seems fine, Logs have also been fine just the usual logging of Bots.
Any help would be much appreciated!
EDIT: also note that this code is in a method that is called and wrapped in A try catch
UPDATE: well forget about the "not sending", thats because i was just told my code was rolled back last week

A call to another web site while processing the order can degrade performance, as you are calling to a site that you do not control. You don't know how much time it is going to take. Furthermore, the GetResponse method can throw an exception, if you don't log anything in your outer try/catch block then you won't be able to know what's happening.
The best way to perform such a task is to implement something like the "Send Emails" scheduled task, and send data when you can afford to wait for the remote service. It is easy if you try. It is more resilient and easier to maintain if you upgrade the nopCommerce code base.
This is how I do similar things:
Avoid modifying the OrderProcessingService: Create a custom service or plugin that consumes the OrderPlacedEvent or the OrderPaidEvent (just implement the IConsumer<OrderPaidEvent> or IConsumer<OrderPlacedEvent> interface).
Do not call to a third party service directly while processing the request if you don't need the response at that moment. It will only delay your process. At the service created in step 1, store data and send it to Feefo later. You can store data to database or use an static collection if you don't mind losing pending data when restarting the site (that could be ok for statistical data for instance).
Best way to implement point #2 is to add a new scheduled task implementing ITask (remember to add a record to the ScheduleTask table). Just recover the stored data do your processing.
Add some logging. It is easy, just get an ILogger instance and call Insert.

As far as I can see, you are making a blocking synchronous call to other websites, which will definitely slow down your site in between the request-response process. What Marco has suggested is valid, try to do it in an ITask. Or you can use an asynchronous web request to potentially remove the block, if you need things done immediately instead of scheduled. :)

Getting a web response timesout

I've been asked to write a small program which checks that all the pages we have online are not erroring.
To do this, I use the following code (where pathsToCheck is List, each string is a URL like http://www.domain.com/webpage)
foreach (string path in pathsToCheck)
{
HttpWebResponse response = null;
try
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(path);
webRequest.AllowAutoRedirect = true;
response = (HttpWebResponse)webRequest.GetResponse();
System.Diagnostics.Debug.Assert(response.StatusDescription == "OK", "Look into this, it doesn't like the response code");
System.Threading.Thread.Sleep(1000);
}
catch (Exception ex)
{
Console.WriteLine("Failed : " + path);
}
finally
{
Write(--totalPathsToCheck);
}
}
The problem I am having is it always fails (timesout) from the third item in the list (everything fails from the third). Naturally, I guessed there must be a fault with the third item, but there isn't.
Since the first item doesn't time out, I created a new list, with 5 items, all of the same URL (one I know doesn't time out). The same issue occurs, on the third iteration, it times out and continues to time out for the remainder of the list.
I then decided to test a different URL (on a different domain) and the same issue persists.
I added the sleep to the code, and increased it in-case there were too many requests within a given period but that made no differences.
What should I be doing?

You need to close your connections. Add
response.Close();
From http://msdn.microsoft.com/en-us/library/system.net.httpwebresponse.close.aspx:
The Close method closes the response stream and releases the connection to the resource for reuse by other requests.
You must call either the Stream.Close or the HttpWebResponse.Close method to close the stream and release the connection for reuse. It is not necessary to call both Stream.Close and HttpWebResponse.Close, but doing so does not cause an error. Failure to close the stream can cause your application to run out of connections.

SqlConnection timeout oddities

I am experiencing a weird issue where the timeout remaining for a connection attempt seems to be persisting through multiple attempts.
I have a simple window where connection information is entered, with a connect and cancel button.
When the user hits connect, the following code executes:
DisableControls();
if((bool)AutoAuthenticated.IsChecked)
{
((MainWindow)Owner).myConnection = new System.Data.SqlClient.SqlConnection
("server="+ServerName.Text + ";"+
"Trusted_Connection=yes;" +
"database="+DatabaseName.Text + ";" +
"connection timeout=3");
}
else
{
((MainWindow)Owner).myConnection = new System.Data.SqlClient.SqlConnection
("user id="+UserName.Text+";" +
"password="+Password.Password+";" +
"server="+ServerName.Text+";" +
"Trusted_Connection=no;" +
"database="+DatabaseName.Text+";" +
"connection timeout=3");
}
await ConnectToServer();
This is the ConnectToServer function:
private async Task ConnectToServer()
{
//using (((MainWindow)Owner).myConnection)
//{
await ((MainWindow)Owner).myConnection.OpenAsync();
//}
}
The timeout property is small right now for the purposes of testing.
Anyway, if the user hits the cancel button whilst connecting:
private void Cancel_Click(object sender, RoutedEventArgs e)
{
if (((MainWindow)Owner).myConnection != null &&
((MainWindow)Owner).myConnection.State ==
System.Data.ConnectionState.Connecting)
{
((MainWindow)Owner).myConnection.Close();
EnableControls();
}
else
{
this.Close();
}
}
Now, if I enter in bogus details and leave it to timeout, then I catch the exception with 'The network path was not found.'.
If I try and connect again after this (or maybe the one after the next attempt depending on how quick I was to hit the cancel button), it times out almost instantly after hitting the connect button, without waiting the amount of time it was supposed to.
I'm not sure what I'm doing wrong here, I've tried looking in to resetting the SqlConnection object, but I thought the setting of it to a new SqlConnection each time I hit connect should be doing this already?

I found out what I was missing. Even when closing the connection, disposing of the object, keeping it all in a using statement, the issue persisted because there was another element that I was unaware of - the connection pool.
I still have to do some reading on this, but basically it reuses old connection information and so on to save on resources, etc.
There's more information here, but in case this helps someone else with similar issues, just call the ClearPool() function when you close the connection:
Example (in my exception handling code, and after I call Close()):
System.Data.SqlClient.SqlConnection.ClearPool(((MainWindow)Owner).myConnection);
Obviously from a resource usage perspective, it would probably be best to call that function only when you don't need to reuse the connection, but that said, I still need to read up on this so don't take my word for it!

How do I increase the performance of HttpWebResponse?

I am trying to build an application that sends and receives responses from a website.
None of the solutions I've read on Stack Overflow have solved my problem, so I think that my code could use optimization.
I have the following thread:
void DomainThreadNamecheapStart()
{
while (stop == false)
{
foreach (string FromDomainList in DomainList.Lines)
{
if (FromDomainList.Length > 1)
{
// I removed my api parameters from the string
string namecheapapi = "https://api.namecheap.com/foo" + FromDomainList + "bar";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(namecheapapi);
request.Proxy = null;
request.ServicePoint.Expect100Continue = false;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader sr = new StreamReader(response.GetResponseStream());
status.Text = FromDomainList + "\n" + sr.ReadToEnd();
sr.Close();
}
}
}
}
This thread is called when a button is clicked:
private void button2_Click(object sender, EventArgs e)
{
stop = false;
Thread DomainThread = new Thread(new ThreadStart(DomainThreadNamecheapStart));
DomainThread.Start();
}
I only receive around 12 responses in 10 seconds using the above code. When I try to make the same request in JavaScript or using a simple IFrame, it's more than twice as fast. The Browser doesn't use multiple threads for the connection, it waits until one is finished and then starts the new one.
I tried setting request.Proxy = null;, but it had negligible impact.
I have noticed that HTTPS is 2-3 times slower than HTTP. Unfortunately, I have to use HTTPS. Is there anything I can do to make it faster?

My bet would be on the aspect you pointed out - the HTTPS protocol.
The iteration between Client(browser) and Server for pure HTTP protocol is quite straightforward: Ask for the info, get info. If 1.0, close connection; if 1.1, it may stay alive for reuse. (Check image 1 for details.)
But when you do a HTTPS request, the initial protocol overhead is considerable (image 2); but, once the initial negotiation is done, some symmetric encryption takes place, and no further certificate negotiation is necessary, thus speeding up data transfer.
What I think the problem is, if you destroy the HTTPWebRequest object and creates a new one, the full HTTPS exchange takes place once again, slowing your iteration. (HTTPS + HTTP 1.1 Keepalive should do fine, though.)
So, suggestions: Switch to HTTP only, or reuse the connection objects.
And i hope it works for you. =)
(1) HTTP protocol handshake and response
(2) Initial HTTPS protocol handshake

Try to modify the System.Net.ServicePointManager.DefaultConnectionLimit value (the default value is 2).
Other reference (Performance Issues Part).

try these, it helped me to improve the performance,
ServicePointManager.Expect100Continue = false;
ServicePointManager.DefaultConnectionLimit = 200;
ServicePointManager.MaxServicePointIdleTime = 2000;
ServicePointManager.SetTcpKeepAlive(false, 0, 0);

C# HttpWebRequest hangs program suddenly. Did not earlier

Earlier I made an HttpWebRequest that worked perfectly fine, and my StreamReader read the HTML of the website perfectly.
But all of the sudden, after having tested it's functionality and confirmed that it worked many times, it hangs the program when it comes to the StreamReader line.
I have tried removing this line, and the code continued.
The thing is; I tried inputting a different website than the one I need to use, (I put in www.google.com) and it worked perfectly fine. So my error conclusion is, that it is only the website I need to use that I can't access anymore which makes me think that the endpart (the website) is cancelling my connection or blocking me or something. BUT! The HttpWebRequest itself doesn't hang or anything, which must mean that it successfully established a request to the website?
Enough chit-chat, here's the code:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("website here");
MessageBox.Show("1"); //This is shown.
string HTMLLink = (new StreamReader(request.GetResponse().GetResponseStream())).ReadToEnd(); //This is where the program hangs....
MessageBox.Show("2"); //This is not shown! Below this isn't being executed.
if (HTMLLink.Length > 0)
{
HTMLLink = HTMLLink.Substring(HTMLLink.IndexOf("uuu"), HTMLLink.Length - HTMLLink.IndexOf("uuu"));
HTMLLink = HTMLLink.Substring(0, HTMLLink.IndexOf("\" TARGET="));
request = (HttpWebRequest)WebRequest.Create(HTMLLink);
string HTML = (new StreamReader(request.GetResponse().GetResponseStream())).ReadToEnd();
if (HTML.Length > 0 && HTML.Contains(" </script><br><br><br>") && HTML.Contains(" <br><br><script "))
{
HTML = HTML.Substring(HTML.IndexOf(" </script><br><br><br>") + 22, HTML.IndexOf("<br><br><script "));
HTML = HTML.Substring(0, HTML.IndexOf("<br><br><script "));
HTML = HTML.Replace("\r\n", "");
HTML = HTML.Replace("\n", "");
HTML = HTML.Replace("<br>", "\r\n");
HTML = HTML.Replace("<BR>", "\r\n");
HTML = HTML.Replace("<br />", "\r\n");
HTML = HTML.Replace("<BR />", "\r\n");
textBox.Text = HTML;
}
}
And please keep in mind that it worked perfectly earlier then all of the sudden it started hanging, and that it works fine with www.google.com.
And by the way, yes I have done many searches. No useful results.
I have tried the timeout already, it does timeout.
Maybe the website has blocked my program thinking it's a spider? what then?
Everytime when I reach the StreamReader (no matter how I set it up) it starts to hang.
And it keeps hanging, it doesn't deliver any result.
This ONLY happens with lyrics007.com which is the exact website I need. It works fine with google.
Help, please!
Thanks in advance!

WebRequest.GetResponse() is a blocking call. It will wait until it can successfully connect and receive the response before it returns control to the caller, or will throw an exception if unsuccessful. This behaviour can't be modified.
You usually don't want your application to sit waiting for something to happen though, so you usually delegate the GetResponse() call to another thread, so you can continue doing other work in the current thread.
The usual way to overcome this problem is to call asynchronously. Rather than a call to GetResponse, you will call BeginGetResponse(), passing in a function which should be executed when the operation completes (eg, containing the remainder of your current method, plus a call to EndGetResponse()). Control of execution can be passed back to the caller whilst the response is being waited for in a background thread, handled for you automatically by the .NET threadpool.

The request is not sent until the call to GetResponse. If that is where it is hanging, I would be inclined to say the site is not responding. Did you try using a web browser to connect to that URL and see if it works?