I've been asked to write a small program which checks that all the pages we have online are not erroring.
To do this, I use the following code (where pathsToCheck is List, each string is a URL like http://www.domain.com/webpage)
foreach (string path in pathsToCheck)
{
HttpWebResponse response = null;
try
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(path);
webRequest.AllowAutoRedirect = true;
response = (HttpWebResponse)webRequest.GetResponse();
System.Diagnostics.Debug.Assert(response.StatusDescription == "OK", "Look into this, it doesn't like the response code");
System.Threading.Thread.Sleep(1000);
}
catch (Exception ex)
{
Console.WriteLine("Failed : " + path);
}
finally
{
Write(--totalPathsToCheck);
}
}
The problem I am having is it always fails (timesout) from the third item in the list (everything fails from the third). Naturally, I guessed there must be a fault with the third item, but there isn't.
Since the first item doesn't time out, I created a new list, with 5 items, all of the same URL (one I know doesn't time out). The same issue occurs, on the third iteration, it times out and continues to time out for the remainder of the list.
I then decided to test a different URL (on a different domain) and the same issue persists.
I added the sleep to the code, and increased it in-case there were too many requests within a given period but that made no differences.
What should I be doing?
You need to close your connections. Add
response.Close();
From http://msdn.microsoft.com/en-us/library/system.net.httpwebresponse.close.aspx:
The Close method closes the response stream and releases the connection to the resource for reuse by other requests.
You must call either the Stream.Close or the HttpWebResponse.Close method to close the stream and release the connection for reuse. It is not necessary to call both Stream.Close and HttpWebResponse.Close, but doing so does not cause an error. Failure to close the stream can cause your application to run out of connections.
Related
About:
I have this Windows Form application which every 60 seconds it captures information from two common web pages, do some simple string treatment with the result and do something (or not) based in the result.
One of those sites doesn't have any protection, so I can easily get it's HTML code using HttpWebRequest and it's HttpWebResponse.GetResponseStream().
The other one has some code protection and I can't use the same approach. The solution was use the WebBrowser class to select all text of the site and copy to the clipboard, as Jake Drew posted here (method 1).
Extra information:
When the timer reachs 1 min, each method is asynchronously execuded using Task. At the end of each Task the main thread will search some information in those texts and take or not some decisions based in the result. After this process, not even the captured text will relevant anymore. Basically everything can be wipe out from memory, since I'll get everything new and process it in about 1 minute.
Problem:
Everything is working fine but the problem is that it's gradually increasing the memory usage (about 20mb for each ticking), which are unecessary as I said before I don't need to maintain data in running in memory more than I had in the begin of app's execution:
and after comparing two snapshots I've found these 3 objects. Apparently they're responsible for that excess of memory usage:
So, even after I put the main execution in Tasks and do everything I could to help the Garbage Collector, I still have this issue.
What else could I do to avoid this issue or dump the trash from memory??
Edit:
Here's the code that is capturing the HTML of the page using HttpWebRequest:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) {
if (response.StatusCode == HttpStatusCode.OK) {
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = null;
if (response.CharacterSet == null) {
readStream = new StreamReader(receiveStream);
} else {
readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
}
PB_value = readStream.ReadToEnd();
readStream.Close(); //Ensure
}
response.Close(); //Ensure
}
Solved:
After some research I've found a solution. I actually feel kind of ashamed because it is a quite simple solution that I haven't tried before, still, it's important to share.
The first thing I did was create an Event to identify when my two Tasks were finished then I assigned two functions to this event. The first function forced the Garbage Collector (GC.Collect()). The second function disposed the two Tasks, since all the main processes were done inside them (T.Dispose()). Then I've got the result I wanted:
I'm writing a program that reads data from a series of web pages; this works for the first page, however it then hangs while waiting for a response from the next page until it times out. Whether I start by reading page 1, 2, 140, etc. it will always successfully read the first page but none after that.
I think this might be related to the "cookieHeader" variable, which is needed to access the pages as the website requires a user to login first. However when I inspect this element, its expiration time is set as the following day, so I don't see how it could be expired yet.
I am new to this so I'm hoping someone who has encountered this problem before or who has a better understanding of cookies could help me. I would appreciate any input! Below is a snippet of code, where the timeout error is being caught by the try-catch wrapper.
// loop through each page
for (int i = 1; i <= totalPages; i++)
{
string thisUrl = chatUrl + i; //add page number to url
WebRequest getReq = WebRequest.Create(thisUrl);
getReq.Headers.Add("Cookie", cookieHeader);
try
{
WebResponse getResp = getReq.GetResponse();
Console.WriteLine("Page " + i + " read successfully");
}
catch (Exception e)
{
Console.WriteLine("Page " + i + " failed");
}
}
I think it has got something to do with not closing the response properly. WebResponse implements IDisposable if I remember it right. So try disposing it using using statement. Also, there are some restrictions on some client versions of .Net where, by default, you can only have 2 connections open to the same server simultaneously. This restriction might kick in if you do not dispose the connections (WebResponse) properly (potentially because of keep alive settings).
Using Visual studio 2012, C#.net 4.5 , SQL Server 2008, Feefo, Nopcommerce
Hey guys I have Recently implemented a new review service into a current site we have.
When the change went live the first day all worked fine.
Since then though the sending of sales to Feefo hasnt been working, There are no logs either of anything going wrong.
In the OrderProcessingService.cs in Nop Commerce's Service, i call a HttpWebrequest when an order has been confirmed as completed. Here is the code.
var email = HttpUtility.UrlEncode(order.Customer.Email.ToString());
var name = HttpUtility.UrlEncode(order.Customer.GetFullName().ToString());
var description = HttpUtility.UrlEncode(productVariant.ProductVariant.Product.MetaDescription != null ? productVariant.ProductVariant.Product.MetaDescription.ToString() : "product");
var orderRef = HttpUtility.UrlEncode(order.Id.ToString());
var productLink = HttpUtility.UrlEncode(string.Format("myurl/p/{0}/{1}", productVariant.ProductVariant.ProductId, productVariant.ProductVariant.Name.Replace(" ", "-")));
string itemRef = "";
try
{
itemRef = HttpUtility.UrlEncode(productVariant.ProductVariant.ProductId.ToString());
}
catch
{
itemRef = "0";
}
var url = string.Format("feefo Url",
login, password,email,name,description,orderRef,productLink,itemRef);
var request = (HttpWebRequest)WebRequest.Create(url);
request.KeepAlive = false;
request.Timeout = 5000;
request.Proxy = null;
using (var response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusDescription == "OK")
{
var stream = response.GetResponseStream();
if(stream != null)
{
using (var reader = new StreamReader(stream))
{
var content = reader.ReadToEnd();
}
}
}
}
So as you can see its a simple webrequest that is processed on an order, and all product variants are sent to feefo.
Now:
this hasnt been happening all week since the 15th (day of the
implementation)
the site has been grinding to a halt recently.
The stream and reader in the the var content is there for debugging.
Im wondering does the code redflag anything to you that could relate to the process of website?
Also note i have run some SQL statements to see if there is any deadlocks or large escalations, so far seems fine, Logs have also been fine just the usual logging of Bots.
Any help would be much appreciated!
EDIT: also note that this code is in a method that is called and wrapped in A try catch
UPDATE: well forget about the "not sending", thats because i was just told my code was rolled back last week
A call to another web site while processing the order can degrade performance, as you are calling to a site that you do not control. You don't know how much time it is going to take. Furthermore, the GetResponse method can throw an exception, if you don't log anything in your outer try/catch block then you won't be able to know what's happening.
The best way to perform such a task is to implement something like the "Send Emails" scheduled task, and send data when you can afford to wait for the remote service. It is easy if you try. It is more resilient and easier to maintain if you upgrade the nopCommerce code base.
This is how I do similar things:
Avoid modifying the OrderProcessingService: Create a custom service or plugin that consumes the OrderPlacedEvent or the OrderPaidEvent (just implement the IConsumer<OrderPaidEvent> or IConsumer<OrderPlacedEvent> interface).
Do not call to a third party service directly while processing the request if you don't need the response at that moment. It will only delay your process. At the service created in step 1, store data and send it to Feefo later. You can store data to database or use an static collection if you don't mind losing pending data when restarting the site (that could be ok for statistical data for instance).
Best way to implement point #2 is to add a new scheduled task implementing ITask (remember to add a record to the ScheduleTask table). Just recover the stored data do your processing.
Add some logging. It is easy, just get an ILogger instance and call Insert.
As far as I can see, you are making a blocking synchronous call to other websites, which will definitely slow down your site in between the request-response process. What Marco has suggested is valid, try to do it in an ITask. Or you can use an asynchronous web request to potentially remove the block, if you need things done immediately instead of scheduled. :)
I am currently facing a strange problem. I'm testing website response times, but when the method is looped on the third time (of timing the connection) it hangs:
internal class Program
{
Console.ReadLine();
loop();
}
}
The output before it hangs is: "HTTP Response Timer", so I assume it's something to do with the running instance.
You have to close the response. Otherwise you reach the maximum of open connections.
Use a using statement, this is the most simple way to close and dispose the response:
using(HttpWebResponse response = (HttpWebResponse)request.GetResponse()){
// .. here some action if any
}
You're not disposing of the response, which means it's hanging onto the connection from the previous requests. The new request is then waiting to get that connection from the pool, and blocking because the old response still "owns" the connection Just change your code to:
// You weren't really using the HttpWebResponse class anyway, so why cast?
using (var response = request.GetResponse())
{
}
Did you really mean to recurse though? Why aren't you looping using:
while(true)
{
GetResponse();
}
or something like that?
Earlier I made an HttpWebRequest that worked perfectly fine, and my StreamReader read the HTML of the website perfectly.
But all of the sudden, after having tested it's functionality and confirmed that it worked many times, it hangs the program when it comes to the StreamReader line.
I have tried removing this line, and the code continued.
The thing is; I tried inputting a different website than the one I need to use, (I put in www.google.com) and it worked perfectly fine. So my error conclusion is, that it is only the website I need to use that I can't access anymore which makes me think that the endpart (the website) is cancelling my connection or blocking me or something. BUT! The HttpWebRequest itself doesn't hang or anything, which must mean that it successfully established a request to the website?
Enough chit-chat, here's the code:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("website here");
MessageBox.Show("1"); //This is shown.
string HTMLLink = (new StreamReader(request.GetResponse().GetResponseStream())).ReadToEnd(); //This is where the program hangs....
MessageBox.Show("2"); //This is not shown! Below this isn't being executed.
if (HTMLLink.Length > 0)
{
HTMLLink = HTMLLink.Substring(HTMLLink.IndexOf("uuu"), HTMLLink.Length - HTMLLink.IndexOf("uuu"));
HTMLLink = HTMLLink.Substring(0, HTMLLink.IndexOf("\" TARGET="));
request = (HttpWebRequest)WebRequest.Create(HTMLLink);
string HTML = (new StreamReader(request.GetResponse().GetResponseStream())).ReadToEnd();
if (HTML.Length > 0 && HTML.Contains(" </script><br><br><br>") && HTML.Contains(" <br><br><script "))
{
HTML = HTML.Substring(HTML.IndexOf(" </script><br><br><br>") + 22, HTML.IndexOf("<br><br><script "));
HTML = HTML.Substring(0, HTML.IndexOf("<br><br><script "));
HTML = HTML.Replace("\r\n", "");
HTML = HTML.Replace("\n", "");
HTML = HTML.Replace("<br>", "\r\n");
HTML = HTML.Replace("<BR>", "\r\n");
HTML = HTML.Replace("<br />", "\r\n");
HTML = HTML.Replace("<BR />", "\r\n");
textBox.Text = HTML;
}
}
And please keep in mind that it worked perfectly earlier then all of the sudden it started hanging, and that it works fine with www.google.com.
And by the way, yes I have done many searches. No useful results.
I have tried the timeout already, it does timeout.
Maybe the website has blocked my program thinking it's a spider? what then?
Everytime when I reach the StreamReader (no matter how I set it up) it starts to hang.
And it keeps hanging, it doesn't deliver any result.
This ONLY happens with lyrics007.com which is the exact website I need. It works fine with google.
Help, please!
Thanks in advance!
WebRequest.GetResponse() is a blocking call. It will wait until it can successfully connect and receive the response before it returns control to the caller, or will throw an exception if unsuccessful. This behaviour can't be modified.
You usually don't want your application to sit waiting for something to happen though, so you usually delegate the GetResponse() call to another thread, so you can continue doing other work in the current thread.
The usual way to overcome this problem is to call asynchronously. Rather than a call to GetResponse, you will call BeginGetResponse(), passing in a function which should be executed when the operation completes (eg, containing the remainder of your current method, plus a call to EndGetResponse()). Control of execution can be passed back to the caller whilst the response is being waited for in a background thread, handled for you automatically by the .NET threadpool.
The request is not sent until the call to GetResponse. If that is where it is hanging, I would be inclined to say the site is not responding. Did you try using a web browser to connect to that URL and see if it works?