I have a user control that is featured on several pages of a heavily hit site. Some of these pages include our blog sidebar, our forum sidebar, and smack right in the middle of our home page. That means this control is rendered a lot. The control it meant to read in a twitter RSS feed for a specific account and write out the last 2 tweets. Below is the majority of my Page_Load for the control. I have it in a try {} catch because it seems that on production the site is unable to load the XML often, so I need to flip on a friendly error message.
try {
var feed = XmlReader.Create(ResourceManager.GetString("TwitterRSS"));
var latestItems = SyndicationFeed
.Load(feed)
.GetRss20Formatter()
.Feed
.Items
.Take(2);
rptTwitter.ItemDataBound += new RepeaterItemEventHandler(rptTwitter_ItemDataBound);
rptTwitter.DataSource = latestItems;
rptTwitter.DataBind();
} catch (Exception ex) {
phError.Visible = true;
}
As you can see, I just fetch the 2 most recent items and repeat over them in a repeater for the front-end. If anything fails, I flip on the error PlaceHolder which says something like "Twitter is unavailable".
I often see this error message on the prod site so I'm wondering if it's making too many requests to the RSS feed. I was thinking about output caching the control for 10 minutes but I thought, "what if it gets cached in the error state"? Then it's guaranteed to display the error message panel for 10 minutes. My question is, is true that if it displays the error from the catch when its creating a newly cached version, will that truly be cached for 10 minutes (assuming I set Duration="600")? Does anyone have any tips as to how I can make this work better or cache when only real Twitter data is rendered, not the error message?
Thanks in advance
Instead of caching the entire page, I would cache the application data returned by your
var latestItems = ....
statement as well as if you receive an error. You can make the cache duration of each different so if you successfully get the data, cache it longer than if you got an error. One implementation would look like this:
object Twitter = Cache["MyTwitter"];
if(Twitter==null)
{
// cache empty
try
{
var latestItems = (load items)
Cache.Insert("MyTwitter", latestitems, null, DateTime.Now.AddSeconds(600),
Cache.NoSlidingExpiration);
Twitter = latestitems;
}
catch(Exception ex)
{
Cache.Insert("MyTwitter", ex.ToString(), null, DateTime.Now.AddSeconds(60),
Cache.NoSlidingExpiration);
Twitter = ex.ToString();
}
}
if(Twitter is string)
{
phError.Visible = true;
}
else
{
rptTwitter.DataSource = Twitter;
// rest of data binding code here
}
There are two parts here. The first part is to check the cache and if the object is not in the cache, do your loading. If there's an error just store a string in the cache.
Then with you object, if it's a string you know you've got an error. Otherwise it's the result of retrieving the twitter feed.
Related
I am new to ASP.Net Web API. I need to achieve the following. I have two web API end points
http://someurl.azurewebsites.net/api/recipe/recipes This returns the recipes that are available.
[ID, Name, Type]
http://someurl.azurewebsites.net/api/recipe/{ID} This returns the recipe with [ID, Ingredients, Cost....]
I need to build a application to get the cheapest recipe in a timely manner.
I am able to get the desired result using the following code but at times it crashes and throws the following exception System.Threading.Tasks.TaskCanceledException: A task was canceled.
How can I achieve this in an efficient manner both using Controller or Javascript.
public ActionResult Index()
{
List<Receipe> receipelist = new List<Receipe>();
var baseAddress = "http://someurl.azurewebsites.net/api/recipe/recipes";
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("x-access-token", "sjd1HfkjU83");
using (var response = client.GetAsync(baseAddress).Result)
{
if (response.IsSuccessStatusCode)
{
var jsonString = response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
var Receipes = JsonConvert.DeserializeObject<List<Receipe>>(jsonString.Substring(jsonString.IndexOf("Receipes") + 8, (jsonString.Length - jsonString.IndexOf("Receipes") - 9)));
if (Receipes != null)
{
foreach (Receipe Receipe in Receipes)
{
var baseAddress1 = "http://someurl.azurewebsites.net/api//api/recipe/" + Receipe.ID;
using (var client1 = new HttpClient())
{
client1.DefaultRequestHeaders.Add("x-access-token", "sjd1HfkjU83");
using (var response1 = client1.GetAsync(baseAddress1).Result)
{
var jsonString1 = response1.Content.ReadAsStringAsync().GetAwaiter().GetResult();
receipelist.Add(JsonConvert.DeserializeObject<Receipe>(jsonString1));
}
}
}
}
}
else
{
Console.WriteLine("{0} ({1})", (int)response.StatusCode, response.ReasonPhrase);
}
}
}
return View(receipelist);
}
There are several ways to go about this. Without experimenting, it is not really possible to say which would be the most efficient way. Or may be we really don't need to be so efficient. Consider the following steps:
Try to cache the result of the first response, i.e list of recipes available (Endpoint #1). You could cache that for 'x' timespan depending on your business requirement. That will most likely save you a first trip. My guess is that the 'list of available recipes' will not change very often; so you could probably cache that for a while(hours/days?). You could use a dictionary object as a data structure and cache this.
If you have control of the Endpoint #2, then I'd recommend providing an api that takes in 'list of ids' instead of just one id. So in essence you are asking Endpoint#2 to return 'a list of recipes with price' in one api call rather than looping for each recipe at a time. You could probably cache this data as well, depending on how often the price changes. My guess is this won't change very often either. When you get the 'list of prices with id', efficiently plug the price info to existing dictionary.
When you hit an exception when making an api calls, you could always return the stale data to the users instead of not displaying any data/errors. Be sure to let the users know by labeling in the UI something like "Recipe as of 2 hours ago".
With the above changes, your application should be able to perform well and in the worst case, display stale data.
Keep in mind that depending on the problem domain, we don't always have to be up to the second/minute consistent with the data. In real life and in reality the data on the internet is always stale. For example, by the time your end users sees the price and decides to click to buy a recipe, the price of the recipe might have already changed!
Hope this helps.
I wrote a HttpModule to intercept, evaluate and authorize requests, checking if logged user has appropriate access to the url being requested, in a pretty old legacy system written in ASP.NET 2.0(Web pages, not Web app), whose customer does not want to port to a newer framework. Restrictions have been loaded and cached at login time.
Everything works fine, except when some page contains an <asp:MultiView> component or when there is a button that launch an ajax method. When one of these situations occur, and user doesn't have rights to access that url, an alert box pops up with an "Unknown error" message, that came from a ThreadAbortException thrown by Response.End() method.
The question is: Why does my "Unauthorized" message is being overwritten by "Unknown Error" from the exception, only on these two situations?
Is there a way of doing an Url Authorization system, using database and caching and without cluttering Web.config with roles like those older ASP.NET samples?
// My module init method.
public void Init(HttpApplication context)
{
context.PreRequestHandlerExecute += new EventHandler(context_PreRequestHandlerExecute);
// PreRequestHandlerExecute is the first stage at ASP.NET Pipeline
// where we could get a fulfilled Session variable
}
private void context_PreRequestHandlerExecute(object sender, EventArgs e)
{
HttpApplication application = (HttpApplication)sender;
HttpContext context = application.Context;
// additional request filtering/validation/etc.
LoggedUser user = (LoggedUser)application.Session["user"];
string path = context.Request.Path;
// more checks and rules...
if (!checkUserAuthorization(path, user))
{
context.Response.Write("<script>alert('Unauthorized. Contact your manager.');</script>");
context.Response.Write("<script>window.history.back();</script>");
context.Response.StatusCode = 403;
context.Response.End();
}
}
EDIT: What I've already tried (with no goal):
Response.OutputStream.Close();
Response.Flush();
HttpApplication.CompleteRequest();
it's by design. you must ignore it and add a catch for that exception.
try {
context.Response.End();
}
catch{}
Foreword
After a lot of research, finally I got it. Considering ASP.NET 2.0, concerning AJAX operations, the project I'm working uses a Microsoft component called "Atlas", which in turn got renamed to ASP.NET AJAX. At the time this system was written, the developers used the beta ASP.NET AJAX (codename "Atlas") to address all ajax and partial rendering needs.
I needed to dig deeper in source code (thanks to Reflector), to understand and inspect from where does that "Unknown Error" comes.
Inside the Microsoft.Web.Atlas, there is a file named Microsoft.Web.Resources.ScriptLibrary.*.Atlas.js (where * could be Debug or Release) which is rendered at runtime through a WebResource.axd "proxy".
This javascript file have a bug, because it expects to ASP.NET request always return an HTTP 200 (OK) response code, which in my code it's not happening (I'm returning a 403 Forbidden code at my module).
Code
From Microsoft.Web.Resources.ScriptLibrary.*.Atlas.js taken from WebResource.axd:
this._onFormSubmitCompleted = function(sender, eventArgs) {
var isErrorMode = true;
var errorNode;
var delta;
if (sender.get_statusCode() == 200) {
delta = sender.get_xml();
if (delta) {
errorNode = delta.selectSingleNode("/delta/pageError");
if (!errorNode) {
isErrorMode = false;
}
}
}
if (isErrorMode) {
if (errorNode) {
pageErrorMessage = errorNode.attributes.getNamedItem('message').nodeValue;
}
else {
pageErrorMessage = 'Unknown error';
}
this._enterErrorMode(pageErrorMessage);
return;
}
// Code continues.
}
From this code, we can see that since response code is not an 200 OK, that errorNode variable won't be set, and this if (errorNode) statement will always be false.
In this case, I was left with two options: Always return HTTP 200 and modify all pages that have an <atlas:ScriptManager> with and add an ErrorTemplate tag on each, or supersede that script with one that consider non-HTTP 200 responses, loading it below </form> tag at the Master page.
There is a lot of tutorials on how to do a proper error handling when using ScriptManager and UpdatePanels (an official one here), by subscribing to the AsyncPostBackError event), but this beta version (Atlas) simply don't have this event.
Please bear with me for this slightly long winded description but I'm having a strange problem with C# screen scraping an ASP.NET web forms page. The steps I'm trying to do are as follows:-
1) The site is secured using basic authentication over HTTPS so I need to login appropriately.
2) I'm performing a GET request on the page to retrieve the __VIEWSTATE value (darn thing does nothing if I don't set this thing!)
3) Once logged in there are several form fields to complete then a submit button which POST's the form to the server
4) When the submit button is pressed the form is POST'd to the server and response is the same page and form but now with an extra little HTML table at the bottom of the form with some data I need to get at.
I've so far managed to sort the login and form post using the WebClient class. I've used fiddler (and firebug) to check the POST field values that are being sent when completing the form normally using a browser. I can successfully get a response from the POST request with the data table in question appearing below the form as expected. The problem however is that although the table is populated with data it is populated with data I don't expect. The data that appears is if I completed the form in a browser as normal but with one particular parameter (a drop down list) set to a different value than I'm passing in my POST request to the server. I've confirmed using fiddler and firebug that I'm passing exactly the same POST parameters that are sent as normal using a web browser human completed form. I'm now totally stuck as to why this one parameter is not being 'taken into consideration' by the server?
The one difference is that this particular control is a select list and it performs a page reload or 'postback' when changed. However this doesn't seem to do anything apart from change some other select lists content later in the form.
I guess I'm asking is there anything else I'm missing that would cause this? I'm totally tearing my hair out on this one. Can anyone help? I've posted the code below (with addresses and parameters blanked out for privacy).
// a place to store the html
string responseBody = "";
// create out web client to handle the request
using (WebClient webClient = new WebClient())
{
// space to store responses from the remote site
byte[] responseBytes;
// site uses basic authentication over HTTPS so we'll need to login
CredentialCache credentials = new CredentialCache();
credentials.Add(new Uri(Url), "Basic", new NetworkCredential(Username, Password));
// set the credentials in the web client
webClient.Credentials = credentials;
// a place for __VIEWSTATE
string viewState = "";
// try and get __VIEWSTATE from the web site
try
{
responseBytes = webClient.DownloadData(Url);
viewState = GetHtmlInputValue(Encoding.UTF8.GetString(responseBytes), "__VIEWSTATE");
}
catch (Exception e)
{
bool cancel = false;
ComponentMetaData.FireError(10, "Read web page data", "Error whilst trying to get __VIEWSTATE from web page: " + e.Message, "", 0, out cancel);
}
// add our POST parameters (don't forget the __VIEWSTATE or it won't work as its an ASP.NET web page)
NameValueCollection requestParameters = new NameValueCollection();
// add ASP.NET fields
requestParameters.Add("__EVENTTARGET", __EVENTTARGET);
requestParameters.Add("__EVENTARGUMENT", __EVENTARGUMENT);
requestParameters.Add("__LASTFOCUS", __LASTFOCUS);
// add __VIEWSTATE
requestParameters.Add("__VIEWSTATE", viewState);
// all other form parameters
requestParameters.Add("btnSubmit", btnSubmit);
/* I've hidden the rest of the parameters hidden for privacy just in case */
// see if we can connect and get data
try
{
// set content type
webClient.Headers.Clear();
webClient.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
// 'POST' the form data using web client and hope we get a response
responseBytes = webClient.UploadValues(Url, "POST", requestParameters);
// transform the response to a string
responseBody = Encoding.UTF8.GetString(responseBytes);
}
catch (Exception e)
{
bool cancel = false;
ComponentMetaData.FireError(10, "Read web page data", "Error whilst trying to connect to web page: " + e.Message, "", 0, out cancel);
}
}
Please ignore the 'ComponentMetaData' references as this is part of SSIS script source.
Any ideas or help will be greatly appreciated - cheers!
RE: thanks for the quick responses, all I can say to those comments is...
There's the normal ASP session cookie but there's no values in the cookie (apart from the session ID of course), I figured as the site is using basic authentication not forms authentication I could just ignore the cookie - and as I'm getting into the site and getting data returned this was ok. I guess it's worth a try but I'll have to just alter the code to use the WebRequest class method instead...
As for the select list javascript, no there's no javascript changing the value of the select list after page load. The only javascript on the select list is an onchange event to do a 'postback' which only seems to change some other select lists on the form that are empty anyway in the final POST. Note I'm including all the POST parameters when generating the POST request even if they're empty and I'm also including all the 'web forms' special fields such as __VIEWSTATE, __EVENTTARGET etc...
I'm no expert in web forms (MVC man myself) but is there anything else that the web forms 'engine' is expecting? I've sent 1 header for the 'Content-Type' of 'application/x-www-form-urlencoded' but I've tried setting others such as copying the 'User-Agent' header from the original POST but this ends up with me getting a 500 error from the server, not sure why that would happen??
Here's the code for the 'GetHtmlInputValue' its a bit simple/basic and could be done better but:-
private string GetHtmlInputValue(string html, string inputID)
{
string valueDelimiter = "value=\"";
int namePosition = html.IndexOf(inputID);
int valuePosition = html.IndexOf(valueDelimiter, namePosition);
int startPosition = valuePosition + valueDelimiter.Length;
int endPosition = html.IndexOf("\"", startPosition);
return html.Substring(startPosition, endPosition - startPosition);
}
If I understand you correctly, then selecting an item in the dropdown will cause a POST to be performed, and the server alters the available options in another part of the form. The server will then include the current value of the dropdown in the __VIEWSTATE field value.
When you perform the scraping, you should make sure that the __VIEWSTATE contains the desired value for the dropdown. To investigate further, try to decode the viewstate from the server and see which values are sent back.
This question already has answers here:
Google Weather API gone?
(5 answers)
Closed 6 years ago.
I decided to pull information from Google's Weather API - The code I'm using below works fine.
XmlDocument widge = new XmlDocument();
widge.Load("https://www.google.com/ig/api?weather=Brisbane/dET7zIp38kGFSFJeOpWUZS3-");
var weathlist = widge.GetElementsByTagName("current_conditions");
foreach (XmlNode node in weathlist)
{
City.Text = ("Brisbane");
CurCond.Text = (node.SelectSingleNode("condition").Attributes["data"].Value);
Wimage.ImageUrl = ("http://www.google.com/" + node.SelectSingleNode("icon").Attributes["data"].Value);
Temp.Text = (node.SelectSingleNode("temp_c").Attributes["data"].Value + "°C");
}
}
As I said, I am able to pull the required data from the XML file and display it, however if the page is refreshed or a current session is still active, I receive the following error:
WebException was unhandled by user code - The remote server returned
an error: 403 Forbidden Exception.
I'm wondering whether this could be to do with some kind of access limitation put on access to that particular XML file?
Further research and adaptation of suggestions
As stated below, this is by no means best practice, but I've included the catch I now use for the exception. I run this code on Page_Load so I just do a post-back to the page. I haven't noticed any problems since. Performance wise I'm not overly concerned - I haven't noticed any increase in load time and this solution is temporary due to the fact this is all for testing purposes. I'm still in the process of using Yahoo's Weather API.
try
{
XmlDocument widge = new XmlDocument();
widge.Load("https://www.google.com/ig/api?weather=Brisbane/dET7zIp38kGFSFJeOpWUZS3-");
var list2 = widge.GetElementsByTagName("current_conditions");
foreach (XmlNode node in list2)
{
City.Text = ("Brisbane");
CurCond.Text = (node.SelectSingleNode("condition").Attributes["data"].Value);
Wimage.ImageUrl = ("http://www.google.com/" + node.SelectSingleNode("icon").Attributes["data"].Value);
Temp.Text = (node.SelectSingleNode("temp_c").Attributes["data"].Value + "°C");
}
}
catch (WebException exp)
{
if (exp.Status == WebExceptionStatus.ProtocolError &&
exp.Response != null)
{
var webres = (HttpWebResponse)exp.Response;
if (webres.StatusCode == HttpStatusCode.Forbidden)
{
Response.Redirect(ithwidgedev.aspx);
}
}
}
Google article illustrating API error handling
Google API Handle Errors
Thanks to:
https://stackoverflow.com/a/12011819/1302173 (Catch 403 and recall)
https://stackoverflow.com/a/11883388/1302173 (Error Handling and General Google API info)
https://stackoverflow.com/a/12000806/1302173 (Response Handling/json caching - Future plans)
Alternative
I found this great open source alternative recently
OpenWeatherMap - Free weather data and forecast API
This is related to a change / outage of the service. See: http://status-dashboard.com/32226/47728
I have been using Google's Weather API for over a year to feed a phone server so that the PolyCom phones receive a weather page. It has run error free for over a year. As of August 7th 2012 there have been frequent intermittent 403 errors.
I make a hit of the service once per hour (As has always been the case) so I don't think frequency of request is the issue. More likely the intermittent nature of the 403 is related to the partial roll-out of a configuration change or a CDN change at Google.
The Google Weather API isn't really a published API. It was an internal service apparently designed for use on iGoogle so the level of support is uncertain. I tweeted googleapis yesterday and received no response.
It may be better to switch to a promoted weather API such as:
WUnderground Weather or
Yahoo Weather.
I have added the following 'unless defined' error handling perl code myself yesterday to cope with this but if the problem persists I will switch to a more fully supported service:
my $url = "http://www.google.com/ig/api?weather=" . $ZipCode ;
my $tpp = XML::TreePP->new();
my $tree = $tpp->parsehttp( GET => $url );
my $city = $tree->{xml_api_reply}->{weather}->{forecast_information}->{city}->{"-data"};
unless (defined($city)) {
print "The weather service is currently unavailable. \n";
open (MYFILE, '>/home/swarmp/public_html/status/polyweather.xhtml');
print MYFILE qq(<?xml version="1.0" encoding="utf-8"?>\n);
print MYFILE qq(<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "xhtml11.dtd">\n);
print MYFILE qq(<html xmlns="http://www.w3.org/1999/xhtml">\n);
print MYFILE qq(<head><title>Weather is Unavailable!</title></head>\n);
print MYFILE qq(<body>\n);
print MYFILE qq(<p>\n);
print MYFILE qq(The weather service is currently unavailable from the data vendor.\n);
print MYFILE qq(</p>\n);
print MYFILE qq(</body>\n);
print MYFILE qq(</html>\n);
close MYFILE;
exit(0);
}...
This is by no means a best practice, but I use this API heavily in some WP7 and Metro apps. I handle this by catching the exception (most of the time a 403) and simply re-calling the service inside of the catch, if there is an error on the Google end it's usually briefly and only results in 1 or 2 additional calls.
That`s the same thing we found out.
Compare the request header in a bad request and a working request. The working request includes cookies. But where are they from?
Delete all your browser cookies from google. The weather api call will not work in your browser anymore. Browse to google.com and then to the weather api, it will work again.
Google checks the cookies to block multiple api calls. Getting the cookies one time before handling all weather api requests will fix the problem. The cookies will expire in one year. I assume you will restart your application more often then once a year. So that you will get a new one. Getting cookies for each request will end in the same problem: Too many different requests.
One tip: Weather does not often change, so cache the json information (for maybe a hour). That will reduce time-consuming operations as requests!
I found that If you try the request in a clean browser (like new window incognito mode on chrome) the google weather service works. Possible problem of cookies?
I need to write a simple WinForms apps that can be fired to test if a website is still alive and that that website is able to read from a database.
I am using the whole "(HttpWebResponse)myHttpWebRequest.GetResponse()" thing in c# to test whether the site is alive, but I am at a lose for how to get a test page in my website to write something to the "Response" to indicate that it was able to test it's own connectivity to the database.
Here is the sample code for my Winforms side (ripped from the MSDN):
private void CheckUrl()
{
try
{
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create("http://www.google.com");
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
myHttpWebResponse.Close();
label1.Text = myHttpWebRequest.Address.AbsoluteUri;
}
catch (WebException e)
{
label1.Text = "This program is expected to throw WebException on successful run." +
"\n\nException Message :" + e.Message;
if (e.Status == WebExceptionStatus.ProtocolError)
{
label1.Text = String.Format("Status Code : {0}", ((HttpWebResponse)e.Response).StatusCode);
label2.Text =String.Format("Status Description : {0}", ((HttpWebResponse)e.Response).StatusDescription);
}
}
catch (Exception e)
{
label1.Text = e.Message;
}
}
I was hoping for some help on the webform side of things to return to the above code.
Thanks for any help that you folks can provide.
Richard
You can create a webservice inside of the project called IAMALIVE and have it return a single char.
On your WinForms area, consume said WebService and if it works, your site is alive.
In the essence of Papuccino's answer: you can actually create web services that are embedded in the C# code-behind of your WebForms pages by marking them with the [WebMethod] attribute. Those will reside within the web application, not just the server.
What happens when your site fails? Does it return a 500 status code or timeout?
Another way to look at it: does it always do something expected if it succeeds?
You might call a URL in your web app that you know will either return a 200 response code or will have some expected HTML markup in the response if things are working fine.
Have your winform call this URL and examine the Response.status or the text in the output buffer for your expected markup. You should also create a timeout in your httprequest. If the page does not load within the timeout, you will get a web exception and you will know the site is failing.
Also, if you have the budget, there are external monitoring services like gomez.com that can automate this and provide reporting on site availability.
have your webform page open a database connection and perform something simple/low-impact, e.g.
select SystemTableId from dbo.[SystemTable] where SystemTableId = 1
where SystemTable is a single-row table.
If the page throws an exception for any reason, Response.Write the exception message, otherwise Response.Write("SUCCESS") or similar.