I'm using an action filter that checks what browser version is being used on my site, if it's an older browser I put up a div at the top asking them to upgrade. I don't want web crawlers to get the div message, so I've implemented HttpBrowserCapabilitiesBase.Crawler and it looks like it works for Google, but Bing and the others don't seem to register as crawlers. Strange for a Microsoft product to not notice Bing as a crawler!
Is there some way to add user agents to the crawler property or something?
Thanks!
Edited: I'm using asp.net mvc 3, it looks like I need to user .Browser files(?). Anyone know of a comprehensive set of .Browser files out there for Bing and the rest of the crawlers?
You will probably need to update your browscap.ini file as the one shipped with IIS is probably old. You can get a new one at one of the following URLs:
http://browsers.garykeith.com/downloads.asp
http://owenbrady.net/browsercaps/
browscap.ini usually lives at: c:\windows\system32\inetsrv\browscap.ini
We're not using MVC but we do this:
Regex.IsMatch(Request.UserAgent, #"bot|crawler|baiduspider|80legs|ia_archiver|voyager|curl|wget|yahoo! slurp|mediapartners-google", RegexOptions.IgnoreCase);
More options in my answer here:
Detecting honest web crawlers
Related
I need a web search be my app and it would be fine to use google web search for.
This works in different cases when I do the code by myself or use assemblies like GAPI.
The problem is that I ever get an error because of paging the results:
ResponseStatus: 400, Reason: out of range start
About this problem I found this by the GAPI developer:
Google AJAX Search API does not allow more than 64 results (e.g. max page is 56).
But I need more results as this is to less for my usage I need at least 500...
Is there another way to get more result by searching the web?
It looks like GAPI is using the old Google Search API which has been officially deprecated as of November 1, 2010.
If you want more results you should look into another library or implementing your own using the Google Custom Search API:
https://developers.google.com/custom-search/v1/overview
I have a subdomain that is http://trade.businessbazaar.in . I am dynamically creating urls from database something in this manner http://trade.businessbazaar.in/mycompany. To display details, I have an index.aspx file there,thinking that on every request the index.aspx page will load and display data accodingly. Also, There is a masterpage on the index.aspx page from where i am capturing the text mycompany and query it in database to fetch result. But nothing seems to work.
A genuine link is http://trade.businessbazaar.in/Symparlife. But its unable to load index.aspx. I need a clean approach without any third party dll or rewriters. Directly to push some lines in config and start working. That is url will be the same but index page will get loaded...
In short, i want to say
I need the StackOverflow type clean url mechanism to fetch pages
Thanks in Advance
You can handle the Begin_Request event in Global.asax and add custom code to redirect to index.aspx and convert the parts of the URL into query string arguments. You should use Server.Transfer to keep the URL in the browser.
I'd recommend upgrading to 4.0 and using the Routing enine though. You should check if the standard routing is available as a download for ASP.NET 3.5. I am sure your code will get messy very soon. Been there, done that.
As #Mike Miller mentions in the comments the Routing engine ships with ASP.NET 3.5. You can check the documentation here - http://msdn.microsoft.com/en-us/library/system.web.routing(v=vs.90).aspx
Here is a tutorial on how to use it with Web Forms - http://weblogs.asp.net/scottgu/archive/2009/10/13/url-routing-with-asp-net-4-web-forms-vs-2010-and-net-4-0-series.aspx
For your case the code would be something like:
routes.MapPageRoute("company-index", "/{company}", "~/index.aspx")
And in index.aspx you can access the route value for company like this:
string company = (string)Page.RouteData.Values["company"];
Keep in mind that you'd better add something in the URL before your actual argument (the company name). If you don't you will have problems later on when because you may want to add a URL like "/Login" but then you will have to validate that users can't create a company named "Login". Not how Stack Overflow has "/questions/" before the actual question info in the URL.
Ok I had a huge Issue giving this a proper title, my excuses for that.
Anyways I have started slowly to look at Web and ASP.NET again, I am a C# developer but I have mostly worked with Windows applications the past 5 years or so, It is not that I haven't touched the web as such in that time, but this is as web services (Restfull as well as the ugly SOAP services) I have also worked with more "raw" web requests.
But I have not worked with IIS or ASP.NET in all that time.
What I would like to do is hos a web page that uses a URL style I could best describe with "like rest", hence the "Restfull urls" title. Because I think most people thinks of such URL's in terms of:
http://example.com/item/
http://example.com/item/23/
and so forth. Not that they have to look like that, however I would like to use such URL's instead of
http://example.com/item?id=23
I know subtext does this, but i have not had any luck finding it in their code base.
Now as far as I can tell I could just implement some IHttpHandler's, but at least for the examples I have seen of that, they write the page source back in code, and I still have master pages etc. I wish to use instead of taking over all that stuff my self, I really just kinda wants to route http://example.com/item/23/ to http://example.com/item and asking for the item with id 23...
I hope this makes sense at all >.<... And that someone has some better examples at hand that what I have been able to find.
You can achieve this using Routing here is a link to an MSDN blog, The .Net Endpoint - Using Routes to Compose WCF WebHttp Services that should get you started.
If you're looking at asp.net/IIS, another option to look at is ASP.Net MVC. It's pretty straight forward to create RESTful services.
Here's a tutorial:
http://www.codeproject.com/Articles/233572/Build-truly-RESTful-API-and-website-using-same-ASP
So here are your options-
For .net 3.5 sp1 framework with IIS7 you can use asp.net routing feature to have MVC style urls that you mentioned should create a custom route handler implementing IRouteHandler interface as explained here How to: Use Routing with Web Forms and register your route rules in Application_Start method in Global.asax. For your example you can register a route like this
routes.Add("ItemRoute", new Route
(
"item/{itemId}",
new CustomRouteHandler("~/item.aspx")
));
and then you can access itemId in your routed item.aspx page by checking request context item
requestContext.HttpContext.Items["itemId"]
For .net framework 4 MVC you dont have to create a custom handler, you can directly use
routes.MapPageRoute("ItemRoute", "item/{itemId}", "~/item.aspx");
in you global.asax Application_Start method.
This link explains more about the Routing
A way of achieve this is using URL rewriting.
If you're planning to host your Web application in Internet Information Services 7.x, you can take advantage of IIS URL Rewriting Module:
http://www.iis.net/download/urlrewrite
URL rewriting is just mapping a friendly URL to an unfriendly, common one, which is programming-friendly to inspect GET parameters.
For example:
http://yourdomain.com/item/48 => http://yourdomain.com/Items.aspx?Id=48
I use DotNetOpenAuth in my Asp.Net application.
I want to create a login page where user can choose among most common OpenID providers like google, yahoo...
I'd like to have image buttons like SO login.
1) Is there a best practice to achieve this? (Like a free/open asp.net control)
2) Do you have a list of common Discovery URLS?
https://www.google.com/accounts/o8/id (google)
https://me.yahoo.com (yahoo)
...
update
Just for info I've found also this:
socialauth-net
I believe this is what you're looking for: http://code.google.com/p/openid-selector/
You can check out this page which shows how to do OpenID Authentication using ASP.NET/MVC. If you are not using MVC, then check out this page (see cautionary comment below), which shows how to do it in C#/ASP.NET without any MVC.
Also check out StackApps and Stacky which help you implement some of the Stack Overflow type stuff in .NET.
Finally, if you need a list of providers/discovery list, check out this list!
I have a custom site I'm building with automatic url rewriting using a custom engine. The rewriting works fine as long as the page url doesn't end in somehting like .htm or .html. For these pages it goes directly to the iis 404 page instead of hitting my rewriting engine first.
I have the * wildcard handler in the "Home Directory" section of the IIS6 configuration of that website but these urls seem to be ignored by it (altho things like css, jpg, js, etc to get sent to the url handler in my web project). How do i set up IIS6 to force these urls to get sent to the handler, while still serving the page if it exists normally?
The handler basically does this
if (!File.Exists(Request.Path))
{
doMyRewriting();
}
I have to assume that using a block like this (just and example, the real one does some other stuff to format the Request.Path to be proper with everything) should run the "doMyRewriting()" if the requested file does not exist otherwise it will serve the page normally. Am I mistaken?
If I tell IIS specifically to send .htm and .html pages thru to the .NET handler the rewriting works but if the page is actually there it will not serve it.
Any help would be greatly appreciated.
Thanks in advance!
Don't know if you can or would want to do this, but there is the Ionics Isapi url rewriter you can use.
http://www.codeplex.com/IIRF
Basically install that then set a rule to remove the .html that way it hits your rewrite engine. I use it on IIS 6 with several of my blogs.
I think if you are having IIS send all requests to .NET and your handler, then your handler will need to detect if the page exists and serve it instead of rewriting.
UrlRewriting.NET has an option to do this - you might want to look at their code to see how they're handling this case.
In my opinion, rewriting URLs with IIS 6 is best handled with an ISAPI filter written as unmanaged native code. Otherwise, you run into the issues you've mentioned - having to map all extensions to ASP.Net and losing the ability for simple file handling. With an ISAPI filter, you can choose to not rewrite some URLs and let IIS handle them as normal.
To get started, I suggest reading the ISAPI Filter Overview on MSDN.
If your filter absolutely needs the .Net framework runtime, it is possible to write a small ISAPI filter shell that hosts the CLR and forwards the requests to some managed code. The Filter.Net Framework takes this approach and may be suitable for your needs. There is the small drawback to this approach in that you will have to use the same .Net version as any ASP.Net applications that are run in the main IIS process.