Both WebClient and HttpWebRequest suddenly failing to pull basic Amazon pages? - c#

We have an in-house application that pulls some data from some of Amazon's pages occassionally (We know they have APIs for certain operations... what we're doing requires some custom info not included in the APIs). We have never had a problem pulling their pages, but suddenly Amazon is returning "(503) Server Unavailable" on pretty much every request, and this has been happening for several days, so we doubt it's a temporary thing. Even something as simple as this:
System.Net.WebClient client = new System.Net.WebClient();
string data = client.DownloadString(new Uri("http://www.amazon.com/Bose-Companion-multimedia-speaker-Graphite/dp/B000HZBR64/"));
The strange thing is that these pages load just fine in a web browser, but any time we try to pull them through code, it is failing.
What could cause these functions to fail? Is it possible that they changed something on their end and that we need to do some custom logic with our calls?

After some further testing, it turns out that this was happening because Amazon needs the Accept parameter of the HttpWebRequest to be specifically set. When setting it to:
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
Everything worked fine. This is a recent change, so they must have altered something on their end.

check the user-agent of you request. make the user-agent the same as your browser. And check if you set any proxy for your app? maybe your browser and your app are using different proxies

Related

C# calling a webservice result in 401

I'm trying to call a simple web service to get long and lat of an address, which works when I try it manually :
https://services.gisgraphy.com/geocoding/?address=paris
But with the code, I get 401 unauthorize.. What am I doing wrong ?
WebRequest request = WebRequest.Create("https://services.gisgraphy.com/geocoding/?address=paris");
request.Credentials = CredentialCache.DefaultCredentials;
request.Method = "GET";
WebResponse response = request.GetResponse(); // It happens here.
What am I doing wrong?
You haven't read the big blue bar on top of the website belonging to the API you're calling:
Gisgraphy is open sources and only use open data. This server is for demonstration only. Test it, Play with webservices but then Install Gisgraphy locally or subscribe to premium hosted services
So they're probably detecting that you're calling their API from outside their playground, and are denying you to do so.
So, either install it locally, or subscribe to their hosted services. The latter probably gives you an API key that allows you to make API calls.
Of course you can fake your way around this by imitating that your request comes from a browser, for example with User-agent and Accept headers, but surely they'll try and detect this and block your IP address entirely. Just pay up, or host it locally.
Few things on this question :
Check if the URL/web-service is directly accessible via
browser/postman tool.
Secondly , as one of my fellow friend there mentioned this
web-service has some security protocol. May be its not allowing
requests from outside (i.e. other than its domain to come in)
Once you drill down the above two possibility ,you should be in a good shape to move ahead.

HttpClient one response from different countries

I have a simple aplication in Windows Store.
This application download and parse HTML from website.
I using a HttpClient class
Now I have a big problem becouse a page looks diffrent form specific countries and my parsing is not success.
Example: When someone from USA using my app then app downloading diffrent HTML content becouse webpage looks diffrent in specific countries.
How to set a default location in http client?
I want to have a the same HTML in all executes.
EDIT
I calling this page: LINK
You need to set the default language header when you make the request and/or consider making it a user definable setting.
http://www.w3.org/TR/WCAG20-TECHS/SVR5
ignoring the initial question for a moment
PLEASE don't write an app that depends on any kind of HTML parsing for any functionality. All the site you are calling has to do is change an ID or two in the "wrong" place and your app will fail for every user until you put out an update.
back to the answer
OK, assuming that screen-scraping is the way you want to go with your app, and assuming, of course, that the site you are scraping from allows such behaviour in their terms of use (check - it wouldn't be fun for you to get sued if you didn't read them) then I'd suggest a slightly different approach.
Since you are not guaranteed to get the same page layout for any locale your users access your app from, why not set up a web service that does the parsing work for you, and interrogate that service from your app instead of going direct to the site?
Your app <--> Your web service <--> the site providing data
That way, you always know that the data you are getting back is consistently formatted as if for a specific locale (your web server), and then you only have to maintain one piece of code to parse it. That will be much simpler whenever there is a change to the underlying data structure (and believe me, there will be changes)
The answer to this depends on how the website implements default language selection. Both of the other answers are potentially correct depending on how the specific site works.
If you can share the site URL, we can tell you a suitable strategy to use.
Setting the design flaw consideration aside for a moment (you may have or have not good reason to do screen scraping), here's how to set the Accept-Language header:
var httpClient = new HttpClient();
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, new Uri("http://www.livescore.com"));
httpRequestMessage.Headers.Add("Accept-Language", "en");
var response = await httpClient.SendAsync(httpRequestMessage);
string content = await response.Content.ReadAsStringAsync();
Try to always call the url in question with the cultureInfo path param if it has one, for example say that you are targeting microsoft.com then you would have something like this:
http://www.microsoft.com/en-us/default.aspx for english
http://www.microsoft.com/de-DE/default.aspx for german
and so on. If this is applicable to you, this would be an ideea.

Testing a REST Logging Web Service using C#

I've created a pretty basic REST Web Service in VS2012 using C#. The approach I've taken is seen below in one example:
[OperationContract]
[WebInvoke(Method = "GET",
ResponseFormat = WebMessageFormat.Json,
BodyStyle = WebMessageBodyStyle.Wrapped,
UriTemplate = "OpenSession/{key}/{source_userid}/{source_id}/{source_version}/{source_ip}")]
string OpenSession(string key, string source_userid, string source_id, string source_version, string source_ip);
The code works just fine and the service is up and running. I get responses, the tokens are handed out, etc. I figured wrapping the data in the URL was just fine as the service is built for logging, so in the end, calls would be (for example):
www.service.site/service.svc/2/5/12/HELLO
(BTW if there's a better way to do this while keeping it a REST service please let me know)
The problem I have is... how do I test this properly? I need to simulate 1,000 concurrent hits to this service and have built a custom C# Windows Application to do this, but I'm encountering a number of problems.
First, I have no idea how to use HttpWebRequest and the stream readers/writers to send/receive data properly in a manageable format. I see The requests have GET and PUT methods, etc. but I honestly can't seem to make heads or tails of separating the post data vs. the URL information. AFAIK I'm not really posting any data as all my data is embedded in the URL request itself; is this correct?
Also, I have no idea what "content type" to set the HttpWebRequest to in order to be compatible with my Operation Contract (I can see the enumeration of JSON but what's the "text" supposed to be?)
And on top of that, HttpWebRequest is monstrously slow. When I open a URL directly to the service (e.g. "http://localhost:51849/Scribe.svc"), it's fast, but the second I try to call a method (e.g. ""http://localhost:51849/Scribe.svc/GetStatus"), I get a 1.5-2 second delay (and I've done all the proxy fixes to ensure it's as fast as possible).
Any help would be greatly appreciated.
I don't know if you are aware of this utility:
SoapUI
You can use it to reference your service and generate the requests to test your service.
The installation is pretty straightforward. After installed, you have to click on file/new SoapUI Project.
Then, in the popup window, you fill in the "Initial WSDL/WADL:" field with your service's wsdl, like this:
http://localhost:51849/Scribe.svc?wsdl
It'll generate the client to correctly do the calls to your service.
Using soap ui you can even test doing a bunch of calls to stress up your service.
This is a good way to test your services.
AFAIK I'm not really posting any data as all my data is embedded in the URL request itself; is this correct?
Yes. That's how a GET request works. The data is passed on your url. You should only use GET requests when not changing the state of the data on the server, otherwise you should use post.
PUT methods you'll use when you want to create/update the data on the server.
I hope it helps.
Regards,
Rodrigo
Maybe you should consider using some performance tool.
There are plenty and them, should be easy to find one suitable for your pourpose.
Jmeter could be a good option.
This is a guide that show how to use Jmeter to test a REST Web Service.
Hope it can help.
Thanks for the responses! I actually have the testing system working now and can slam my service with thousands of timer-controlled sessions. The key to getting around the PUT request issue was to use empty settings:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.Method = "PUT";
req.ContentType = "application/json";
req.ContentLength = 0;
I did look at those tools, but for my specific needs, I found writing a custom timer-controlled app was the best approach.
I'm not sure if the delay is still problematic, or possibly even linked to the debug state. once I publish the service to Azure and test it remotely, I'll get a better idea if there's any latency issues with connectivity which are service based...

HTML message or login pages intercepting web service requests

I am busy working on a mobile application that retrieves data from a web service.
Ofcourse everything is working perfectly, I get everything I need to and I can consume the services without much effort... on the emulator.
However, when I move over to testing this application on the device, instead of getting back the data that I am expecting I am getting a website returned. How am I supposed to handle this?
Currently I am using this to call my service: (using system.net - I dont know if this is what I should be using on windows phone 7 either)
WebClient proxy = new WebClient();
string strURI = "http://www.google.co.za";
proxy.OpenReadCompleted +=
new OpenReadCompletedEventHandler(proxy_openreadcompleted);
proxy.OpenReadAsync(new Uri(strURI));
Please note: I am not really calling google, it is just an example. So anyways I am expecting my JSON to be returned, instead I am getting a message from the service provider to change mobile options... I can put this into isolated storage and render it with a browser, however I do not know what the source of the message is so when you click on a button, the forms use relative URLs, so instead of it doing what it is intened to do I just see what it is trying to call.
Is there anyway to get the source of the response? I am looking for a source like http://vodafonelive.mobi/ or something like that. If someone can tell me what to do I would greatly appreciate it, my current thinking is that if I can identify the source I can create a webbrowser task so that my application does not need to handle this, however... since I am calling a specific source I don't know how to identify where the response is comming from.
Any help is appreciated.
This is most likely due to differences in the user agent of the emulator and the device. Check what is sent or set this explicitly to ensure that the 2 behave in the same way. To ensure the server doesn't try and redirect to a different location.
Alternatively, it could be a mobile operator proxy being "helpful" and adjusting the request that goes over their network.
Ok so after spending some time on this, I finally found a way that I could get a response uri (where the response was comming from) by using another method to do the call to the service:
So basically this is the call:
WebRequest request = HttpWebRequest.Create(strURI);
var result = (IAsyncResult)request.BeginGetResponse(ResponseCallback, request);
So in the function ResponseCallBack function I do something like this:
WebRequest request = (HttpWebRequest)result.AsyncState;
WebResponse response = request.EndGetResponse(result);
which then allows me to check the response uri (the source of the intercepted message) and have the native browser handle the html that I was not expecting.
WebBrowserTask webBrowserTask = new WebBrowserTask();
webBrowserTask.Uri = response.ResponseUri;
Thanks for the help though, hopefully this will help someone with a similar issue.

Integrate a C# client into a node.js + socket.io chat app

As part of learning node.js, I just created a very basic chat server with node.js and socket.io. The server basically adds everyone who visits the chat.html wep page into a real time chat and everything seems to be working!
Now, I'd like to have a C# desktop application take part in the chat (without using a web browser control :)).
What's the best way to go about this?
I created a socket server in nodejs, and connected to it using TcpClient.
using (var client = new TcpClient())
{
client.Connect(serverIp, port));
using (var w = new StreamWriter(client.GetStream()))
w.Write("Here comes the message");
}
Try using the HttpWebRequest class. It is pretty easy to use and doesn't have any dependencies on things like System.Web or any specific web browser. I use it simulating browser requests and analyzing responses in testing applications. It is flexible enough to allow you to set your own per request headers (in case you are working with a restful service, or some other service with expectations of specific headers). Additionally, it will follow redirects for you by default, but this behavior easy to turn off.
Creating a new request is simple:
HttpWebRequest my_request = (HttpWebRequest)WebRequest.Create("http://some.url/and/resource");
To submit the request:
HttpWebResponse my_response = my_request.GetResponse();
Now you can make sure you got the right status code, look at response headers, and you have access to the response body through a stream object. In order to do things like add post data (like HTML form data) to the request, you just write a UTF8 encoded string to the request object's stream.
This library should be pretty easy to include into any WinForms or WPF application. The docs on MSDN are pretty good.
One gotcha though, if the response isn't in the 200-402 range, HttpWebRequest throws an exception that you have to catch. Fortunately you can still access the response object, but it is kind of annoying that you have to handle it as an exception (especially since the exception is on the server side and not in your client code).

Categories

Resources