I have some code that calls HttpWebRequest's GetResponse() method to retrieve HTML from a URL and return it to the calling method.
This has been working perfectly fine within my Development and QA environments but now that I have uploaded it to my UAT server, I keep getting the following error:
The remote server returned an error: (404) Not Found.
The main difference between Dev/QA and UAT is that UAT uses SSL/HTTPS based URLs whereas Dev/QA uses HTTP. I introduced the following line of code to help progress me a little futher:
ServicePointManager.ServerCertificateValidationCallback = new System.Net.Security.RemoteCertificateValidationCallback(AcceptAllCertifications);
where AcceptAllCertifications always returns true but I still get my 404 error.
I that people who previously had this error have been able to resolve the issue by merely ensuring the URI used for the HttpWebRequest doesn't have a slash at the end (see: Simple HttpWebRequest over SSL (https) gives 404 Not Found under C#) but this does not make a difference to me.
I have now tried what was suggested at this post (see: HttpWebResponse returns 404 error) where I render the exception on the page. This bypassed the yellow-warning screen and gives me a bit more informtion, including the URL it is trying to get a response from. However, when I copy and paste the URL into my browser, it works perfectly fine and renders the HTML on the page. I'm quite happy therefore that the correct URL is being used in the GetResponse call.
Has anyone got any ideas as to what may be causing me this grief? As said, it only seems to be a problem on my UAT server where I am using SSL.
Here is my code to assist:
public static string GetHtmlValues()
{
var webConfigParentUrlValue = new Uri(ConfigurationManager.AppSettings["ParentUrl"]);
var destinationUrl = HttpContext.Current.Request.Url.AbsoluteUri;
var path = "DestinationController" + "/" + "DestinationAction" + "?destinationUrl=" + destinationUrl;
var redirect = new Uri(webConfigParentUrlValue, path).AbsoluteUri;
ServicePointManager.ServerCertificateValidationCallback = new System.Net.Security.RemoteCertificateValidationCallback(AcceptAllCertifications);
var request = (HttpWebRequest)WebRequest.Create(redirect);
//Ensures that if the user has already signed in to the application,
// their authorisation is carried on through to this new request
AttachAuthorisedCookieIfExists(request);
HttpWebResponse result;
try
{
result = (HttpWebResponse)request.GetResponse();
}
catch (WebException ex)
{
result = ex.Response as HttpWebResponse;
}
String responseString;
using (Stream stream = result.GetResponseStream())
{
StreamReader reader = new StreamReader(stream, Encoding.UTF8);
responseString = reader.ReadToEnd();
}
return responseString;
}
More details of the error as it is rendered on the page:
I ran into a similar situation, but with a different error message. My problem turned out to be that my UAT environment was Windows 2008 with .NET 4.5. In this environment the SSL handshake/detecting is performed differently than most web browsers. So I was seeing the URL render without error in a web browser but my application would generate an error. My error message included "The underlying connection was closed: An unexpected error occurred on a send". This might be your issue.
My solution was to force the protocol change. I detect the specific error, then I force a change in the security protocol of my application and try again.
This is the code I use:
catch (Exception ex)
{
if(ex.Message.Contains("The underlying connection was closed: An unexpected error occurred on a send."))
{
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
// retry the retrieval
}
}
I finally found the solution to my problem...
The first clue to get me on the right track was the wrong physical path being displayed in the 404 error from IIS. It turns out that this incorrect physical path was mapped to another site in my IIS setup. This particular naturally had a binding also; port 443. As you may know, port 443 is the default port for https.
Now looking at my URL that I was trying to pass into the HTTPWebRequest.GetResponse() method, it looked something like this:
https://www.my-web-site.com
Taking this into account, when this application was hosted on IIS within the bounds of SSL, the error was occuring as follows:
Code enters the aforementioned method GetHtmlValues()
The code gets https://www.my-web-site.com from the web.config file
A response is requested from https://www.my-web-site.com
At this point, as no port has been specified and application is now out there on the open internet, it tries to get a response from https://www.my-web-site.com:443
The problem is, my application isn't hosted via IIS on port 443. A different application lives here. Subsequently, as the page can't be found on port 443, a 404 error is produced.
Now for the solution...
Looking in IIS, I found the port that my application sits on. Let's say port 16523.
Whereas previously in my web.config I had my key of ParentUrl decalred with a value of https://www.my-web-site.com, this is to be changed to http://www.my-web-site.com:16523
Note how the https has become http and the port number is specified at the end. Now when the application tries to get the response, it no longer uses the default ssl port as the correct one was specified.
Related
We are running into a situation where sendAsync post call from a server is not working. Here's my scenario
We have a Web API hosted on a server outside our internal network (DMZ) which has a simple GET implemented to it as
Public HttpResponseMessage Get(strind id){ (API 1)
//do some work
using(HttpClient client = new HttpClient()){
//We Invoke another web API which is hosted inside our network and do a post // as
HttpRequestMessage message = new HttpRequestMessage(HttpMethod.Post, url);
message.Content = new StringContent(someStr);
message.Content.Header.ContentType = new MediaTypeHeaderValue("application/json");
try{
var response = client.SendAsync(message).Result (API 2)
}
catch(Exception e){
//Something
}
}
The post SendAsync().Results fails with the following exception
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) and the message says as One or More errors Occurred.
The innerexception is also not telling anything good. It just InnerException.StackTrace is coming blank and the InnerException.Message is An error occurred while sending request.
In order to test the communication, we ran a GET request from the external sever for API 2 and it worked fine but the POST's are not working.
Everything works fine when both the API's (the one for get and the other one that does the post) are hosted on our internal servers.
Any Suggestions why this might not be working when executed from the external server?
Short answer: Regardless of what I'm betting on, you have to inspect those inner exceptions and find out the exact message.
Longer answer:
The fact that when you host the application in your internal network, the code works (the POST call is successful) and when moved to DMZ it fails strongly indicates the server making the call has no access to the remote endpoint.
I'd bet on the lack of network connectivity between your App Server and the API 2.
Scope:
I am trying to get all my HttpRequests issued via C# to get routed through the TOR Network.
After some quick research I've found some stack overflow questions like This One and This One, so i followed their examples and tried it myself.
Code Sample:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(
"http://whatismyipaddress.com/"
);
request.Proxy = new WebProxy("127.0.0.1:8118");
using (var response = request.GetResponse())
{
using (var reader = new StreamReader(
response.GetResponseStream(),
Encoding.GetEncoding("utf-8")
))
{
string resp = reader.ReadToEnd();
}
}
Results:
I have Privoxy installed and running (netstat -b -a shows it is running/listening on the port 8118).
The Request is not logged onto the Privoxy client, although it seems like it is working.
The Problem:
As the user #Junior Mayhé have pointed out, i have to uncomment this line on the privoxy config file
forward-socks5 / 127.0.0.1:9050
After doing so, my web requests start to get Error 503 - Server Unavailable.
I have tried starting the Tor Browser but it still raises me this error.
What am i doing wrong?
Edit One:
After playing a bit with Netstat -b -a seems like Firefox's Tor is actually running on Port:9151 instead of Port:9050 as stated by these older questions.
After changing the port number on the Privoxy config file to 9151 i no longer get the Server Unavailable error, instead i get a Operation TimedOut. I already increased the value of the request timeout (both connection timeout and readwrite timeout) to 2 minutes and i still get this error.
Maybe missing a period? That's what I have in my config file.
forward-socks5 / 127.0.0.1:9050 .
Use this port: 9150 in privoxy settings. Worked for me
Aright, so here's how it goes I'm trying to set a up a polling system to pull log files from several laser systems each with their own ftp. However, I'm running into difficulty when attempting to call the FtpWebResponse call to download the log file the following is the code I'm using:
// Get the object used to communicate with the server.
FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://192.168.10.140/param.dat");
request.Method = WebRequestMethods.Ftp.DownloadFile;
request.Credentials = new NetworkCredential("user", "pass");
request.UsePassive = false;
request.Proxy = null;
request.UseBinary = true;
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
So I freeze up on that last line with: "The remote server returned an error: (502) Command not implemented."
I've a few different ways to grab files from the system just to see if it's some kind of setting I'm missing this is my results:
Microsoft CMD.exe: Connects up fine and can download files and perform standard ftp commands
Internet Explorer: Entering in address to file it downloads the file just fine
Firefox: "The remote server returned an error: (502) Command not implemented."
Chrome: "Error 606 (net::ERR_FTP_COMMAND_NOT_SUPPORTED): Unknown error."
Now there's not a lot of information I can get on the actual ftp set-up on the laser systems due to a long story I wont get into here but from what I'm seeing perhaps it uses some kind of legacy protocol that IE and CMD support or I'm missing something obvious. I've attempted flipping around the FtpWebRequest setting but nothing seems to work. I would really love to use this solution and not have the program auto build ftp batch files as it would really just make be sad as having everything run in program would be so much more elegant and easier to work with. Any ideas folks?
One of the things that could be causing your 502 error is attempting to use active mode when it is disabled on the server. Try using passive mode:
request.UsePassive = true
Also, from the documentation:
The URI may be relative or absolute. If the URI is of the form
"ftp://contoso.com/%2fpath" (%2f is an escaped '/'), then the URI is
absolute, and the current directory is /path. If, however, the URI is
of the form "ftp://contoso.com/path", first the .NET Framework logs
into the FTP server (using the user name and password set by the
Credentials property), then the current directory is set to
/path.
Try changing your URI to an absolute form - it may help avoid the PWD you're seeing.
Is there a way to get a System.Net.WebRequest or System.Net.WebClient to respect the hosts or lmhosts file?
For example: in my hosts file I have:
10.0.0.1 www.bing.com
When I try to load Bing in a browser (both IE and FF) it fails to load as expected.
Dns.GetHostAddresses("www.bing.com")[0]; // 10.0.0.1
WebRequest.Create("http://10.0.0.1").GetResponse(); // throws exception (expected)
WebRequest.Create("http://www.bing.com/").GetResponse(); // unexpectedly succeeds
Similarly:
WebClient wc = new WebClient();
wc.DownloadString("http://www.bing.com"); //succeeds
Why would System.Net.Dns respect the hosts file but System.Net.WebRequest ignore it? What do I need to change to make the WebRequest respect the hosts file?
Additional Info:
If I disable IPv6 and set my IPv4 DNS Server to 127.0.0.1, the above code works (fails) as expected. However if I add my normal DNS servers back as alternates, the unexpected behavior resumes.
I've reproduced this on 3 Win7 and 2 Vista boxes. The only constant is my company's network.
I'm using .NET 3.5 SP1 and VS2008
Edit
Per #Richard Beier's suggestion, I tried out System.Net tracing. With tracing ON the WebRequest fails as it should. However as soon as I turn tracing OFF the behavior reverts to the unexpected success. I have reproduced this on the same machines as before in both debug and release mode.
Edit 2
This turned out to be the company proxy giving us issues. Our solution was a custom proxy config script for our test machines that had "bing.com" point to DIRECT instead of the default proxy.
I think that #Hans Passant has spotted the issue here. It looks like you have a proxy setup in IE.
Dns.GetHostAddresses("www.bing.com")[0]; // 10.0.0.1
This works because you are asking the OS to get the IP addresses for www.bing.com
WebRequest.Create("http://www.bing.com/").GetResponse(); // unexpectedly succeeds
This works because you are asking the framework to fetch a path from a server name. The framework uses the same engine and settings that IE frontend uses and hence if your company has specified by a GPO that you use a company proxy server, it is that proxy server that resolves the IP address for www.bing.com rather than you.
WebRequest.Create("http://10.0.0.1").GetResponse(); // throws exception (expected)
This works/fails because you have asked the framework to fetch you a webpage from a specific server (by IP). Even if you do have a proxy set, this proxy will still not be able to connect to this IP address.
I hope that this helps.
Jonathan
I'm using VS 2010 on Windows 7, and I can't reproduce this. I made the same hosts-file change and ran the following code:
Console.WriteLine(Dns.GetHostAddresses("www.bing.com")[0]); // 10.0.0.1
var response = WebRequest.Create("http://www.bing.com/").GetResponse(); // * * *
Console.WriteLine(new StreamReader(response.GetResponseStream()).ReadToEnd());
I got an exception on the line marked "* * *". Here's the exception detail:
System.Net.WebException was unhandled
Message=Unable to connect to the remote server
Source=System
StackTrace:
at System.Net.HttpWebRequest.GetResponse()
at ConsoleApplication2.Program.Main(String[] args) in c:\Data\Projects\ConsoleApplication2\ConsoleApplication2\Program.cs:line 17
InnerException: System.Net.Sockets.SocketException
Message=A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.0.0.1:80
Source=System
ErrorCode=10060
Maybe it's an issue with an earlier .NET version, that's now fixed in .NET 4 / VS 2010? Which version of .NET are you using?
I also found this thread from 2007, where someone else ran into the same problem. There are some good suggestions there, including the following:
Turn on system.net tracing
Work around the problem by using Dns.GetHostAddresses() to resolve it to an IP. Then put the IP in the URL - e.g. "http://10.0.0.1/". That may not be an option for you though.
In the above thread, mariyaatanasova_msft also says: "HttpWebRequest uses Dns.GetHostEntry to resolve the host, so you may get a different result from Dns.GetHostAddresses".
You should overwrite the default proxy.
HttpWebRequest & WebRequest will set a default proxy if present in Internet Explorer and your file hosts will be bypassed.
request.Proxy = new WebProxy();
The following is just an example of code:
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("www.bing.com");
request.Proxy = new WebProxy();
request.Method = "POST";
request.AllowAutoRedirect = false;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
//some code here
}
}
catch (exception e)
{
//Some other code here
}
How to detect that a WebRequest failed due to a web proxy error and not a target web server error?
try
{
var request = (HttpWebRequest)WebRequest.Create("http://www.example.com");
request.Proxy = new WebProxy("localhost");
var response = request.GetResponse();
return response.GetResponseStream();
}
catch(WebException webex)
{
//Detect proxy failure
}
It is difficult. Here are some suggestions:
The webex.Response.ResponseUri property contains the URI of your proxy server instead of the server you were trying to contact.
The webex.Response.StatusCode property is one that always refers to a proxy problem, e.g. ProxyAuthenticationRequired. Unfortunately most statuses could refer to either a proxy error or a server error.
The webex.Response.Headers collection contains non-standard entries that you recognise as being generated by your proxy server. For example, the Squid proxy returns the header "X-Squid-Error", with its own proprietary set of statuses.
The webex.Response.ResponseStream stream contains an HTML or plain text error message in a format that you recognise as being generated by your proxy server. You might test to see if it contains the URI of your proxy server.
In your catch block, make sure that you log full details of the WebException object, including all the properties mentioned above. You can then analyse the log data and develop an accurate test for proxy errors.
I think you could catch InvalidOperationException and then check the message for "proxy".
The message would say:
The proxy name could not be resolved: 'localhost'