Develop Reference

'System.Net.WebException' when accessing WebClient. Works fine on browser

I want to go to download a string from a website, I made this php file to show an example.
(This won't work around my whole website)
The link http://swageh.co/information.php won't be downloaded using a webClient from any PC.
I prefer using a webClient.
No matter what I try, it won't downloadString.
It works fine on a browser.
It returns an error 500 An unhandled exception of type 'System.Net.WebException' occurred in System.dll
Additional information: The underlying connection was closed: An unexpected error occurred on a send. is the error

Did you change something on the server-side?
All of the following options are working just fine for me as of right now (all return just "false" with StatusCode of 200):
var client = new WebClient();
var stringResult = client.DownloadString("http://swageh.co/information.php");
Also HttpWebRequest:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://swageh.co/information.php");
request.GetResponse().GetResponseStream();
Newer HttpClient:
var client = new HttpClient();
var req = new HttpRequestMessage(HttpMethod.Get, "http://swageh.co/information.php");
var res = client.SendAsync(req);
var stringResult = res.Result.Content.ReadAsStringAsync().Result;

it's because your website is responding with 301 Moved Permanently
see Get where a 301 URl redirects to
This shows how to automatically follow the redirect: Using WebClient in C# is there a way to get the URL of a site after being redirected?
look at Christophe Debove's answer rather than the accepted answer.
Interestingly this doesn't work - tried making headers the same as Chrome as below, perhaps use Telerik Fiddler to see what is happening.
var strUrl = "http://theurl_inhere";
var headers = new WebHeaderCollection();
headers.Add("Accept-Language", "en-US,en;q=0.9");
headers.Add("Cache-Control", "no-cache");
headers.Add("Pragma", "no-cache");
headers.Add("Upgrade-Insecure-Requests", "1");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(strUrl);
request.Method = "GET";
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
request.Accept = "text/html,application/xhtml+xml,application/xml; q = 0.9,image / webp,image / apng,*/*;q=0.8";
request.Headers.Add( headers );
request.AllowAutoRedirect = true;
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
var strLastRedirect = response.ResponseUri.ToString();
StreamReader reader = new StreamReader(dataStream);
string strResponse = reader.ReadToEnd();
response.Close();

Httpwebrequest error 403 when using proxy to access a https site c#

Here is my code:
HttpWebRequest firstRequest = (HttpWebRequest)HttpWebRequest.Create("http://opizo.com/OEdwHe");
firstRequest.CookieContainer = new CookieContainer();
firstRequest.Method = "GET";
firstRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0";
WebProxy tempProxy = new WebProxy(Proxy+':'+ ProxyPort /*"127.0.0.1:8888"*/);
tempProxy.BypassProxyOnLocal = false;
tempProxy.UseDefaultCredentials = true;
firstRequest.Accept = "text/html";
firstRequest.UseDefaultCredentials = true;
firstRequest.Credentials = System.Net.CredentialCache.DefaultCredentials;
firstRequest.Proxy = tempProxy;
firstRequest.Timeout = 20000;
WebResponse firstResponse = firstRequest.GetResponse();
System.IO.Stream firstStream = firstResponse.GetResponseStream();
StreamReader firstStreamReader = new StreamReader(firstStream);
string data = firstStreamReader.ReadToEnd();
The link i want to connect is https , but i enter it in http format.
Proxy + ProxyPort in proxy creation, is of a list of proxies.
The error occurs at WebResponse firstResponse = firstRequest.GetResponse();
line.
The exception is "The remote server returned an error: (403) Forbidden."
Sometimes when i use ssl proxy, "The operation has timed out" exception is thrown and Rarely , no exception is thrown but its probability is one in thousand.
UPDATE: I deleted these two lines
firstRequest.UseDefaultCredentials = true;
firstRequest.Credentials = System.Net.CredentialCache.DefaultCredentials;
and the error 403's probability is changed from 99% to 1%.
now the exception "The operation has timed out" is thrown.
UPDATE 2: The 403 error is back.
I think there is no way to connect to a https site with http proxy in C# and the best way to solve the issue is to use a SOCKS library.

403 error from HttpWebResponse Instagram

First of all sorry for my bad english.
Task is to authenticate in instagram with API. When I try to get a token after the user clicked "OK" I get 403 error from HttpWebResponse and I can't get my token:(. I don't know what to do. Help me please. And when I enter this link in me browser - it's ok
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("https://instagram.com" + url);
request.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36";
request.Method = "POST";
request.CookieContainer = cookies;
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
request.Headers.Add("Accept-Encoding: gzip, deflate");
request.Headers.Add("Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4");
request.Headers.Add("Cache-Control:max-age=0");
request.ContentType = "application/x-www-form-urlencoded";
request.Host = "instagram.com";
request.Headers.Add("Origin: https://instagram.com");
StreamWriter writer = new StreamWriter(request.GetRequestStream());
writer.Write("csrfmiddlewaretoken=" + par + "&username=LOGIN&password=PASSWORD");
writer.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse(); //403 ERORR

You cannot get an access token by simply sending a username/password pair to the API! You have to direct a real user to the Instagram's authorization url, and after the user logged in successfully Instagram will send your application a code to be used for getting the access token.
More information: https://instagram.com/developer/authentication/

The underlying connection was closed exception while WebClient DownloadString

Just a piece of code
WebClient wc = new WebClient();
String str = wc.DownloadString(new Uri("http://content.warframe.com/dynamic/rss.php"));
And I got exception:
An unhandled exception of type 'System.Net.WebException' occurred in
System.dll
Additional information: The underlying connection was closed: The
connection was closed unexpectedly.
I've head that this is a bug in .NET (I am using 3.5), but I tried other methods to obtain the content of this link (its rss, xml). No lucky shot yet
var webrequest = (WebRequest)HttpWebRequest.Create(#"http://content.warframe.com/dynamic/rss.php");
var resp = webrequest.GetResponse();
//HttpWebResponse webresponse = (HttpWebResponse)webrequest.GetResponse(); // Wont work also
This code above won't work either, both casts the same exception
Fiddler logs:
SESSION STATE: Aborted.
Response Entity Size: 512 bytes.
== FLAGS ==================
BitFlags: [ResponseGeneratedByFiddler] 0x100
X-ABORTED-WHEN: Done
X-CLIENTIP: 127.0.0.1
X-CLIENTPORT: 2747
X-EGRESSPORT: 2748
X-FAILSESSION-WHEN: ReadingResponse
X-HOSTIP: 205.185.216.10
X-PROCESSINFO: willitwork.vshost:3300
== TIMING INFO ============
ClientConnected: 10:29:11.706
ClientBeginRequest: 10:29:11.713
GotRequestHeaders: 10:29:11.713
ClientDoneRequest: 10:29:11.713
Determine Gateway: 0ms
DNS Lookup: 164ms
TCP/IP Connect: 74ms
HTTPS Handshake: 0ms
ServerConnected: 10:29:11.953
FiddlerBeginRequest: 10:29:11.953
ServerGotRequest: 10:29:11.953
ServerBeginResponse: 10:29:12.372
GotResponseHeaders: 00:00:00.000
ServerDoneResponse: 10:29:12.372
ClientBeginResponse: 10:29:12.385
ClientDoneResponse: 10:29:12.385
Overall Elapsed: 0:00:00.672
The response was buffered before delivery to the client.
== WININET CACHE INFO ============
This URL is not present in the WinINET cache. [Code: 2]
* Note: Data above shows WinINET's current cache state, not the state at the time of the request.
* Note: Data above shows WinINET's Medium Integrity (non-Protected Mode) cache only.
Also - 504, this does not makes sense since I can get data from link via chrome / firefox / ie...
I just did it to work in other language, but I am forced to do it with C# (I' ve made 2 much code to rewrite it)
I've added some settings like fiddler said
myHttpWebRequest1.ProtocolVersion = HttpVersion.Version11;
myHttpWebRequest1.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36";
myHttpWebRequest1.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
At least now I get 504 error instead of "unknown", but I can still view the content via webbrowser, so the 504 error is fake
Edit: There is no response error when I added
myHttpWebRequest1.Headers["Accept-Encoding"] = "gzip";
but now the output is messed and unreadable

I have same error.
You can add User Agent to your httpRequest
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";

Ok, i got this all fixes & working!
static void Main(string[] args)
{
Uri url = new Uri(#"http://content.warframe.com/dynamic/rss.php");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
// MAGIC LINE GOES HERE \/
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
// Assign the response object of HttpWebRequest to a HttpWebResponse variable.
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream streamResponse = response.GetResponseStream())
{
using (StreamReader streamRead = new StreamReader(streamResponse))
{
Char[] readBuff = new Char[2000];
int count = streamRead.Read(readBuff, 0, 2000);
while (count > 0)
{
String outputData = new String(readBuff, 0, count);
Console.Write(outputData);
count = streamRead.Read(readBuff, 0, 2000);
}
}
}
}
Console.ReadKey();
}
Besides of not-using WebClient.DownloadString method i had to add decompresion line
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
Thanks for tips (especially fiddler one, Decode button saved my time to find what's wrong)

Check this answer:
..The underlying connection was closed: An unexpected error occurred on a receive
So this may work for you:
var webRequest = (HttpWebRequest)HttpWebRequest.Create(#"http://content.warframe.com/dynamic/rss.php");
webRequest.KeepAlive = false;
var resp = webRequest.GetResponse();
EDIT:
You are right, check rather this:
http://msdn.microsoft.com/cs-cz/library/system.net.httpwebrequest.keepalive%28v=vs.110%29.aspx
Here is working code that will print out the recieved response content:
static void Main(string[] args)
{
// Create a new HttpWebRequest object.Make sure that
// a default proxy is set if you are behind a firewall.
HttpWebRequest myHttpWebRequest1 =
(HttpWebRequest)WebRequest.Create(#"http://content.warframe.com/dynamic/rss.php");
myHttpWebRequest1.KeepAlive=false;
// Assign the response object of HttpWebRequest to a HttpWebResponse variable.
HttpWebResponse myHttpWebResponse1 =
(HttpWebResponse)myHttpWebRequest1.GetResponse();
Console.WriteLine("\nThe HTTP request Headers for the first request are: \n{0}", myHttpWebRequest1.Headers);
Stream streamResponse = myHttpWebResponse1.GetResponseStream();
StreamReader streamRead = new StreamReader(streamResponse);
Char[] readBuff = new Char[256];
int count = streamRead.Read(readBuff, 0, 256);
Console.WriteLine("The contents of the Html page are.......\n");
while (count > 0)
{
String outputData = new String(readBuff, 0, count);
Console.Write(outputData);
count = streamRead.Read(readBuff, 0, 256);
}
Console.WriteLine();
// Close the Stream object.
streamResponse.Close();
streamRead.Close();
Console.ReadKey();
}

C# HttpWebRequest returning 500 Internal Error

I would like to know why my HTTP Request returned an 500 Server Internal Error on the Response.
I use this C# Code
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://svabyss.66ghz.com/getmsg.php");
req.ContentLength = ("receiver=" + b.ToString() + "&PHPSESSID=" + _SESSIONID).Length;
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
req.KeepAlive = true;
req.CookieContainer = new CookieContainer();
req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13";
req.Headers[HttpRequestHeader.Pragma] = "cache";
req.UseDefaultCredentials = true;
req.Credentials = CredentialCache.DefaultNetworkCredentials;
req.PreAuthenticate = true;
req.Proxy = new WebProxy("http://svabyss.66ghz.com:80", true);
StreamWriter writer = new StreamWriter(req.GetRequestStream());
writer.Write("receiver=" + b.ToString() + "&PHPSESSID=" + _SESSIONID);
writer.Flush();
writer.Dispose();
try
{
HttpWebResponse response = (HttpWebResponse)req.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
OperationCompleted(new SVWorkerEventArgs("GET|" + b.ToString(), reader.ReadToEnd()));
response.Close();
reader.Dispose();
}
catch (WebException ex)
{
StreamReader str = new StreamReader(ex.Response.GetResponseStream());
string err = str.ReadToEnd();
}
I dont know wich header i missed. ANy help would be great !
thanks..
PROBLEM FIXED
I received the reply via error message returned by the server. The error message is actually the result i want.
Now, i dont care what the server replying to me. Wether its 404 or 505, the error message returned is what i want..
Thanks guys :)

The 500 error code is returned because the server (not your client) experienced an internal error. To find out what the error is, you would need access to the server's logs, or need to ask someone who has access.
It is entirely possible that the error is caused by something in your code, but without knowing what the error is on the other end, there's no way for you to be sure. It could just as easily be something in their code, or a connection to their databse, etc.

I suggest you make the request with a browser while running Fiddler:
http://www.fiddler2.com/fiddler2/
Look at the header used there.