Develop Reference

'System.Net.WebException' when accessing WebClient. Works fine on browser

I want to go to download a string from a website, I made this php file to show an example.
(This won't work around my whole website)
The link http://swageh.co/information.php won't be downloaded using a webClient from any PC.
I prefer using a webClient.
No matter what I try, it won't downloadString.
It works fine on a browser.
It returns an error 500 An unhandled exception of type 'System.Net.WebException' occurred in System.dll
Additional information: The underlying connection was closed: An unexpected error occurred on a send. is the error

Did you change something on the server-side?
All of the following options are working just fine for me as of right now (all return just "false" with StatusCode of 200):
var client = new WebClient();
var stringResult = client.DownloadString("http://swageh.co/information.php");
Also HttpWebRequest:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://swageh.co/information.php");
request.GetResponse().GetResponseStream();
Newer HttpClient:
var client = new HttpClient();
var req = new HttpRequestMessage(HttpMethod.Get, "http://swageh.co/information.php");
var res = client.SendAsync(req);
var stringResult = res.Result.Content.ReadAsStringAsync().Result;

it's because your website is responding with 301 Moved Permanently
see Get where a 301 URl redirects to
This shows how to automatically follow the redirect: Using WebClient in C# is there a way to get the URL of a site after being redirected?
look at Christophe Debove's answer rather than the accepted answer.
Interestingly this doesn't work - tried making headers the same as Chrome as below, perhaps use Telerik Fiddler to see what is happening.
var strUrl = "http://theurl_inhere";
var headers = new WebHeaderCollection();
headers.Add("Accept-Language", "en-US,en;q=0.9");
headers.Add("Cache-Control", "no-cache");
headers.Add("Pragma", "no-cache");
headers.Add("Upgrade-Insecure-Requests", "1");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(strUrl);
request.Method = "GET";
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
request.Accept = "text/html,application/xhtml+xml,application/xml; q = 0.9,image / webp,image / apng,*/*;q=0.8";
request.Headers.Add( headers );
request.AllowAutoRedirect = true;
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
var strLastRedirect = response.ResponseUri.ToString();
StreamReader reader = new StreamReader(dataStream);
string strResponse = reader.ReadToEnd();
response.Close();

C# Code to download an image from a "not so easy" CDN

Im trying to download an image from a specific website.
Actually, my code is up and running in production for months, but it's not able to download imagens from this specific website
The image URL I need to download is this one: (for instance)
http://static7.kabum.com.br/produtos/fotos/64297/64297_index_g.jpg
The codes I tried so far:
Method 1 -> (failed)
string url = "http://static7.kabum.com.br/produtos/fotos/64297/64297_index_g.jpg";
var request = (HttpWebRequest)WebRequest.Create(url);
request.Timeout = (timeout == 0 ? 30 : timeout) * 1000;
request.KeepAlive = false;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36";
var response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
const int BUFFER_SIZE = 16 * 1024;
var buffer = new byte[BUFFER_SIZE];
// if the remote file was found, download it
using (Stream inputStream = response.GetResponseStream())
using (Stream outputStream = File.Create(fileName, BUFFER_SIZE))
{
int bytesRead;
do
{
bytesRead = inputStream.Read(buffer, 0, buffer.Length);
outputStream.Write(buffer, 0, bytesRead);
} while (bytesRead != 0);
}
}
Method 2 -> (also failed)
[..]
using(Image webImage = Image.FromStream(response.GetResponseStream()))
{
webImage.Save(fileName);
}
[..]
Both methods fail with the following exception
“Parameter not valid” exception loading System.Drawing.Image
StackTrace = " em System.Drawing.Image.FromStream(Stream stream,
Boolean useEmbeddedColorManagement, Boolean validateImageData) in
System.Drawing.Image.FromStream(Stream stream) in
MonitorLib.Helper.RequestPageHelper.RequestDowloadPage(Boolean proxy,
Strin...
I guess the image data is incomplete or compacted, but the URL Works fine on any browser
any thoughts?
thanks a lot friends

you could use the WebClient.DownloadFile() method.
var fileName = #"C:\path\to\file.jpg";
var url = "http://static7.kabum.com.br/produtos/fotos/64297/64297_index_g.jpg";
using (var client = new WebClient())
{
client.DownloadFile(url, fileName);
}

This seems to be problem with the server responding with bad header that the browsers are able to ignore and get past. You need to tell your application to do the same. There are several options for doing that. The server committed a protocol violation. Section=ResponseHeader Detail=CR must be followed by LF, In WinForms? should be able to guide you to the right direction.

Not able to access a page from US as a country

I want to use the US as a country to access this
url = http://www.tillys.com/product/Say-What/Short-Dresses/SAY-WHAT--Ribbed-Tank-Midi-Dress/Heather-Grey/285111595,
I've tried with cookies and all but the url still it redirects to the site's home page.
I want to see if there is any way i can access this page. Below is the code with which i am trying.
Below is the function with which i am trying to do this:
public static string getUrlContent (string url)
{
var myHttpWebRequest = (HttpWebRequest)WebRequest.Create(url);
myHttpWebRequest.Method = "GET";
myHttpWebRequest.AllowAutoRedirect = true;
myHttpWebRequest.ContentLength = 0;
myHttpWebRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
myHttpWebRequest.Headers.Add("Cookie", "=en%5FUS;");
myHttpWebRequest.UserAgent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/49.0.2623.108 Chrome/49.0.2623.108 Safari/537.36";
//myHttpWebRequest.Headers.Add("Accept-Encoding", "gzip, deflate, sdch");
myHttpWebRequest.Headers.Add("Accept-Language", "en-US,en;q=0.8");
myHttpWebRequest.Headers.Add("Cookie", "wlcme=true");
//myHttpWebRequest.CookieContainer = new CookieContainer();
//myHttpWebRequest.Headers.Add("X-Macys-ClientId", "NavApp");
var response = (HttpWebResponse)myHttpWebRequest.GetResponse();
var rmyResponseHeaders = response.Headers;
Console.WriteLine ("Content length is {0}", response.ContentLength);
Console.WriteLine ("Content type is {0}", response.ContentType);
// Get the stream associated with the response.
Stream receiveStream = response.GetResponseStream ();
// Pipes the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader (receiveStream, Encoding.UTF8);
//Console.WriteLine ("Response stream received.");
Console.WriteLine (readStream.ReadToEnd ());
var josnStr = readStream.ReadToEnd ();
Console.WriteLine (josnStr);
return josnStr;
//Encoding enc1 = Encoding.GetEncoding(1252);
}

If the site www.tillys.com is using Geo-fencing it will show you different content based on a lookup of the requesting IP address. In this case there's nothing C# or other languages can do.
You'll need to either proxy your request through a VPN (see How to send WebRequest via proxy?) or deploy your code to a data center in the US. For example, if you use Azure you can deploy to several different data centers through out the world including several data centers in the US. Once your code is running in the US it should be able to access the US version of the page.

C# HttpWebRequests 403 Forbidden

after hours of trying I decided to ask you for help.
I am trying to send a post request to a website (https://www.instagram.com/)
More exactly here: "https://instagram.com/web/likes/xxxxxxxxxxxxxxx/like/"
The problem doesn't always appear, for example I can succesfully send the request 4-5 times, and then I get this error 3-4 times, then I can send it again etc.
That's why I can't understand what's causing the problem.
My code:
byte[] bytes = ASCIIEncoding.UTF8.GetBytes("");
HttpWebRequest postReq = (HttpWebRequest)WebRequest.Create("https://instagram.com/web/likes/" + ID + "/like/");
WebHeaderCollection postHeaders = postReq.Headers;
postReq.Method = "POST";
postReq.Host = "instagram.com";
postReq.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0";
postReq.Accept = "*/*";
postHeaders.Add("Accept-Language", "it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3");
postHeaders.Add("Accept-Encoding", "gzip, deflate");
postHeaders.Add("X-Instagram-AJAX", "1");
postHeaders.Add("X-CSRFToken", CSRF);
postHeaders.Add("X-Requested-With", "XMLHttpRequest");
postReq.Referer = "https://instagram.com/";
postReq.CookieContainer = cookies;
postReq.KeepAlive = true;
postHeaders.Add("Pragma", "no-cache");
postHeaders.Add("Cache-Control", "no-cache");
postReq.ContentLength = bytes.Length;
Stream postStream = postReq.GetRequestStream();
postStream.Write(bytes, 0, bytes.Length);
postStream.Close();
HttpWebResponse postResponse;
postResponse = (HttpWebResponse)postReq.GetResponse();
StreamReader reader = new StreamReader(postResponse.GetResponseStream());
ID is a 26 characters number, always different for each photo.
I got the headers using Live HTTP Headers, and they are the same like my request.
This is the full error description:
"An unhandled exception of type 'System.Net.WebException' occurred in System.dll
Additional information: The remote server returned an error: (403) Forbidden."
Any ideas? Thanks in advance for any help :).

Issue WebRequest while reading NetworkStream of TcpListener

Following on from the post How to create a simple proxy in C#? I have been playing around with implementing a basic proxy.
Where I am getting stuck and confused is trying to issue a WebRequest with the information provided in the original request.
Using the following code.
WebRequest webRequest = WebRequest.Create("http://www.google.com");
(webRequest as HttpWebRequest).UserAgent = "MOZILLA/5.0 (WINDOWS NT 6.1; WOW64) APPLEWEBKIT/537.1 (KHTML, LIKE GECKO) CHROME/21.0.1180.75 SAFARI/537.1";
webRequest.Method = "GET";
WebResponse webResponse = webRequest.GetResponse();
Stream responseStream = webResponse.GetResponseStream();
byte[] responseBytes = responseStream.ReadFully();
I can successfully issue a request and return the page content.
However when I put it inside a Proxy request (IE: TcpListener) like such.
TcpListener _listener = new TcpListener(IPAddress.Any, 1234);
this._listener.Start();
byte[] bytes = new byte[1024];
while (true)
{
TcpClient client = this._listener.AcceptTcpClient();
NetworkStream networkStream = client.GetStream();
int i = networkStream.Read(bytes, 0, bytes.Length);
while (i != 0)
{
string data = System.Text.Encoding.ASCII.GetString(bytes, 0, i);
RequestHeader header = new RequestHeader(data.ToUpper());
WebRequest webRequest = WebRequest.Create(header.URL);
(webRequest as HttpWebRequest).UserAgent = header.UserAgent;
webRequest.Method = "GET";
WebResponse webResponse = webRequest.GetResponse(); //It gets here and never returns
Stream responseStream = webResponse.GetResponseStream();
byte[] responseBytes = responseStream.ReadFully();
networkStream.Write(responseBytes, 0, responseBytes.Length);
i = networkStream.Read(bytes, 0, bytes.Length);
}
client.Close();
}
It blocks at the line WebResponse webResponse = webRequest.GetResponse(); and never returns.
This has definitely got nothing to do with the data provided by the RequestHeader class I created as I've also tried hardcoding the values.
I'm assuming I'm missing something fundamental about the way sockets work in such a scenario and the approach required. Hopefully someone can clarify for me.

Yeah, you are assuming you have read all the header.
Instead of this, some kind of state machine should be implemented to parse the incoming HTTP-request. The state machine must collect the information about the request and, of course, detect the end of request, then you process the request (proxy stuff) and send the response. Just Google C# http state machine for examples.

So it turned out to be a proxy issue.
Basically for testing I needed to set the machine proxy to 127.0.0.1:1234 or similar.
This in turn was being used in the default settings when initializing a WebRequest.
So all I needed to do in the end was the following to bypass the proxy.
(webRequest as HttpWebRequest).UserAgent = header.UserAgent;
webRequest.Method = "GET";
webRequest.Proxy = null; //Adding this line cleared the proxy.