Is it possible to read an image attachment from System.Net.HttpWebResponse?
I have a url to a java page, which generates images.
When I open the url in firefox, the download dialog appears. Content-type is application/png.
Seems to work.
When I try this in c#, and make a GET request I retrieve the content-type: text/html and no content-disposition header.
Simple Code:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
response.GetResponseStream() is empty.
A try with java was successful.
Do I have to prepare webrequest or something else?
You probably need to set a User-Agent header.
Run Fiddler and compare the requests.
Writing something in the UserAgent property of the HttpWebRequest does indeed make a difference in a lot of cases. A common practice for web services seem to be to ignore requests with an empty UserAgent.
See: Webmasters: Interpretation of empty User-agent
Simply set the UserAgent property to a non-empty string. You can for example use the name of your application, assembly information, impersonate a common UserAgent, or something else identifying.
Examples:
request.UserAgent = "my example program v1";
request.UserAgent = $"{System.Reflection.Assembly.GetExecutingAssembly().GetName().Name.ToString()} v{System.Reflection.Assembly.GetExecutingAssembly().GetName().Version.ToString()}";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36";
And just to give a full working example:
using System.IO;
using System.Net;
void DownloadFile(Uri uri, string filename)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uri);
request.Timeout = 10000;
request.Method = "GET";
request.UserAgent = "my example program v1";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream receiveStream = response.GetResponseStream())
{
using (FileStream fileStream = File.Create(filename))
{
receiveStream.CopyTo(fileStream);
}
}
}
}
Related
I want to go to download a string from a website, I made this php file to show an example.
(This won't work around my whole website)
The link http://swageh.co/information.php won't be downloaded using a webClient from any PC.
I prefer using a webClient.
No matter what I try, it won't downloadString.
It works fine on a browser.
It returns an error 500 An unhandled exception of type 'System.Net.WebException' occurred in System.dll
Additional information: The underlying connection was closed: An unexpected error occurred on a send. is the error
Did you change something on the server-side?
All of the following options are working just fine for me as of right now (all return just "false" with StatusCode of 200):
var client = new WebClient();
var stringResult = client.DownloadString("http://swageh.co/information.php");
Also HttpWebRequest:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://swageh.co/information.php");
request.GetResponse().GetResponseStream();
Newer HttpClient:
var client = new HttpClient();
var req = new HttpRequestMessage(HttpMethod.Get, "http://swageh.co/information.php");
var res = client.SendAsync(req);
var stringResult = res.Result.Content.ReadAsStringAsync().Result;
it's because your website is responding with 301 Moved Permanently
see Get where a 301 URl redirects to
This shows how to automatically follow the redirect: Using WebClient in C# is there a way to get the URL of a site after being redirected?
look at Christophe Debove's answer rather than the accepted answer.
Interestingly this doesn't work - tried making headers the same as Chrome as below, perhaps use Telerik Fiddler to see what is happening.
var strUrl = "http://theurl_inhere";
var headers = new WebHeaderCollection();
headers.Add("Accept-Language", "en-US,en;q=0.9");
headers.Add("Cache-Control", "no-cache");
headers.Add("Pragma", "no-cache");
headers.Add("Upgrade-Insecure-Requests", "1");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(strUrl);
request.Method = "GET";
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
request.Accept = "text/html,application/xhtml+xml,application/xml; q = 0.9,image / webp,image / apng,*/*;q=0.8";
request.Headers.Add( headers );
request.AllowAutoRedirect = true;
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
var strLastRedirect = response.ResponseUri.ToString();
StreamReader reader = new StreamReader(dataStream);
string strResponse = reader.ReadToEnd();
response.Close();
I want to fill my MultiLine textbox from webpage's this is my code:
WebRequest request = WebRequest.Create(urltxt.Text.Trim());
WebResponse response = request.GetResponse();
Stream data = response.GetResponseStream();
string html = String.Empty;
using (StreamReader sr = new StreamReader(data))
{
html = sr.ReadToEnd();
}
var htmlDoc = new HtmlDocument();
htmlDoc.LoadHtml(html);
var htmlBody = htmlDoc.DocumentNode.SelectSingleNode("//body");
valuetxt.Text = htmlBody.InnerText;
This code is working fine for some url but for some url (https) this gave me an error:
Could not find file 'C:\Program Files\IIS Express\www.justdial.com
or:
The remote server returned an error: (403) Forbidden
Can anyone help me? Thanks in advance, sorry for my bad English.
Are you behind a proxy? Even on open internet, depending on your network configuration, you might need to set credentials in your connection before requesting.
WebRequest request = WebRequest.Create(urltxt.Text.Trim());
request.Credentials = new NetworkCredential("user", "password");
It seems your address doesn't have http:// or https:// at the beginning; in the urltxt variable and you get error because of relative addressing.
Add a UserAgent to your request to connect https properly:
request.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36";
from here
I have a website with webservice active(prestashop)
This site require an authentication.
I use this code:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36";
request.Method = "GET";
request.Credentials = new NetworkCredential("key", "");
request.PreAuthenticate = true;
//request.Connection
request.Host = "localhost";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
String R = reader.ReadToEnd();
The code is ok but my problem is that there is a login form for the webservice
In fact, the HttpWebRequest object , sends two requests:
with the first answer is not authorized while the second was ok status.
I used fiddler web debbuger.
I apologize for my English.
if the form is submitted using GET method you must pass the form paramaters in the url query string, for instance http://url?username={0}&pass={1}. If it is POST method, you must pass the form info into the http body request. There is a lot of examples in stackoverflow of this. Also you must handle the cookies witch is achieve using the CookieContainer. In the first request intialize the container
request.CookieContainer = new CookieContainer();
when the request comeback with ok status the cookies will be in request.Cookies witch is a CookieCollection instance. Later for further request you must have to pass this cookies in order to retrieve the correct data.
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(userCookies);
Hope it helps!
Hey I'm trying to figure out using HttpWebRequest to do a Post request to a login page, say yahoo mail, and examine the returned page source.
But using my Post method I still got the login page.
Here is my method:
public static string GetResponse(string sURL, ref CookieContainer cookies, string sParameters)
{
HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(sURL);
httpRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36";
httpRequest.CookieContainer = cookies;
httpRequest.Method = "Post";
httpRequest.ContentType = "application/x-www-form-urlencoded";
httpRequest.ContentLength = sParameters.Length;
httpRequest.AllowAutoRedirect = true;
using (Stream stream = httpRequest.GetRequestStream())
{
stream.Write(Encoding.UTF8.GetBytes(sParameters), 0, sParameters.Length);
}
HttpWebResponse httpWebResponse = (HttpWebResponse)httpRequest.GetResponse();
string sResponse;
using (Stream stream = httpWebResponse.GetResponseStream())
{
StreamReader reader = new StreamReader(stream, System.Text.Encoding.GetEncoding(936));
sResponse = reader.ReadToEnd();
}
return sResponse;
}
The code to call the method is:
string sParameter = ".tries=1&.src=ym&.md5=&.hash=&.js=&.last=&promo=&.intl=us&.lang=en-US&.bypass=&.partner=&.u=eip09319532h1&.v=0&.challenge=3QjvX9eEFtJRrABhZp9kgS9IT.VO&.yplus=&.emailCode=&pkg=&stepid=&.ev=&hasMsgr=0&.chkP=Y&.done=http%3A%2F%2Fmail.yahoo.com&.pd=ym_ver%3D0%26c%3D%26ivt%3D%26sg%3D&.ws=1&.cp=0&nr=0&pad=3&aad=3&login=username%40yahoo.com&passwd=xxxxx&.persistent=&.save=&passwd_raw=";
System.Net.CookieContainer coookies = null ;
string sResponse;
sResponse = GetResponse(sUrl, ref coookies, sParameter);
The string sParameter was obtained by examining the data posted to the server in Firefox's Firebug plugin. But in the parameters I posted above, I masked my user id and password.
I wanted to re-use the session so I passed a CookieContainer object as reference to the method.
It compiles and runs, but the page returned to me is not logged-in status.
I have read several similar questions on stackoverflow, but still can't make my method work. Your help is appreciated.
I have url like:
http://www.matweb.com/search/DataSheet.aspx?MatGUID=849e2916ab1541be9ff6a17b78f95c82
I want to download source code from that page using this code:
private static string urlTemplate = #"http://www.matweb.com/search/DataSheet.aspx?MatGUID=";
static string GetSource(string guid)
{
try
{
Uri url = new Uri(urlTemplate + guid);
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.Method = "GET";
HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();
Stream responseStream = webResponse.GetResponseStream();
StreamReader responseStreamReader = new StreamReader(responseStream);
String result = responseStreamReader.ReadToEnd();
return result;
}
catch (Exception ex)
{
return null;
}
}
When I do so I get:
You do not seem to have cookies enabled. MatWeb Requires cookies to be enabled.
Ok, that I understand, so I added lines:
CookieContainer cc = new CookieContainer();
webRequest.CookieContainer = cc;
I got:
Your IP Address has been restricted due to excessive use. The problem may be compounded when an IP address may be shared by many people in a company or through an internet service provider. We apologize for any inconvenience.
I can understand this but I'm not getting this message when I try to visit this page using web browser. What can I do to get the source code? Some cookies or http headers?
It probably doesn't like your UserAgent. Try this:
webRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 (.NET CLR 3.5.30729)"; //maybe substitute your own in here
It looks like you're doing something that the company doesn't like, if you got an "excessive use" response.
You are downloading pages too fast.
When you use a browser you might get up to one page per second. Using a application you can get several pages per second and that's probably what their web server is detecting. Hence the excessive usage.