I have a list of URLs linking directly to PDFs on a database website. It would be very easy to automate the download process, except for the fact that I have to access the website through a proxy server. The code I've been trying to use has been this:
public void Download()
{
WebClient wb2 = new WebClient();
WebProxy proxy = new WebProxy("PROXY_URL:port", true);
proxy.Credentials = new NetworkCredential("USERNAME", "PASSWORD");
GlobalProxySelection.Select = proxy;
try
{
for(int i = 0; i < URLList.Length; i++)
{
byte[] Data = DownloadData(URLList[i]);
FileStream fs = new FileStream(#"D:\Files\" + i.toString() + ".pdf", FileMode.Create)
fs.Write(Data, 0, Data.Length);
fs.Close();
}
}
catch(WebException WebEx)
{
MessageBox.Show(WebEx.Message);
}
}
public byte[] DownloadData(string path)
{
WebClient wb2 = new WebClient();
wb2.Credentials = new NetworkCredential("USERNAME","PASSWORD");
return wb2.DownloadData(path);
}
For some reason, it returns the error "(400): Bad Request" every time. I'm obviously able to get to these PDFs just fine through Firefox, so I'm wondering what I'm doing wrong here. I'm fairly new to programming in general, and very new to web protocols through C#. Any help would be appreciated.
use fiddler to work out the difference between the request your code is sending vs the one via your browser.
the 400 error is due to a malformed request; opposed to the proxy denying you (407) or the site requiring authentication (401).
Incidently, the line "wb2.Credentials = ..." is providing your username/password to the target server. is this intended?
Haven't used WebClient for a while, but you can use var request = HttpWebRequest.Create();
request.Proxy = proxy; request.GetResponse().GetResponseStream() and read the bytes using BinaryReader().
That will give you the byte array that you can write to a file using File.WriteAllBytes() rather than having to use a FileStream.
hth
Related
I am working on a website which is old and uses AMF packages to send data to server. I need to send some text data to this website and save that data to this website. In other words I want to automate this website.
I found fluorinefx library but it is no longer supported by owner and there is no documentation on the internet about that. I tried to use fluorinefx's serialize class to serialize my dictionary data and send that to server with content-type header application/x-amf. But httpclient doesnt support AMF bytearrays.
When I tried to use fluorinefx's NetConnection class and use netConnection.Connect(url) method, if url starts with http:// there is no problem, but if url starts with https// it gives an URİFormat exception. Since the website I am working on uses https:// I cannot use this Connect method either.
So please help me, how can I prepare a correctly structured AMF object and send this with HttpClient. Or is there any other libraries which I can use for sending AMF packages.(I looked WebOrb and DotAmf too but theese are not working either)
Thanks.
public static object SendAMF(string method, object[] arguments)
{
AMFMessage message = new AMFMessage(3);
message.AddHeader(new AMFHeader("headerKey", false, "headerValue"));
message.AddBody(new AMFBody(method, "/1", arguments));
MemoryStream ms = new MemoryStream();
AMFSerializer serializer = new AMFSerializer(ms);
serializer.WriteMessage(message);
serializer.Flush();
serializer.Dispose();
var request = (HttpWebRequest)WebRequest.Create($"{Endpoint}/Gateway.aspx?method={method}");
byte[] data = Encoding.Default.GetBytes(Encoding.Default.GetString(ms.ToArray()));
request.GetRequestStream().Write(data, 0, data.Length);
try
{
var response = (HttpWebResponse)request.GetResponse();
ms = new MemoryStream();
response.GetResponseStream().CopyTo(ms);
dynamic obj = DecodeAMF(ms.ToArray());
ms.Dispose();
response.Dispose();
return obj;
}
catch(Exception ex)
{
ms.Dispose();
return "ERROR! " + ex.ToString();
}
}
public static dynamic DecodeAMF(byte[] body)
{
MemoryStream memoryStream = new MemoryStream(body);
AMFDeserializer amfdeserializer = new AMFDeserializer(memoryStream);
AMFMessage amfmessage = amfdeserializer.ReadAMFMessage();
dynamic content = amfmessage.Bodies[0].Content;
memoryStream.Dispose();
amfdeserializer.Dispose();
return content;
}
`
I've tried pretty much every code sample and technique recommended on the first few pages of Google. All of these samples fail though with a timeout or similar error.
static void Main(string[] args)
{
using (WebClient client = new WebClient { Credentials = new NetworkCredential("username", "password", "domain") })
{
var tmpFile = Path.GetTempFileName();
String url = "https://sharepoint.organisation.net/GetFileByServerRelativeUrl/folder/file.docx')/$value')/Files";
client.DownloadFile(new Uri(url), tmpFile);
//OR
// client.DownloadFileAsync(new Uri(url), tmpFile);
}
}
If anyone could give me a suggestion on why my code, regardless of method used to download, keeps timing out on the actual request, that would be great.
This method is to test file analysis on sharepoint.
item = 535;
String URI = "http://localhost:3033/WebFormDesigner.aspx?fm_id=" + item.ToString();
//String URI = "http://www.google.com";
WebClient wc = new WebClient();
Stream s = wc.OpenRead(URI);
XPathDocument doc = new XPathDocument(s);
It doesnt work with:
XPathDocument doc = new XPathDocument(URI);
in the above method, using the google URI works, but it doesnt seem to like the URI for localhost at all. I confirmed that the URI works when put into the browser. Not quite sure whats going on with it.
The error occurs on the Stream declaration.
The remote server returned an error: (401) Unauthorized.
This is an extension of my other post: How do you get the markup of a webpage in asp.net similar to php's get_file_contents
Edit: The issue being looked at here, is that there arent permissions to see the page. The page being visited is a secure page. What I am trying to do is to give my Request the permissions because the pages are within the same project, but using a WebRequest which assumes it is an outside source.
I was trying to accomplish with a couple different attempted. Both involved the cookies.
System.Net.HttpWebRequest request = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(URI);
request.KeepAlive = true;
request.Credentials = CredentialCache.DefaultCredentials;
request.CookieContainer = new CookieContainer();
HttpCookieCollection cookieJar = Request.Cookies;
//foreach (string cookieString in Request.Cookies)
for(int i = 0; i < cookieJar.Count; i++)
{
System.Web.HttpCookie cookie = cookieJar.Get(i);
Cookie oC = new Cookie();
oC.Domain = Request.Url.Host;
oC.Expires = cookie.Expires;
oC.Name = cookie.Name;
oC.Path = cookie.Path;
oC.Secure = cookie.Secure;
oC.Value = cookie.Value;
request.CookieContainer.Add(oC);
}
System.Net.HttpWebResponse response = (System.Net.HttpWebResponse)request.GetResponse();
Stream s = response.GetResponseStream();
This does some pretty simple stuff, but i guess i am still doing it wrong. I get the cookies for my current page, and pass them into the cookiecollection for the target page. After that, i submit the request for a response. It gives the error above as well.
I'm trying to download a file from a url. I try two approachs but it isn't working with external files.
I think it's happening because I have internet over proxy. I can download internal network files (images, mp3, mp4, whatever...) but when I try to download something in external network it gives me timeout or 404 Not Found.
1st approach: System.Net.WebResponse, System.IO.FileStream
try
{
var credentials = new NetworkCredential("myNetworkUserName", "myNetworkPassword", "myNetworkDomain");
var proxy = WebProxy.GetDefaultProxy(); //new WebProxy("myNetworkProxy") <-- I TRY BOOTH WAYS
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://farm1.static.flickr.com/83/263570357_e1b9792c7a.jpg"); //External image link
proxy.Credentials = credentials;
request.Proxy = proxy;
responseExternalImage = request.GetResponse();//explode here ex=""Unable to connect to the remote server""
string fileName = GetFileName(response.ResponseUri.OriginalString);
Stream stream = response.GetResponseStream();
using (BinaryReader br = new BinaryReader(stream))
{
content = br.ReadBytes(50000000);//perto de 50Mb
br.Close();
}
response.Close();
FileStream fs = new FileStream(pathToSaveFile + "\\" + fileName, FileMode.Create);
BinaryWriter bw = new BinaryWriter(fs);
try
{
bw.Write(content);
}
finally
{
fs.Close();
bw.Close();
}
}
catch (Exception ex)
{
}
The exception caught at GetResponse() says: "Unable to connect to the remote server", ""A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 77.238.160.184:80""
2nd approach: System.Net.WebClient, DownloadFile
try
{
var credentials = new NetworkCredential("xpta264", ConfigurationManager.AppSettings["Xpta264Password"], "ptsi");
var proxy = WebProxy.GetDefaultProxy();
proxy.Credentials = credentials;
// Create a new WebClient instance.
using (WebClient myWebClient = new WebClient())
{
myWebClient.Proxy = proxy;
// Download the Web resource and save it into the current filesystem folder.
myWebClient.DownloadFile("http://farm1.static.flickr.com/83/263570357_e1b9792c7a.jpg", pathToSaveFile + "\\testImage.jpg");
}
}
catch (Exception e)
{
}
The exception was caught at DownloadFile method and gives me the same error.
Hope someone can help me.
Sorry for my English
WebProxy.GetDefaultProxy() is obsolete and doesn't handle all cases. From the docs:
Note: This API is now obsolete.
The GetDefaultProxy method does not pick up any dynamic settings that
are generated from scripts run by Internet Explorer, from automatic
configuration entries, or from DHCP or DNS lookups.
Applications should use the WebRequest.DefaultWebProxy property and
the WebRequest.GetSystemWebProxy method instead of the GetDefaultProxy
method.
A better approach is to try to request a url and see if a proxy was used. I use this code in my own production system and it seems to work pretty well. Once you find the proxy, you can cache the address somewhere:
WebProxy proxy = null;
Uri testUri = new Uri("http://www.example.com/"); // replace with REAL url
Uri proxyEndpoint = WebRequest.GetSystemWebProxy().GetProxy(testUri);
if (!testUri.Equals(proxyEndpoint))
proxy = new WebProxy(proxyEndPoint.ToString());
You can also just try calling WebRequest.GetSystemWebProxy() and see if that gives you the proxy, but I remember having problems with that which is why I went the request route above. YMMV.
I need to write a simple C# app that should receive entire contents of a web page currently opened in Firefox. Is there any way to do it directly from C#? If not, is it possible to develop some kind of plug-in that would transfer page contents? As I am a total newbie in Firefox plug-ins programming, I'd really appreciate any info on getting me started quickly. Maybe there are some sources I can use as a reference? Doc links? Recommendations?
UPD: I actually need to communicate with a Firefox instance, not get contents of a web page from a given URL
It would help if you elaborate What you are trying to achieve. May be plugins already out there such as firebug can help.
Anways, if you really want to develop both plugin and C# application:
Check out this tutorial on firefox extension:
http://robertnyman.com/2009/01/24/how-to-develop-a-firefox-extension/
Otherwise, You can use WebRequest or HttpWebRequest class in .NET request to get the HTML source of any URL.
I think you'd almost certainly need to write a Firefox plugin for that. However there are certainly ways to request a webpage, and receive its HTML response within C#. It depends on what your requirements are?
If you're requirements are simply receive the source from any website, leave a comment and I'll point you towards the code.
Uri uri = new Uri(url);
System.Net.HttpWebRequest req = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(uri.AbsoluteUri);
req.AllowAutoRedirect = true;
req.MaximumAutomaticRedirections = 3;
//req.UserAgent = _UserAgent; //"Mozilla/6.0 (MSIE 6.0; Windows NT 5.1; Searcharoo.NET)";
req.KeepAlive = true;
req.Timeout = _RequestTimeout * 1000; //prefRequestTimeout
// SIMONJONES http://codeproject.com/aspnet/spideroo.asp?msg=1421158#xx1421158xx
req.CookieContainer = new System.Net.CookieContainer();
req.CookieContainer.Add(_CookieContainer.GetCookies(uri));
System.Net.HttpWebResponse webresponse = null;
try
{
webresponse = (System.Net.HttpWebResponse)req.GetResponse();
}
catch (Exception ex)
{
webresponse = null;
Console.Write("request for url failed: {0} {1}", url, ex.Message);
}
if (webresponse != null)
{
webresponse.Cookies = req.CookieContainer.GetCookies(req.RequestUri);
// handle cookies (need to do this incase we have any session cookies)
foreach (System.Net.Cookie retCookie in webresponse.Cookies)
{
bool cookieFound = false;
foreach (System.Net.Cookie oldCookie in _CookieContainer.GetCookies(uri))
{
if (retCookie.Name.Equals(oldCookie.Name))
{
oldCookie.Value = retCookie.Value;
cookieFound = true;
}
}
if (!cookieFound)
{
_CookieContainer.Add(retCookie);
}
}
string enc = "utf-8"; // default
if (webresponse.ContentEncoding != String.Empty)
{
// Use the HttpHeader Content-Type in preference to the one set in META
doc.Encoding = webresponse.ContentEncoding;
}
else if (doc.Encoding == String.Empty)
{
doc.Encoding = enc; // default
}
//http://www.c-sharpcorner.com/Code/2003/Dec/ReadingWebPageSources.asp
System.IO.StreamReader stream = new System.IO.StreamReader
(webresponse.GetResponseStream(), System.Text.Encoding.GetEncoding(doc.Encoding));
webresponse.Close();
This does what you want.
using System.Net;
var cli = new WebClient();
string data = cli.DownloadString("http://www.heise.de");
Console.WriteLine(data);
Native messaging enables an extension to exchange messages with a native application installed on the user's computer.