I'd like my application to query a csv file from a secure website. I have no experience with web programming so I'd appreciate detailed instructions. Currently I have the user login to the site, manually query the csv, and have my application load the file locally. I'd like to automate this by having the user enter his login information, authenticating him on the website, and querying the data. The application is written in C# .NET.
I've tested the following code already and am able to access the file once the user has already authenticated himself and created a manual query.
System.Net.WebClient Client = new WebClient();
Stream strm = Client.OpenRead("https://<URL>/file.csv");
Here is the request sent to the site for authentication. I've angle bracketed the real userid and password.
POST /pwdVal.asp HTTP/1.1
Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-shockwave-flash, */*
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; Tablet PC 2.0; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; .NET4.0C; .NET4.0E)
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Cookie: ASPSESSIONID<unsure if this data contained password info so removed>; ClientId=<username>
Host: www3.emidas.com
Content-Length: 36
Connection: Keep-Alive
Cache-Control: no-cache
Accept-Language: en-US
client_id=<username>&password=<password>
Most likely the server sends a cookie once login is performed. You need to submit the same values as the login form. (this can be done using UploadValues()) However, you need to save the resulting cookies in a CookieContainer.
When I did this, I did it using HttpWebRequest, however per http://couldbedone.blogspot.com/2007/08/webclient-handling-cookies.html you can subclass WebClient and override the GetWebRequest() method to make it support cookies.
Oh, also, I found it useful to use Fiddler while manually accessing the web site to see what actually gets sent back and forth to the web site, so I knew what I was trying to reproduce.
edit, elaboration requested: I can only elaborate how to do it using HttpWebRequest, I have not done it using WebClient. Below is the code snippet I used for login.
private CookieContainer _jar = new CookieContainer();
private string _password;
private string _userid;
private string _url;
private string _userAgent;
...
string responseData;
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(_url);
webRequest.CookieContainer = _jar;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.UserAgent = _userAgent;
string requestBody = String.Format(
"client_id={0}&password={1}", _userid, _password);
try
{
using (StreamWriter requestWriter = new StreamWriter(webRequest.GetRequestStream()))
{
requestWriter.Write(requestBody);
requestWriter.Close();
using (HttpWebResponse res = (HttpWebResponse)webRequest.GetResponse())
{
using (StreamReader responseReader = new StreamReader(res.GetResponseStream()))
{
responseData = responseReader.ReadToEnd();
responseReader.Close();
if (res.StatusCode != HttpStatusCode.OK)
throw new WebException("Logon failed", null, WebExceptionStatus.Success, res);
}
}
}
Before you go down this rabbit hole, contact the web site and ask them if they provide a web service to query user account info from. The simulated login method you are proposing should be a last resort only.
Another way you can do it is to automate IE, e.g. use a WebBrowser control. That will more accurately simulate all the clever stuff that IE does like running Javascript, which might be necessary. Although if Javascript or other clever stuff isn't necessary then using IE is a little heavy-handed and possibly prone to other problems.
Related
This question already has answers here:
.NET: Simplest way to send POST with data and read response
(8 answers)
Closed 6 years ago.
Say I have a web page that prompts for input then searches a database for the user's username and password, then decrypts the password and checks if the entered password is the same as the user's password in the database. What if I want to switch the login to a C# winforms app? How could I make that http request with the username and entered password, then having the site accept the username/password strings and search the db for those the same way I said earlier, then send a bool of true: the user entered correct info or false: the user entered incorrect info. How could I do that?
To do so you first need to know how the server accept the request which is most likely by the post method since you have to input the user name and password and then you will have to get the input the string which can be obtained using a
some browser extensions like Live Http Headers.
it should look something like this
http://yourwebsite/extension
POST /extension HTTP/1.1
Host: yourwebsite
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Length: 85
Content-Type: application/x-www-form-urlencoded
Connection: keep-alive
HereShouldBeThePoststring
Then you would create a httpwebrequest using the website url
using System.Net;
string postUrl = "YourWebsiteUrlWithYourExtension";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(postUrl);
then you would post your data
string postString = "ThePostStringYouObtainedUsingLiveHttpHeaders";
request.Method = "POST";
byte[] Content = Encoding.ASCII.GetBytes(postString);
request.ContentLength = Content.Length;
using (Stream stream = request.GetRequestStream())
{
stream.W
then you would get the response string
string responsestring = null;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
responsestring = reader.ReadToEnd();
}
then you would parse your string to get the data your need.
A good library for parsing is Html Agility Pack.
My code is :
string result = new WebClient().DownloadString("https://www.facebook.com/profile.php?id=123456789");
the result giving me supported browser.
body class=\"unsupportedBrowser
What i intend to do:
Download a source code from facebook for the particular page.
Problem encounter :
I get the stream from facebook, facebook block me since i access from the apps due to this is not a valid browser.
Expectation : How i can submit the browser type like chrome to cheat the facebook as this is a valid browser.
You can add headers to the webclient object. However, I prefer to go the HttpWebRequest/HttpWebResponse method of scraping since I believe it gives more options.
Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
Headers.Add("Accept-Encoding", "gzip, deflate");
Headers.Add("Accept-Language", "en-US,en;q=0.5");
Headers.Add("Cookie", "has_js=1");
Headers.Add("DNT", "1");
Headers.Add("Host", host);
Headers.Add("Referer", url);
Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0");
I use following code to post querystring
string URI = "http://somewebsite.com/default.aspx";
string myParameters = "param1=value1¶m2=value2¶m3=value3";
using (WebClient wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
string HtmlResult = wc.UploadString(URI, myParameters);
}
But somehow default.aspx does not accept that post call.
The point is when I manually in browser go to http://somewebsite.com/default.aspx all code there is working fine.
My questions is following what do I am missing here to archive the same result when I open page manually as I do it with WebClient?
Thank you in advance!
P.S. 1
I just tried to use GET method to that URL and it has no effect also. How is it possible?
What is difference between manual navigation to page and sending GET/POST?
P.S. 2
I even tried this
wc.Headers["Accept"] = "application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*";
wc.Headers["User-Agent"] = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDC)";
and and Load event of Default.aspx is not hiting. :(
From your description of what you want to achieve, I think you may have chosen the wrong WebClient method. Instead of UploadString, try DownloadString:
using (WebClient wc = new WebClient())
{
string HtmlResult = wc.DownloadString("http://somewebsite.com/default.aspx?param1=value1¶m2=value2¶m3=value3");
}
So that comment is correct one
"What is difference between manual navigation to page and sending
GET/POST?" - see for yourself, for example using Fiddler. –
CodeCaster
I checked all requests with Fiddler and found that there is code of base page class that redirects to Index page. So Load event is never happened.
I have a local website (not mine) that requires authentication before doing some queries. The authentication header looks like this:
Host: 192.168.7.9
Connection: keep-alive
Content-Length: 185
Origin: http://192.168.7.9
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/27.0.1453.3 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Accept: */*
DNT: 1
Referer: http://192.168.7.9/signin
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: _FProEnterprise_session=BAh7CzoVbmVlZF93ZWxjb21lX21zZ1Q6D3Nlc3Npb25faWQiJTUxNjI5OGRiMDNmNjU4ZDg4ODE3NmFiZjhmMDU3YTI2OglzaXRlSSIKc2l0ZTAGOgZFRjoObGFuZ19wYXRoSSIHZW4GOwhUOg5vbmVfY2xpY2tGOgx1c2VyX2lkaRE%3D--0c6634a714baa7f0e4795aee89b31f9b7ec0565e
And the request body looks like this:
username=myusername&password=mypassword
I'm not super great with how authentication works. So first, is this forms authentication? I'm guessing it is, since I have to enter my username and password on the site then submit to get in.
Second, why is there a Cookie already there? Is it from a previous session perhaps, and I can ignore it?
My goal is to reproduce this in C#, so that I can authenticate, get the cookie and then post data and retrieve results from this site. At least thats what I think I need to do. Links and code would be super helpful. If it's helpful I need to make this request from my web.api app controller.
You use asp.net membership provider and do the authentication like Membership.ValidateUser() and that will authenticate the formsauthentication also. Check if it is authenticated if (Context.User.Identity.IsAuthenticated) - FormsAuthentication.SignOut();
You need sql server or some kind of authentication mechanism first to save the username and password.
This seems to be an AJAX request (X-Requested-With: XMLHttpRequest). Therefore the user has to be on the web page first, which is when the session started. That is when the user gets the session cookie, which is sent every time to keep track of the session. This session is also kept on the server, where login information is stored - whether or not you're logged in, and who you are.
The contents seem to be a simple HTTP form, but since it came from an XMLHttpRequest it could just as well be created using Javascript. This is at least the standard way to send POST data through HTTP.
That is using plain HTTP authentication and the cookies are from an old session.
http://en.wikipedia.org/wiki/Basic_access_authentication
This link solved it for me:
HERE
My final code (in my web.api controller looked like this):
public static string JsonWithAuth( string url, string data )
{
var bytes = Encoding.Default.GetBytes( data );
using ( var client = new WebClientEx() )
{
var values = new NameValueCollection
{
{ "username", "myUsername" },
{ "password", "myPassword" },
};
// Authenticate
client.UploadValues( "http://192.168.7.9/main/signin", values );
// Post data
var response = client.UploadData( url, "POST", bytes );
return Encoding.Default.GetString( response );
}
}
And this was the class that made it work (from the linked answer):
/// <summary>
/// A custom WebClient featuring a cookie container
/// </summary>
public class WebClientEx : WebClient
{
public CookieContainer CookieContainer { get; private set; }
public WebClientEx()
{
CookieContainer = new CookieContainer();
}
protected override WebRequest GetWebRequest( Uri address )
{
var request = base.GetWebRequest( address );
if ( request is HttpWebRequest )
{
( request as HttpWebRequest ).CookieContainer = CookieContainer;
}
return request;
}
}
So my final call was like this:
string sampleInfo = JsonWithAuth(
"http://192.168.7.9/samples/sample_locations_list",
"sort=position&dir=ASC&box_id=");
Hope that helps someone else!
Trying To Loading Html Content Of "http://links.casemakerlegal.com/states/CA/books/Case_Law/results?search[Cite]=214 Cal.App.3d 533" but HttpWebResponse object Giving This Error "(500) Internal Server Error"
And Code Is------
request = WebRequest.Create(urlCheck); request.Timeout = 100000; response = request.GetResponse(); strmRead = new StreamReader(response.GetResponseStream(),System.Text.Encoding.UTF8); result = strmRead.ReadToEnd();
You need to use a tool like Wireshark or Ethereal, or the developer tools in your browser to investigate this further. It is likely the browser is sending some values in the HTTP Header that your code is not, and the server is returning a 500 due to these missing values. Try replicating all of the headers that the browser is using in your code to see if this resolves the problem.
It is usually browser agent. try adding a valid browser agent to your request headers along with Accept and Accept-Encoding headers,
*Edit: For example:
request.UserAgent = "Mozilla/5.0 (Windows NT 5.2; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1";
request.Headers.Add("Accept-Encoding: gzip, deflate");
request.Headers.Add("Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
request.Headers.Add("Accept-Language: en;q=0.8");