I need to create an application (ASP.NET or WinForms or Windows Service, not sure) that needs to make a call to a url including username and password for basic authentication and have the url return a csv file. I then need to use the csv file in the application. I don't know how to do this. How do I call the url in my app. There can be no user interaction, it needs to be completely automated in the returning of the csv file.
Try something like this:
var webClient = new WebClient();
webClient.Credentials = new NetworkCredential("username", "password");
webClient.DownloadFile("http://someurl/file.csv", "c:\\file.csv");
Try out the System.Net.WebRequest class. Here is a page showing general usage:
http://msdn.microsoft.com/en-us/library/system.net.webrequest.aspx
You can swap out the assignment of the Credentials property with your own NetworkCredential object to pass a custom user name and password.
I actually handle the class a little differently than the example. I ensure that every class that is IDisposable is initialized in a using statement to avoid accidentally leaving unclaimed resources. This is especially important if your service will receive frequent or rapid traffic.
Edit:
If you're interested in a CSV parsing library, there are many you can find from any search. You might like the code from this CodeProject article. This library is flexible enough to handle properly-escaped multi-line fields.
Good luck!
Related
So, I have SOAP handling classes from https://soapclient.codeplex.com/SourceControl/latest. Using provided methods, I managed to connect to server and retrieve data, like:
SoapClient client = new SoapClient("http://somelink.someserver.net/~johndoe/gogogo/servis");
XElement myEle = client.Invoke("getProjekti");
What I need to do next is to provide HTTP authentication i.e. to send login credentials to web server. There is a way of adding Username and Password to a provided SoapHeader:
client.Header = new SoapHeader();
client.Header.Name = "AuthHeader";
client.Header.Add("UserName", "student");
client.Header.Add("PassWord", "student");
And there's the roadblock for me. What to do next? Common sense and programming experience make me think that there should be, hypothetically, request method that will now somehow send that header to web server, like:
bool sendLoginRequestToServer(client)
or something, returning true if login is successful or false otherwise. Despite vigorous search I was only left puzzled, since many people use many methods, most of which I failed to understand. What makes it even more difficult for me is that most of tutorials cover C# WebService, while I have ordinary WinForms GUI application. Solution within aforementioned SoapClient would be preferable, but I would settle for anything that works.
I got a little questions for you all! Currently I have a login form on my C# application and you need to enter the right user and pass to open another form that is the real program. To do this I got this line of codes:
string response = SendRequest("http://mysite/login.php?name=" + userName);
string[] back = response.Split('_');
back[0] = back[0].Replace(" ", "");
back[0] = back[0].ToUpper();
and I got this method:
private string SendRequest(string url)
{
try
{
using (WebClient client = new WebClient())
{
return client.DownloadString(new Uri(url));
}
}
catch
{
MessageBox.Show("Error while receiving data from the server.","Something broke.. :(", MessageBoxButtons.OK, MessageBoxIcon.Asterisk);
return null;
}
}
I also have a method that checks if the entered password = the stored passwod in the database
$dbConnection = new mysqli("SERVER", "LOGIN", "PASS", "DBNAME");
$email = $_GET['name'];
$stmt = $dbConnection->prepare('SELECT password, salt FROM TABLE WHERE email = ?');
$stmt->bind_param('s', $email);
$stmt->execute();
$stmt->bind_result($pass, $salt);
$stmt->fetch();
echo "$pass" . "_" . "$salt";
This works fine and I can login when I enter the right password and I can't when I enter the wrong password.
My problem though is if I enter : http://mysite/login.php?name=[username]
where [username] is any username it returns the hashed and salted password. Since it's hashed and salted this is not a big issue but later when I will do inserts this will become a bigger problem since then someone could enter information into the tables with this method. So my question is: Is there anyway to make the PHP file only allow to return the values if the connections come from my C# application?
If you are really interested in managing this in a manner such as that, you should do one of the following:
Implement an OAUTH Configuration
You should implement a form of OAUTH in your PHP app, and generate the right proper tokens for your C# app. You would then use a full OAUTH dialogue to send/retrieve the data from the PHP server. This would eliminate the possibility for random queries to the page to return proper results. You should also implement HTTPS with this. PHP OAUTH implementation basics: http://www.sitepoint.com/creating-a-php-oauth-server/
Advantages: Security. This method provides a greater deal of security than the others, without sacrificing the robustness of the project in general. You would also be able to remove the entire GET/POST request by using tokens for each client, instead of a GET against username. Extensibility. This method can be easily extended to provide features for further apps/programmes.
Disadvantages: Complexity. This method is much more complex and has much more overhead than the others.
Modify the Request to be a POST with Secret
Another option is to change the C# programme to send a POST request to the PHP page, and send some secret value with it as well. This is not recommended, as anyone who knows the secret value could send it from a malicious page. This is equivalent to implementing basic XSS attack prevention. You should also use HTTPS for this as well.
Advantages: Simplicity. This method is the quickest/easiest to implement without removing any current features.
Disadvantages: Insecurity. This method does not provide any security benefits, apart from security through obscurity.
Alter Database Visibility
Since you are using MySQL on the PHP page to return the value, you should modify the C# programme to connect directly to the MySQL database and collect the value. This has the advantage of eliminating the possibility of someone with malicious intent querying the PHP page without your permission. Various MySQL connectors: https://www.mysql.com/products/connector/
Advantages: Moderate Security. This method does remove the possibility of PHP exploits, and also assists in keeping the database secret.
Disadvantages: Moderate Insecurity. This method requires embedding the connection string (with username and password) into the application when distributed. Certain measures could be taken to assist in eliminating some of the issues with this, but in general this method is an average method. Code Rewrites. This method requires an entire rewrite of the programming infrastructure.
Custom User Agent
Edit: Forgot to mention, another simple/easy workaround that is extremely insecure.
You could utilize a custom User Agent (similarly to the secret in the POST method.) This would allow your PHP page to determine that the request likely came from your application. You should, again, use HTTPS for this method as well. This method would not require much code change, and could be combined with HTTP_REFERER to assist in securing the origin as well.
Advantages: Simplicity. This is, by far, the easiest method to implement.
Disadvantages: Insecurity. Much like the POST with Secret method, this is extremely insecure. Anyone who knows how your User Agent is formed could quite simply and readily exploit it. Using HTTPS would likely help mitigate this risk, but it would never go away.
Examples:
C#: client.Headers.Add ("user-agent", "my-super-insecure-user-agent");
https://msdn.microsoft.com/en-us/library/system.net.webclient.aspx
PHP: if ($_SERVER['HTTP_USER_AGENT'] === "my-super-insecure-user-agent") {/*Process request*/}
http://php.net/manual/en/reserved.variables.server.php
Tl;dr
Ultimately, the choice is yours. Given the situation, I would recommend using the following advisories:
If development time is not an issue, utilize the OAUTH model. This is the most expandable and secure method.
If you can eliminate the PHP in general, use the Database Visibility model. This has the advantage of removing the inherit security risks associated with having a publicly-visible PHP page. You could also use this model, with more efficiency and speed, if the database is always local to the users network. This also means that outsiders could not access your information, if properly fire-walled.
If you need a quick-and-dirty solution, use the POST model. This would be the fastest and simplest to implement, but the least secure.
I have a simple aplication in Windows Store.
This application download and parse HTML from website.
I using a HttpClient class
Now I have a big problem becouse a page looks diffrent form specific countries and my parsing is not success.
Example: When someone from USA using my app then app downloading diffrent HTML content becouse webpage looks diffrent in specific countries.
How to set a default location in http client?
I want to have a the same HTML in all executes.
EDIT
I calling this page: LINK
You need to set the default language header when you make the request and/or consider making it a user definable setting.
http://www.w3.org/TR/WCAG20-TECHS/SVR5
ignoring the initial question for a moment
PLEASE don't write an app that depends on any kind of HTML parsing for any functionality. All the site you are calling has to do is change an ID or two in the "wrong" place and your app will fail for every user until you put out an update.
back to the answer
OK, assuming that screen-scraping is the way you want to go with your app, and assuming, of course, that the site you are scraping from allows such behaviour in their terms of use (check - it wouldn't be fun for you to get sued if you didn't read them) then I'd suggest a slightly different approach.
Since you are not guaranteed to get the same page layout for any locale your users access your app from, why not set up a web service that does the parsing work for you, and interrogate that service from your app instead of going direct to the site?
Your app <--> Your web service <--> the site providing data
That way, you always know that the data you are getting back is consistently formatted as if for a specific locale (your web server), and then you only have to maintain one piece of code to parse it. That will be much simpler whenever there is a change to the underlying data structure (and believe me, there will be changes)
The answer to this depends on how the website implements default language selection. Both of the other answers are potentially correct depending on how the specific site works.
If you can share the site URL, we can tell you a suitable strategy to use.
Setting the design flaw consideration aside for a moment (you may have or have not good reason to do screen scraping), here's how to set the Accept-Language header:
var httpClient = new HttpClient();
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, new Uri("http://www.livescore.com"));
httpRequestMessage.Headers.Add("Accept-Language", "en");
var response = await httpClient.SendAsync(httpRequestMessage);
string content = await response.Content.ReadAsStringAsync();
Try to always call the url in question with the cultureInfo path param if it has one, for example say that you are targeting microsoft.com then you would have something like this:
http://www.microsoft.com/en-us/default.aspx for english
http://www.microsoft.com/de-DE/default.aspx for german
and so on. If this is applicable to you, this would be an ideea.
This is how I have currently managed to consume a particular Microsoft web service. Notice that it is located on an HTTPS server and that it requires a username, a password, and a .cer file to be installed in the operating system's "root certificate authorities".
WSHttpBinding binding = new WSHttpBinding();
binding.Security.Mode = SecurityMode.TransportWithMessageCredential;
binding.Security.Message.ClientCredentialType = MessageCredentialType.UserName;
binding.Security.Message.NegotiateServiceCredential = true;
binding.Security.Message.AlgorithmSuite
= System.ServiceModel.Security.SecurityAlgorithmSuite.Default;
binding.Security.Message.EstablishSecurityContext = true;
EndpointAddress endpoint = new EndpointAddress("https://address.of.service");
//"GreatClient" was created for me automatically by running
//"svcutil.exe https://address.of.service?wsdl"
GreatClient client = new GreatClient(binding, endpoint);
//Username and password for the authentication. Notice that I have also installed
//the required .cer certificate into the system's "root certificate authorities".
client.ClientCredentials.UserName.UserName = "username";
client.ClientCredentials.UserName.Password = "password";
//Now I can start using the client as I wish.
My question is this: How can I obtain all the information necessary so that I can consume the web service with a direct POST to https://address.of.service, and how do I actually perform the POST with C#? I only want to use POST, where I can supply raw XML data using POST directly to https://address.of.service and get back the result as raw XML data. The question is, what is that raw XML data and how exactly should I send it using POST?
(The purpose of this question: The reason I ask is that I wish to consume this service using something other than C# and .NET (such as Ruby, or Cocoa on Mac OS X). I have no way of knowing how on earth to do that, since I don't have any easy-to-use "svcutil.exe" on other platforms to generate the required code for me. This is why I figured that just being able to consume the service using regular POST would allow me to more easily to consume the service on other platforms.)
What you are attempting to do sounds painful to do now and painful to maintain going forwards if anything changes in the server. It's really re-inventing the wheel.
If you haven't considered it already, I would:
(a) Research whether you can use the metadata you have for the service and use a proxy generator native to your target plaform. There aren't many platforms that don't have at least some tooling that might get you part of the way if not all of it. Perhaps repost a question targetting Ruby folk asking what frameworks exist to consume an HTTPS service given it's WSDL?
(b) Failing that, if your scenario allows it I would consider using a proxy written in C# that acts as a facade for the service which translates it into something easier to consume (for example, you might use something like ASP.NET MVC WebAPI which is flexible and can easily serve up standards compliant responses over which you can maintain total control).
I suspect one of these may prove easier and more valuable than the road you are on at the moment.
I had to go through something similar when porting .NET WCF code to other platforms. The easiest approach I found was to enable message logging on the WCF client. This can be configured to save both envelope and body and once everything is working on the .NET side of the house, you can use the message log to have "known-good" XML request/response to port to other platforms.
I found this approach to be more elegant since I didn't have to add an additional behavior to log messages, and it can be easily enabled/disabled/tweaked in the config. The Service Trace Viewer Tool that ships with Visual Studio is also handy for reviewing the log files.
I think when you say that the service should be consumed from other platforms, which do not have proxy class generation logic, you can go with REST services. This will allow you to create input as simple string concatenation instead of complex XML. Though its applicability depends on the situation.
Check this discussion : http://social.msdn.microsoft.com/Forums/en-US/wcf/thread/6907d765-7d4c-48e8-9e29-3ac5b4b9c405/
As far as the certificate is concerned, refer http://msdn.microsoft.com/en-us/library/ms733791.aspx on how to configure it.
I know this is not a very precise answer, but you will be the best person to evaluate above procedure, hence posted. Hope it helps.
What I'll do:
1- Create a small c# app that can post on this webservice (using svcutil). And modify it to show the XML send/received. To view the xml there are several ways: logging, wireshark etc. To add it directly to the small app there is another question here that give a good answer.
2- Once you know what you have to send, you can do it in c# like this:
// implement GetXmlString() to return the XML to post
string xml = GetXmlString();
// create the url
string url = new UriBuilder("http","address.of.service",80).ToString();
// create a client object
using(System.Net.WebClient client = new System.Net.WebClient()) {
// performs an HTTP POST
client.UploadString(url, xml);
}
I'm not a .NET programmer but I've had to interoperate with a few .NET services and have lots of SOAP/WSDL experience. Sounds like you've captured the XML for your service. The other problem you'll face is authentication. OOTB, .NET web services use NTLM for authentication. Open-source language support for NTLMv2 can be hit and miss (although a quick google search pulled up a few possibilities for ruby), and using NTLM auth over HTTP may be something that you have to wire together yourself. To answer a question above: where are the auth creds? If the service is using NTLM over the wire, authentication is happening at some layer below HTTP. If the service is using NTLM to authenticate HTTP, your NTLM creds are in the HTTP Authorization header. You should be able to tell with wireshark where they are. You'll also probably need a SOAPAction header; this can also be sniffed with wireshark. For the C# client, I'm sure there are docs explaining how to add headers to your request.
We have an in-house application that pulls some data from some of Amazon's pages occassionally (We know they have APIs for certain operations... what we're doing requires some custom info not included in the APIs). We have never had a problem pulling their pages, but suddenly Amazon is returning "(503) Server Unavailable" on pretty much every request, and this has been happening for several days, so we doubt it's a temporary thing. Even something as simple as this:
System.Net.WebClient client = new System.Net.WebClient();
string data = client.DownloadString(new Uri("http://www.amazon.com/Bose-Companion-multimedia-speaker-Graphite/dp/B000HZBR64/"));
The strange thing is that these pages load just fine in a web browser, but any time we try to pull them through code, it is failing.
What could cause these functions to fail? Is it possible that they changed something on their end and that we need to do some custom logic with our calls?
After some further testing, it turns out that this was happening because Amazon needs the Accept parameter of the HttpWebRequest to be specifically set. When setting it to:
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
Everything worked fine. This is a recent change, so they must have altered something on their end.
check the user-agent of you request. make the user-agent the same as your browser. And check if you set any proxy for your app? maybe your browser and your app are using different proxies