Hi
I'm trying to write a windows application which can read HTML content of a specific website and fill some data in some input fields and submit the page.
what I did untill now was reading page content from a WebBrowser object, navigated to the website.
I know that i need to create some request/response variables and work with them, but I have no good view on what i'm trying to do.
also, my information about HttpRequest and HttpResponse is so low...
HttpWebRequest is what you're looking for, examples for read/write to scrape abound.
WebClient might be simpler for you to implement but in the end is only a wrapper to the above.
You can try to use the approach described in the HttpWebRequest with https in C# thread as a starting point.
Related
I have a scenario where I would like to automate programmatically the following process:
Currently, I have to manually
Navigate to a webpage
Enter some text (an email) in a certain field on the webpage
Press the 'Search' button, which generates a new page containing a Table with the results on it.
Manually scroll through the generated results table and extract 4 pieces of information.
Is there a way for me to do this from a Desktop WPF App using C# ?
I am aware there is a WebClient type that can download a string, presumably of the content of the webpage, but I don't see how that would help me.
My knowledge of web based stuff is pretty non-existent so I am quite lost how to go about this, or even if this is possible.
I think a web driver is what you're looking for, I would suggest using Selenium, you can navigate to sites and send input or clicks to specific elements in them.
Well, I'll write the algorithm for you but you also need to some homework.
UseWebClient get the htm page with the form you want to auto fill and submit
Us regex and extract the action attribute of the form you want to auto submit. That gets you the URL you want to submit your next request to.
Since you know the fields in that form, create a class corresponding to those fields, let's call the class AutoClass
Create a new instance of your auto class and assign values you want to auto fill
Using WebClient to send your new request with the url you extracted from the form previously, attach your object which you want to send to the server either through serialization or any method.
Send the request and wait for feedback, then further action
Either use a web driver like Puppeteer (Selenium is kinda dead) or use HTTPS protocol to make web requests (if you don't get stopped by bot checks). I feel like your looking for the latter method because there is no reason to use a web driver in this case when a lighter method like HTTP requests can be used.
You can use RestSharp or the built in libraries if you want. Here is a popular thread of the ways to send requests with the libraries built in to C#.
To figure out what you need to send you should use a tool like Fiddler or Chrome Dev Tools (specifically the Network tab) to see what you need to send to acheive your goal as you would in a browser.
After searching in google and here, i saw the popular approach is to convert the image to byte array and then to base64 string. this part was easy, but sending it and receiving it over HTTP is harder, and I can't find an easy way to do it.
I have 2 main question which depend one another
Send with android on HTTP:
Android 6 deprecated HttpClient, so i don't want to use that.
I thought to use Volley but i cant figure out how to make it work right.
Can you please give me an example of code to transfer it in a simple and elegant way which will be easy to intercept with C#?
Receive with C# and use of web service:
I'm not sure what is the best way to implement it? Should I create a web service method? in case i should, how can i intercept the post request? Should I create a new page to handle only this part. this way i know how to handle the request.
Edit:
I managed to create a request using HttpClient, but the base64 string after converting the file made the URI too long.
any other ideas?
I found here a post for uploading files from Android to ASP.NET Web API. However the HTTPClient was used for handling HTTP request but I think you can use code as reference.
Found the simple and elegant solution i wanted!
I use loopj library, "android-async-http" for sending files with 3-4 lines.
Then i get the request to a new aspx file in my ASP.NET server, and save it using "Request.Files" object.
I would like to open a webpage in Internet Explorer using the WebBrowserTask but I would also like to pass a POST parameter(s) as well.
Is this possible using the WebBrowserTask?
EDIT: I forgot to mention that this is part of an XNA project. All I would like to do is to display a webpage and passing POST parameters with the request
You can also use the WebClient object to accomplish this though it is much more obscure. Webclient is a wrapper around HTTPRequest. It is like HTTPRequest on easy mode. The trade off is that if you want to send post data then you need to be crafty about it. You can create new headers in WebClient and set their names to what you need. Then when you make the call it will send them out as post data. Sounds complicated but actually pretty easy and clean when implemented. WAY cleaner than HTTPRequest.
WebBrowserTask can only navigate to a specific page and invoke GET request. You can simulate POST requests using the HttpWebRequest, but I doubt that will help you with what you are trying to do.
This is not possible with the WebBrowserTask, however it is possible using the WebBrowser control.
Specifically check out the Navigate method of WebBrowser.
This takes a byte[] with your POST data.
public void Navigate(
Uri uri,
byte[] postData,
string additionalHeaders)
I've application that uses another web sites data so how can i get it because it uses some JavaScript functions to get that data and it not show in page view-source.
Check the NET tab in firebug, XHR and check the resource that is requested, and request the same resource.
Basically you have to render the webpage and ensure the javascript functions are run (evaluated). You could do this by "borrowing" their javascript files (by linking to them from your own page), but this may not work as you don't know what's in those files - they could be accessing DOM elements that you don't have in your page, or calling to other domains which may prevent them from working correctly.
The easiest way to show the same data is to just host the page inside an iframe on your own page. If you are looking to do this from a normal client application (i.e. not a web app) then you will need a browser control that you navigate to the target page. If the browser control is invisible you could then scrape values from it and show them in your app, although this is a very clumsy way to do it, and it's debatable about how ethical it is.
If you want the another web site view source use the HTTPWebRequest to get the response stream in c#.
I have an idea for an App that would really help me out in work but I'm not sure if it's possible.
I want to run a C# desktop application that will ask for a value. When a value is supplied, the application will open a browswer, go to a webpage and add the value into a form on an online website. The form is then submitted and a new page is loaded that contains a table of results. I then want to extract the table of results from the page source and write code to parse the result values.
It is not important that the user see's this happen in an actual browser. In other words if there's a way to do it by reading HTTP requests then thats great.
The biggest problem I have is getting the values into the form and then retrieving the page source after the form is submitted and the next page loads.
Any help really appreciated.
Thanks
Provided that you're only using this in a legal context:
Usually, web forms are sent via POST request to the web server, specifically some script that handles it. You can look at the HTML code for the form's page and find out the destination for the form (form's action).
You can then use a HttpWebRequest in C# to "pretend you are the form", sending a POST request with all the required parameters (adding them to the HTTP header).
As a result you will get the source code of the destination page as it would be sent to the browser. You can parse this.
This is definitely possible and you don't need to use an actual web browser for this. You can simply use a System.Net.WebClient to send your HTTP request and get an HTTP response.
I suggest to use wireshark (or you can use Firefox + Firebug) it allows you to see HTTP requests and responses. By looking at the HTTP traffic you can see exactly how you should pass your HTTP request and which parameters you should be setting.
You don't need to involve the browser with this. WebClient should do all that you require. You'll need to see what's actually being posted when you submit the form with the browser, and then you should be able to make a POST request using the WebClient and retrieve the resulting page as a string.
The docs for the WebClient constructor have a nice example.
See e.g. this question for some pointers on at least the data retrieval side. You're going to know a lot more about the http protocol before you're done with this...
Why would you do this through web pages if you don't even want the user to do anything?
Web pages are purely for interaction with users, if you simply want data transfer, use WCF.
#Brian using Wireshark will result in a very angry network manager, make sure you are actually allowed to use it.