Develop Reference

Using .Net how can I programmatically navigate to a webpage, interact with it via code, then get particular values from a newly generated page

I have a scenario where I would like to automate programmatically the following process:
Currently, I have to manually
Navigate to a webpage
Enter some text (an email) in a certain field on the webpage
Press the 'Search' button, which generates a new page containing a Table with the results on it.
Manually scroll through the generated results table and extract 4 pieces of information.
Is there a way for me to do this from a Desktop WPF App using C# ?
I am aware there is a WebClient type that can download a string, presumably of the content of the webpage, but I don't see how that would help me.
My knowledge of web based stuff is pretty non-existent so I am quite lost how to go about this, or even if this is possible.

I think a web driver is what you're looking for, I would suggest using Selenium, you can navigate to sites and send input or clicks to specific elements in them.

Well, I'll write the algorithm for you but you also need to some homework.
UseWebClient get the htm page with the form you want to auto fill and submit
Us regex and extract the action attribute of the form you want to auto submit. That gets you the URL you want to submit your next request to.
Since you know the fields in that form, create a class corresponding to those fields, let's call the class AutoClass
Create a new instance of your auto class and assign values you want to auto fill
Using WebClient to send your new request with the url you extracted from the form previously, attach your object which you want to send to the server either through serialization or any method.
Send the request and wait for feedback, then further action

Either use a web driver like Puppeteer (Selenium is kinda dead) or use HTTPS protocol to make web requests (if you don't get stopped by bot checks). I feel like your looking for the latter method because there is no reason to use a web driver in this case when a lighter method like HTTP requests can be used.
You can use RestSharp or the built in libraries if you want. Here is a popular thread of the ways to send requests with the libraries built in to C#.
To figure out what you need to send you should use a tool like Fiddler or Chrome Dev Tools (specifically the Network tab) to see what you need to send to acheive your goal as you would in a browser.

Upload a file in browser using Selenium C# without NoFileChosen option

I am trying to upload a file. My application only has a Browse button and no place to send any file path. So I am unable to use SendKeys.SendWait(#"filepath"); I tried using JavaScriptExecutor but getting error while using the below code:
IWebElement upload = _driver.FindElement(By.XPath("//div[starts-with(#class,'addDoc') and contains(text(),'Browse')]"));
string filepath = #"C:/Users/../Amazon vs Walmart.pdf";
String script = "document.upload.value= " + filepath;
((IJavaScriptExecutor)_driver).ExecuteScript(script);
Runtime.evaluate threw exception: SyntaxError: Unexpected token :
Why am I getting the exception?

Interacting with file browsing windows is outside the scope of what the webdriver can do. When you click browse the browser passes you off to the OS to find the file. Javascript cannot act on these windows.
SendWait is used as a trick to pass the uri of the file to the open the window and return. If that is not working you will need to use something to interact with OS windows.
I personally use InputSimulatorCore to handle issues like this when working with IE, which has lots of these type of issues with file downloads.
Edit: If you control the application changing to a forms based upload will allow you to do the testing in Selenium and is the best option. If not you can also directly do a post to the upload endpoint.

There are multiple issues with the code you posted. First, you’re attempting to call ExecuteScript with invalid JavaScript. To wit, once your strings are concatenated, your JavaScript looks like this:
document.upload.value= C:/Users/../Amazon vs Walmart.pdf
Note that there are no quotes around the string you’re actually attempting to set. The syntactically correct JavaScript would be:
// Note carefully the quotes around the string literal.
document.upload.value='C:/Users/../Amazon vs Walmart.pdf'
To yield that, your concatenation code would need to look like this:
string script = "document.upload.value='" + filepath + "'";
The second issue is that you’re attempting to invoke uploading a file. Assuming the upload is accomplished by using standard HTML mechanisms, that means that somewhere on the page, there is am <input type="file"> element. It might be hidden, but it’s there somewhere on the page. To upload a file, you can use the SendKeys method on that element. The file upload case is one of the very few exceptions for SendKeys to the rule that elements to be interacted with must be visible to the user (at least with the most recent versions of browsers and browser drivers. If the application is using some non-standard upload mechanism, like a pure JavaScript implementation, or some sort of Flash component, then you’ll need to use some other method to communicate with that component.

Though I tried using AutoItX to upload file and it worked. Below is the code for the same -
to get AutoItX, I installed Nugget package - AutoItX.DotNet
AutoItX.ControlFocus("Open", "", "Edit1");
AutoItX.ControlSetText("Open", "", "Edit1", filepath);
AutoItX.ControlClick("Open", "", "Button1");
But still, I would like to explore using JS as well.

Nancy.ViewEngines.ViewNotFoundException: Unable to locate view 'index.php'

I am making a project where I need to host a simple website. Nancy seems to be a really easy and straightforward library. Only a couple minutes in and I've got a problem no one has addressed online. How do I host an "index.php" file with Nancy?
I'm hosting a *.php file because I need the php script to run on startup, is this even possible with Nancy?
So far all I've got is a "MainModule" class derived from "NancyModule" and it says Get["/"] = p => View["index.php"];. When I navigate to the page however, in my Firefox browser it gives me the error
Nancy.ViewEngines.ViewNotFoundException: Unable to locate view 'index.php
how do I fix this with Nancy or is there any other http web hosting libraries for C# I could use?

After further research I found that Recaptcha has an explicit way to use it. This way allowed me to instead of getting the response through PHP I can get it using Javascript.
var code=document.createElement('p');
code.innerHTML=grecaptcha.getResponse(widgetId1);
document.body.appendChild(code);
I put this code in the action of the form, so when the Recaptcha is solved it will display the response as a paragraph and then it would be obtained using an HtmlParser such as HtmlAgilityPack for C#.

Inserting text from a C# WinForms application into a web browser textbox

I'm currently working on a WinForms application that I'm hoping will allow users to enter stored passwords into a web browser without a keylogger being able to see the password. I've got the storing part completed, but I can't figure out how to insert the password into a browser without using either the clipboard or .NET's SendKeys class, both of which can be seen by most keyloggers.
I looked into using Windows messages to do this, but, from what I understand, you need to know the hWnd of a control in order to send it a Windows message. Unfortunately, it appears that web browsers do not use hWnds for individual textbox controls.
I know there has to be a way to do this. In Notepad++, for example, you can drag text into a web browser textbox and the text doesn't show up in keyloggers. Same goes for Neo's SafeKeys. Anyone have any ideas on a general approach to doing this? Thanks so much.

What you want is a WebDriver Library. Unfortunately .net does not have plenty of good ones. You can take a look at WatiN or selenium
From what I understand watin isn't supported anymore and is not multi-browser compatible.
Here is an example of what you can do with selenium
using (var driver = new ChromeDriver())
{
driver.Navigate().GoToUrl("http://testing-ground.scraping.pro/login");
var userNameField = driver.FindElementById("usr");
var userPasswordField = driver.FindElementById("pwd");
var loginButton = driver.FindElementByXPath("//input[#value='Login']");
userNameField.SendKeys("admin");
userPasswordField.SendKeys("12345");
loginButton.Click();
}
I image you can link your driver to an existing opened browser
If you are able to run ruby on the computers I would recommend using Watir
and running the ruby script doing the web manipulation from the c# using processes.

What is the best way to crawl a login based sites?

I've to automate a file download activity from a website (similar to, let's say, yahoomail.com). To reach a page which has this file download link, i've to login, jump from page to page to provide some parameters like dates etc., and finally click on download link.
I am thinking of three approaches:
Using WatIN and develop a windows service that periodically executes some WatiN code to traverse through the page and download the file.
Using AutoIT (no much idea)
Using a simple HTML parsing technique (there are several questions here eg., how to maintain a session after doing a login? how to do a logout after doing it?

I use scrapy.org, it's a python library. It's quiet good actually. Easy to write spiders and it's very extensive in it's functionality. Scraping sites after login is available in the package.
Here is an example of a spider that would crawl a site after authentication.
class LoginSpider(BaseSpider):
domain_name = 'example.com'
start_urls = ['http://www.example.com/users/login.php']
def parse(self, response):
return [FormRequest.from_response(response,
formdata={'username': 'john', 'password': 'secret'},
callback=self.after_login)]
def after_login(self, response):
# check login succeed before going on
if "authentication failed" in response.body:
self.log("Login failed", level=log.ERROR)
return
# continue scraping with authenticated session...

I used mechanize for Python with success for a few things. It's easy to use and supports HTTP authentication, form handling, cookies, automatic HTTP redirection (30X), ... Basically the only thing missing is JavaScript, but if you need to rely on JS you're pretty much screwed anyway.

Try a Selenium script, automated with Selenium Remote Control.

Free Download Manager is great for crawling, and you could use wget.