Auto populate / screen scrape multipage webform - c#

Firstly apologies if this is a duplicate question, I have spent a while searching and can't find anything that looks to be the same.
I need to automate the completion of a multi page web form and then process the result from within an asp.net system. The ideal solution would be to create a web service which takes in some data, then processes it through the website and returns a result - I can then use this in any app that requires this functionality. The form that needs completing is quite complicated, and also includes some if / then / else logic when going through e.g:
Complete personal details
Enter postcode
If 1 result found goto 5
Display list of possible addresses, when one selected goto 5
If time at address < 3 years, display new address page
etc.
I have created a solution which launches an ActiveX browser control and controls the website as if a user were entering the data which works well however is subject to a limit on concurrent users (anything > 1 starts to cause issues, >2 is not allowed by windows default settings), is there a better way of doing this? I've heard of Selenium although not used it, would this be a possible solution?
I have looked at using the WebClient class, and have used this in the poast for screen scraping systems however I don't know how I could chain these calls together to allow for the full workflow.
FYI - this work is all being undertaken with the full knowledge and consent of the site owner. They have a queue of development tasks which building an internal API does not feature on so whilst they are happy for this type of solution to be implemented, will not be able to change anything their end.
Thanks in advance

There is a .NET port of Watir called WatiN - it might be worth giving that a try if you haven't already.

Related

Correct way to start back ground work ASP.NET MVC 5

I'm building a ASP.NET MVC 5 web page currently. It's worth noting I'm fairly new to ASP.NET itself, prior to this I've only dealt with windows desktop applications.
My question is I'm unsure where to start my recurring background work. This work fires checks and updates to the MySql database on the server every few seconds. Currently I'm starting these checks just by adding a start method in the RouteConfig.cs class which I know is the wrong way to do it, just a quick hack while I search for the correct method. Where should I be starting this background work which should be kicked off as soon as the web page is available not when it's first searched for via the browser by a user?
This brings me to another question, a lot more work seemingly but would it be a better/more standard solution to write the background work as a windows service and establish a sort of interface between the web page front end and the windows service?
Search for Quartzin package manager, its best schedular for .net

how to make windows startup project with many methods running at the same time in the background

I have explained my project below and asked some questions with "My Question-" tag.
I have working on a project. At the time of windows logon page if I enter a wrong password my cam should take the picture, If I open regedit my system should take a screen shot and save these images in C:\Windows\system32\new folder (I tried a lot making this work with the help of manifest files but failed everytime) and emails it whenever finds an internet connection
I have a form based app because I didn't find any other way to capture image from webcam directly but taking input from pictureBox1.Image.
My cam, screenshot,email (didn't find a way to autocheck if has internet connection available or not) and 3 events checker for "firewall enable/disable, windows logon failure , regedit event called" are done and they are working good.
What I need to do is to assemble these codes to work as an app and running in the background continuously from the time of windows startup to shutdown
To validate positive events I need to make a desktop based db ("My Question"- still figuring out either to choose sql or localdatabase in c#. Please also tell me a suitable solution.I have to delete all the entries from the db once a day is over). The db would contain the following columns (event id, event name, event timestamp).
I want my app to check if this very event exists in the db then it should ignore the event generated on windows event log else it should make a new row with the db columns and it should do the following actions based on the event like taking webcam pic or screenshot.
"My Question"- I want my app to be live at the time of windows logon page. A lot of programs start later when you are authenticated but I need my program to be live at the time of logon page. Do I have to make 1 or many services? or multi-threaded? because in the typical form based app you can only call one function at a time and wait for it to return something or perform some task/action and then you call the second third whatever.
"My Question"- Do I need to use the backgroundworker in c#
Please help!
You have a lot of things going on here for one question.
You can put all your code in a background service that gets started at boot time. There is a walkthrough here to show you how to do that (along with a million other sites).
Addressing some of the other issues you listed:
Google is your friend...
Webcam - Found a quick reference here and here
File Modification - Another SO thread here
SQL vs. Other Database - Not sure you need anything elaborate here, probably something you can put together pretty quickly. Another SO thread addressing that here
Good Luck!

Allowing a WPF browser application to navigate to a separate web page when finished?

I have a WPF browser application that collects user data and adds it to a database to tell them when their software is out of date.
All of that works fine, but the problem is when the application finishes its stuff, I want the web page itself to change (i.e., detect the web app has hit a 'finished' state, then autonagivate to a results page or something).
I can't think of a way to accomplish this, since the web app itself doesn't seem to be able to change the IFRAME it's contained in, much less the page outside of that, or signal to javascript or anything.
Any ideas?
I'd make an variable to keep progress/step of work. And a timer which would check if progress=="done" or sth.
Maybe this is not the best way of solving this but I don't know WPF much and that solution first came to mind

How to get focused elements of specific process in c#

im stuck in my problem, i have invoke a process and now i want its focused elements.
for example. i run a process www.google.com that opens up in Google chrome
now lets say i want to search "hello" the way of doing this as far as i know is to get the keyboard focus element that will be a textbox of google.com .so how to do that through my application i.e programmatically(getting keyboard focused element of chrome process)
Process[] pr = Process.GetProcessesByName("chrome");
var target = pr[0];
Your question a is a bit vague, so I'll try to give several pointers based on several different interpretations:
1) If the process you're referring to is always a web-site, you should skip the screen-scraping techniques, like #vulkanino mentioned, and simply fetch the contents of the web-site directly. If you must get the website as parsed by a real browser (rather than the raw HTML itself), you can use screen-scraping, but remember that not all computers use the same browser.
2) If you want to screen-scrape information from a running process, you (unfortunately) don't have a simple API like the one you expected, where the Process pr object contains a list of elements you can traverse. This is because the process might have been written in .NET/WinForms, .NET/WPF, C++/MFC or any of a dozen other UI frameworks that work very differently.
What I would suggest is using a screen-reader accessibility framework, such as Microsoft's UI Automation, which is able to extract text from the UI of many common applications. This functionality exists to help read out application UI for the hard of sight, but can be used for screen-scraping as well.

How to dynamically run process in asp.net

I am designing a website and it uses Windows Forms (in Visual Studio 10) in which for example i have five-six URLs. Now i am displaying them on home page of my website xyz.com
What i want is, i want to calculate total no. of tweets for all links and display links based on no. of times they are being tweeted/retweeted.
for a url we can calculate no. of tweet using twitter api http://urls.api.twitter.com/1/urls/count.json?url=YourURL
I know all the stuff like receiving JSON values in a string and parsing json to retrieve tweet counts and then compare and display links based on the priority etc.
What i have been using till now it is initiating all the process using a Click_Button.
But i want to know how can i automate this all for each 10 minutes. Its like a end user can see urls priority with just refreshing the page.
One way to do this is to run a scheduled task ever 10 mins which interacts with the DB. The web application also interacts with the DB and thus the two systems are distinct.
Side note: it is strongly recommended to use only console applications as scheduled tasks. If you make a windows form application will will have some issues.
As Kieren Johnstone has pointed out in another answer the best way to do this would be to write a windows service.
I still recommend the solution as described above as a first step since it is easy to debug and test.
Additionally, give some serious consideration to logging and error reporting -- with background tasks you can never know to much about what the heck it was doing when it broke.
If timing itself is not important (it doesn't have to be 10 minutes precisely), I would suggest binding to any event that fires when users use your application. No point in calculating anything if noone is using it :-)
So you could use a login, or page load, or whatever happens at an interval roughly like the interval you wish to achieve.
You can always store a DateTime variable somewhere that you can check to see when the calculation was last made. Something like:
public void MyEventHasFired()
{
DateTime dateLastProcessed = ... //Database? Session data? Anything goes.
if(dateLastProcessed < DateTime.Now.AddMinutes(-10))
{
//calculate
...
dateLastProcessed = DateTime.Now;
}
}
The best solution is definitely a Windows Service. It can be started, stopped and managed well, it's easy to log, maintain..
Scheduled Tasks are very prone to problems. At least in a Windows Service you can configure it to start automatically, re-start if there's a problem, you can control the timing yourself in the code, and catch/handle exceptions as you wish.
The best scheduler i know is Quartz.net
It'is not simple to use but it works great.
You can find an example with asp.net there http://blogs.planetcloud.co.uk/mygreatdiscovery/post/ASPNET-Scheduled-Tasks-with-QuartzNET.aspx
Anyway i agree with Kieren Johnstone: you should use a windows service

Categories

Resources