Screen scraping an application window and interacting with the mouse and keyboard - c#

The other day I found myself addicted to a Flash game and frustrated by the thing at the same time. In a moment of frustration with the game I thought I would make a 'bot' to beat it for me. Well, I really wouldn't, but it made me realize: I don't know how to interact with another application in a way to do this. Which brings me to the question, how would one take screenshots of another running application and interact with it with the keyboard and mouse. Ideally the solution would be in a managed language like c#.
When doing the background reading the net was drowning with articles on scraping HTML. There were not many articles on actually screen scraping an application.
I'm looking for a way to interface with another application rather than script/macro another application.
Could something like Xming be used to redirect the interface? http://www.straightrunning.com/XmingNotes/
Perhaps a Terminal Services client?
http://www.codeproject.com/KB/cs/RemoteDesktop_CSharpNET.aspx

Check out Sikuli, it is basically what you are looking for. It is written in Java however.
http://groups.csail.mit.edu/uid/sikuli/

I ended up making the bot which did all this and documented it in a post
http://www.charlesrcook.com/archive/2010/09/05/creating-a-bejeweled-blitz-bot-in-c.aspx

I have used AutoHotKey for application automation.

Related

Trying to Understand Windows Forms and Threading in C#

Recently I’ve been working on a windows wallpaper manager. It will scan multiple directories for images and have a settings window to customize. In addition, I want to bypass the built-in wallpaper and instead use a windows form based off the ideas discussed here: Setting a Windows form to be bottommost to allow for greater control over image placement. I have built the file operations to find the wallpapers, and various other support operations.
However, I am stumped on how to implement a windows form properly. Not just the one in the Stack Overflow post above. So far none of my CompSci classes (7 so far) have discussed GUIs, and thus I have no knowledge to work from. I turned to the internet to learn how to work with GUIs. While I learned plenty to start a thread, or show a form, I was never able to find anything that described the proper way to engineer an application with a GUI. All I have learned in respect to engineering the application is to limit the number of threads, and try to separate the GUI from the non-GUI related code (if that makes sense).
Like I have said, I have no clue what I am doing. If someone would be willing to give me pointers or even a link to something on this subject, that would be amazing!
EDIT: I have been using this as a learning exercise, and I have mostly been learning from trial and error, probably mostly error
To draw on the true desktop, you need to get a handle to it: GetDesktopWindow is the function to use. But to change the wallpaper you do not draw on the desktop, but instead you use the dedicated API for handling the desktop wallpaper: SystemParametersInfo(SPI_SETDESKWALLPAPER, ...). There is also the legacy COM Shell interface, IActiveDesktop and IActiveDesktop::SetWallpaper.
As you can see, none of the APIs to manipulate the desktop and the wallpaper are managed (C#, .Net) so you will have to jump through some hoops to get this working from C#. If your goal is to learn C# style UI, I would recommend to stick to the 'blessed' way of doing this (winforms, WPF etc) and do not try to jump straight to p-Invoke and COM interop. Perhaps you can pick another project, eg. display the images/pictures in a carousel in a form.
If you're curious still, Google for C# and SystemParametersInfo or IActiveDesktop and you'll find examples of how to do this from managed C#, many examples right here on StackOverflow.
Bonus: of course Raymond Chen has a blog on this topic: How do I put a different wallpaper on each monitor?.

C# XNA code on the web for a standard browser

Apparently, there is a way to add C# XNA projects to websites such that they can be viewed on the site from a normal web browser, presumably there are concessions to safety that the browser will enforce.
How is this done?
edit: I think the C# project might only ever runs on the server, the client being sent something it can understand.
There is http://jsil.org which will convert a compiled .Net application into a client side Javascript "app" ... and has been demonstrated to work rather successfully on XNA games.
It only handles 2D stuff and sound is / was needing to manually rewritten last time I checked but it does a pretty impressive job from what I've seen of it.
It should be noted that whilst this may get the job done, it's certainly not going to produce JS that is anywhere near the quality (i.e. maintainability) of a game that was written in JS from the start (using something like three.js or easle.js)... so if this for a commercial product, you may wish to explore rewriting / porting the game specifically for web as a serious alternative.
This isn't possible without a plugin. The plugin would work only on Windows, obviously, since XNA is built on DirectX.
There's no plugin that currently does this, to my knowledge.
It's not difficult to write a plugin, however, you'd have to do this for each browser you want the plugin to run in.
If you want your game content on the web, a better route is probably HTML5 and its Canvas functionality, which works on all modern browsers. You might be interested in this article, which discusses porting a 2D XNA game to HTML5 Canvas.
If you need 3D support, you could go the HTML5 WebGL route, although Internet Explorer doesn't support it.
Probably what you want is Google Native Client (NaCl). I haven't played with it (yet), but as far as I know, some game engines like Unity works on it.
Hope it helps.

Creating application on Windows platform which enters to the secure area of website

First of all, sorry for my English :)
I'm web developer. Generally coding in PHP.
Recently started to learn C++.
I want to create simple application which will simplfy daily routine works. For ex,Every time when I check my internet banking account, must login to my account via any browser. What I wanna do is, to create mini windows application which will store my login and pass, will login to my account, and show exact page what I see on browser directly after login. Website that I'm talking about is here
The problem, I can't find any tutorial about creating such application. Don't know where to start. Did research but found nothing about this. (Maybe I selected wrong keywords :)) So decided to post question here. I don't need your code, just give me starting point, where to learn to create applications as I described above.
And one more question: Is it important to know website's coding language before building such windows apps? I mean: PHP/C#... etc
Thx in advance
Not sure why you would pick C++ for this type of project, though it's possible, it would be quite tedious. I would recommend looking into Java, perl, ruby, python, or some other language that supports the Selenium Webdriver API. You can also use the mechanize API with Perl, which is also a bit tedious. I've done things like this in the past with success using Selenium (http://seleniumhq.org). Unfortunately, I don't think there is a Webdriver API for C++ as of yet (you could try writing one!), but there are definitely APIs available for the other languages I posted. Check out: http://seleniumhq.org/about/platforms.html#programming-languages.
EDIT
If you're more interested in screen scraping with C++, this question, and some answers within the thread, may prove useful to you as well: https://stackoverflow.com/questions/489522/library-recommendation-c-html-parser

Is it possible to embed Windows Sidebar Widgeds in own .Net Application

I'm trying to embed Windows-Sidebar-Widgets into my own .Net Application (WPF and C#), but I dont't realy know where to start. I googled a bit but only found SDK and tutorials for widget-development. I found that there is a small service in Windows, running the widgets. My thought was that I could perhaps fake this service and provide his functions whithin my programm, so that I can run and show the widgets.
Does anyone have some tips on how to do this? Or some experience in doing so? Perhaps there is even some kind of libary for that kind of stuff?
If this is not possible it is also an option to switch to an OpenSource widged-manager to use this widgeds, but this is only an emergency solution and the windows-gadgeds would be very appreciated.
Some information about my system:
I've written some kind of small windowmanager for childwindows in my application. And on the top of this I want to show the widgets in some kind of HostWindow.
Any hint will be appreciated.
P.S.: I would be willing to share the results if I (or we) get this working :-)
I've never seen a library for hosting gadgets, but since they are basically just websites you could use the MSHTML component to host any arbitrary gadget in theory.
This isn't to say it will be easy.
You'll need to extract a gadget to some temporary directory, spin up something hosting MSHTML (in WinForms WebBrowser would be convenient), navigate to the appropriate file in that temp directory (discovered by parsing gadget.xml), and shove something resembling the System object into the gadget.
As a starting point, inject some script that sets System = window.external; onload and WebBrowerser.ObjectForScripting = new FakeGadgetSystem().
FakeGadgetSystem will need [ComVisible(true)] and to implement a decent amount of stuff. I'd advise grabbing a widget or two to test with that mildly exercise the Gadget API, and then slowly implement the calls they make. It'll be a long haul for 100% compatibility with the official gadget hosting service.

Good framework for a C# developer to use for visually stunning presentation layer

I am about to write a front end app, which will be used as a media center app. It will plug directly into a high definition TV. Essentially transforming my laptop into a media player. While this concept is not new, I want custom functionality, so this is why I am not reusing existing products.
I'm a C# developer, so the app should ideally be written in C#. And there is 1 other consideration, I need to accept input via the MCE Remote.
I was considering using Silverlight for this. Would you recommend this? Or any other recommendations for frameworks before I begin planning around this.
Thanks in advance.
This is the type of stuff that the Windows Presentation Foundation was meant for. You'll get a lot more access to the hardware than Silverlight would provide (I.E. that MCE remote you mentioned). You mark up your UI with vector graphics/XAML, and then perform the logic with C#.
EDIT: WPF also has support out of the box for animations which can make your UI a lot more interactive.
EDIT 2: Scott Hanselman has written a really cool application called BabySmash and posted the source online. It basically intercepts keyboard input and shows shapes and sounds on the computer. It's a good "child-proofing" method for your PC. The code could provide you with some insight into WPF and how to do the animations and interactivity that you're looking for.
Is this a desktop app? If so I would use WPF. Silverlight is a subset of WPF, so using WPF you could potentially do more.
Silverlight or WPF, if you want some extra power. Both have a similar programming model (with XAML and code-behind) so you might be able to start with Silverlight and move up to WPF if you need.
The VLC api might be useful for playing your media, someone has created a C# wrapper for it:
http://wiki.videolan.org/.Net_Interface_to_VLC
WPF is certainly the way to go, and for playing media check out the excellent WPF MediaKit: http://wpfmediakit.codeplex.com/ I've used it successfully in many projects.

Categories

Resources