I want to create an application that will run in the background and track the click actions of the mouse, and when the user clicks on an external WPF or Winforms app, the background app should be able to detect the clicked control and display its ID/name/text.
I'm not sure if this is possible, but since there are automation tools that can perform similar actions, I think it should be possible. For example, with some tools it is possible to get the "window" object using the external app's PID, and then the controls (for example a button) can be accessed by providing the ID of the control. However in my case it is the other way around. I need to get ID/name/text of the control the user has clicked.
I can obtain the mouse position using windows Hook
I can obtain the hwnd and Process ID of the window the user has clicked
So is it possible to obtain the information about the element clicked? Any help and advice is appreciated, thanks in advance.
Related
I'm trying to programmaticly (from my own application) press a button that is located on another running windows (WPF) application.
Todo so, a handle to the button would be needed. There are plenty of posts about how to get such a handle.
However, it seem that WPF applications don't present handles for their graphical components:
https://www.experts-exchange.com/questions/25665615/Spy-or-Windows-Api-Functions-not-locating-controls-on-child-window.html
Any clues on how to achieve an automated press on a button in such an application?
I don't have control of the source code for the application with the button.
I believe you'll need to look into UI Automation https://learn.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-overview
I am working on some UI automation for verify tab order of our WPF application. I was wondering if there was a Windows API or UIA API for getting the current tab order of the application under test. My UI automation is written in C# but I am more then willing to do some pinvokes to make this work.
Here is some clarification based off of #CodyGray's comment. My company's has certain tab orders that we want to rarely change if ever because our users are so used to these tab orders as part of their workflow (they operate the user interface faster than it can update via these tab orders...). The current way I am doing this is through keeping a list of the tab orders then using SendKey to tab and check which control has focus. I then double check if the focused control is the control I expect it to be from a hard coded list. I would prefer to actually ask the application what it's tab order is going to be instead of using SendKey which could send tabs to the incorrect window based on what has focus.
I'm writing a program in C# to automate another application. I use user32 api. When the external application needs user input, my C# program fills the boxes and presses the buttons. When a new window comes up, I can find it and it's controls using findwindow() and findwindowex() function but when this dialog comes up (in the picture) I cannot find the controls in it(the buttons). Even UI Spy cannot show them. I tried to send key events to this window but nothing happened. Can you give me some tips how to go on?
I have a Silverlight control containing an image. I want the user to be able to drag the image out of the Silverlight application and drop it anywhere they would be able to drop an image. For example, to the Desktop or to a PowerPoint slide or Word document. Everything that I have read thus far says it cannot be done but I find that hard to believe. I'm very new to Silverlight and RIA development so any help would be much appreciated.
Below is the code sample in my WinForm Form but the drag never starts.
string[] aString = { imagePath };
DataObject data = new DataObject(DataFormats.FileDrop, aString);
data.SetData(DataFormats.StringFormat, imagePath);
DoDragDrop(data, DragDropEffects.Copy);
Well the trouble is that a drag operation in Silverlight doesn't have simple access to anything outside the browser (by design). Depending on the user's settings you even have to get explicit permission for clipboard operations and sandboxed temporary file storage. This really sounds like a task better suited to a WPF application (perhaps with web deployment?) or some other desktop application technology.
However, that being said here are some things you could try/consider:
Silverlight/Javascript/ActiveX combination hosted in Internet Explorer
Silverlight 5 "Out of Brower" & P/Invoke (I heard P/Invoke will be supported when Silverlight 5 comes out)
Silverlight connecting to a web service running on the same computer (crazy, but you didn't ask for "not crazy", you asked for possible)
I am not very familiar with drag and drop in the Win32 API so it would take a lot of research and experimentation before I could confirm that this was even possible (and I can already tell you it isn't practical).
Edit: Based on the extra information you provided about the question I suspect it is possible to do what you are attempting. First, are you using WPF or WinForms? I assume WPF but one of your comments says WinForms. I wasn't very familiar with WPF drag/drop operations, but having looked into it, I think your code is on the right path. I created a WPF application and initiated a drag during a KeyDown event. This meant that the mouse button was not necessarily pressed. If I initiated the DragDrop while the button was down it worked. If I initiated while the mouse button wasn't down, I had to push the mouse button down and the drag operation would start (this was unexpected since I assumed the mouse would have to already be down). If I pressed the mouse down outside the application, then gave the WPF app focus (ALT+Tab), then initiated the DragDrop while the mouse button was still down, it didn't work. I got a reference to the MouseDevice and checked the LeftButton property, and the state was showing as "Released" even though the button was still being held down. It seems the key here is the way drag/drop interacts with internal mouse state. You might have to find a way to set the mouse state (maybe with the UI Automation API?). At this point it should be painfully obvious that this whole thing is a hack (even though it is probably possible to get it to work somehow).
The solution we came up with was as follows. The RIA i.e. Silverlight sends a message to our Desktop application WinForms with the path of the image to drag along with the bounding rectangle in screen coordinates that we want to start the drag from. The Desktop code creates and places a Panel over the area that we want to drag from. This panel is where we use DoDragDrop to initiate the native drag when the user left clicks. Since this panel is placed outside and above the silverlight control, everything works perfect. Sandbox defeated.
I have been working on some Silverlight apps for the past few months and fully investigated your exact requirements only to find it was not possible. I believe you can drag from the file system, from Silverlight control to control, but not to the file system.
Does Silverlight 4 support drag and drop from app to desktop?
http://msdn.microsoft.com/en-us/library/dd772166%28v=vs.95%29.aspx
Is it possible to get a control at the current mouse location if its in a window not known by the application? (not in Application.OpenForms)
I implemented a low level mouse hook which can get the mouse location anywhere on the screen, but I can't seem to access controls outside of my application. What I want to be able to do is get a button in an OpenFileDialog. I can get a Handle to the dialog, but I can't get the control from the dialog using Control.FromHandle() since this only works for controls that are in the Application.
It's possible with the Win32 API. You need GetDlgItem() to get a handle to button you are looking for. Use Spy++ to find the control ID. An example of the kind of code you need is available in my post in this thread. That example is for an in-process dialog, it would work out-of-process as well.
Beware that your code is liable to break on the next version of Windows.
If you are trying to get a .NET Control, this is not possible. The reason for this is that the Control IS the HWND. With "HWND" here I mean the actual Windows window item. It is not possible to e.g. automatically wrap a new control instance around an existing window, much less as it's possible to get the Control instance from the other .NET application, if it were one.
If you are trying to insert a .NET Control into an OpenFileDialog, this also will not be possible because the OpenFileDialog is not a .NET form.
If you really want the HWND from the mouse location, you can use pinvoke to get it using the WindowFromPoint function.