Develop Reference

Keyboard input lost when using cross-process parent/child windows

I am investigating a bug that results in key-presses being 'lost' when users are providing rapid keyboard input to our application. It seems to happen randomly. If a user writes e.g. "I am writing this text fast on my keyboard" the actual text that ends up in the application's text box looks something like "Ia wrting thistext fstn mykeybrd."
We have managed to find out that the issue seems to be related to how we integrate the text-input component into our main application. Our main app is a .NET Framework 4.8 WinForms process. The text-input component is running as a .NET 4.8 WPF app in a separate process. The WinForms app is responsible for launching the WPF process, passing the window handle of its own main window as an argument to the new process. Once launched, the WPF process uses the provided window handle to reparent itself as a child window of the WinForms app. This is done using Win32 API calls through P/Invoke.
private void Reparent() {
ChangeStyle(this.myHandle);
Win32.SetParent(this.myHandle, this.hostHandle);
}
private static void ChangeStyle(IntPtr myHandle) {
const int GWL_STYLE = -16;
const uint WS_POPUP = 0x80000000;
const uint WS_CHILD = 0x40000000;
uint style = Win32.GetWindowLong(myHandle, GWL_STYLE);
style |= WS_CHILD;
style &= ~WS_POPUP;
Win32.SetWindowLong(myHandle, GWL_STYLE, style);
}
According to MSDN this seems to be legal.
For compatibility reasons, SetParent does not modify the WS_CHILD or WS_POPUP window styles of the window whose parent is being changed. Therefore, if hWndNewParent is NULL, you should also clear the WS_CHILD bit and set the WS_POPUP style after calling SetParent. Conversely, if hWndNewParent is not NULL and the window was previously a child of the desktop, you should clear the WS_POPUP style and set the WS_CHILD style before calling SetParent.
We understand that this is a bit erm... 'unorthodox', but the design has been like this for a long time without any other issues.
Interestingly, removing the code that sets the WS_CHILD flag fixes the lost keys problem, but causes a bunch of other issues instead.
According to Raymond Chen
Creating a cross-thread parent/child or owner/owned window relationship implicitly attaches the input queues of the threads which those windows belong to...
We interpret this like AttachThreadInput is called 'under the hood', but do not fully understand what causes keypresses to be lost. Is it some sort of thread-safety issue? A race condition?
By using the AttachThreadInput function, a thread can share its input states (such as keyboard states and the current focus window) with another thread. Keyboard and mouse events received by both threads are processed in the order they were received until the threads are detached by calling AttachThreadInput a second time and specifying FALSE for the fAttach parameter.
We have used Spy++ to monitor the Windows messages that gets sent to the WPF process and it seems that when you type slowly, WM_KEYDOWN is sent together with WM_CHAR. When the input is lost, we still receive the WM_KEYDOWN messages, but no WM_CHAR. This led us to start investigating the .NET framework source. Among other things, we found the following code in HwndSource.cs
switch ((WindowMessage)msgdata.msg.message)
{
case WindowMessage.WM_SYSKEYDOWN:
case WindowMessage.WM_KEYDOWN:
// MITIGATION: HANDLED_KEYDOWN_STILL_GENERATES_CHARS
// In case a nested message pump is used before we return
// from processing this message, we disable processing the
// next WM_CHAR message because if the code pumps messages
// it should really mark the message as handled.
_eatCharMessages = true;
DispatcherOperation restoreCharMessages = Dispatcher.BeginInvoke(DispatcherPriority.Normal, new DispatcherOperationCallback(RestoreCharMessages), null);
// Force the Dispatcher to post a new message to service any
// pending operations, so that the operation we just posted
// is guaranteed to get dispatched after any pending WM_CHAR
// messages are dispatched.
Dispatcher.CriticalRequestProcessing(true);
The variable named _eatCharMessages could perhaps be related, although debugging this code using dnSpy has not made us any wiser.
We're looking for a Win32-wizard to shed some more light on this. Anything that can point us in the right direction would be greatly appreciated. We need to figure out the root cause of the problem in order to decide how to fix it.

SendKeys with games: with some characters it works, but with some it doesn't

I want to simulate input in games with SendKeys, but I have a hard time.
If I use it with i.e. the letter T, while the cursor in Minecraft is in a textbox (in the main menu), it works, the letter T is written in the textbox.
But with {ESC} it doesn't work. Nothing happens. If I press it manually, it backs to the previous menu. (as it should)
With some applications ESC works:
It works with Discord, Sourcetree, Slack, Chrome, CS2D,
but for some reason it doesn't work with Minecraft, Spelunky, Half-Life.
All of the applications mentioned above were in windowed mode.
Another issue:
If I send 2 to Minecraft while in a text field, it works correctly, 2 is written.
But if I send it while I'm playing, there is no effect. (The character should switch to Item Slot #2)
Same with " " (whitespace). In text fields it works, but the character won't jump in the game.
Code:
[DllImport("USER32.DLL", CharSet = CharSet.Unicode)]
public static extern IntPtr FindWindow(string lpClassName, string lpWindowName);
[DllImport("USER32.DLL")]
public static extern bool SetForegroundWindow(IntPtr hWnd);
public Form1()
{
InitializeComponent();
IntPtr minecraftHandle = FindWindow("GLFW30", "Minecraft* 1.15.2");
if (minecraftHandle == IntPtr.Zero)
{
MessageBox.Show("Minecraft is not running.");
return;
}
SetForegroundWindow(minecraftHandle);
SendKeys.SendWait("{ESC}");
}
I tried it without focus switching: by assigning the SendKey calls to a hotkey, so the target application can be in focus when the SendKeys are called.
The results are the same :\

Do not use SendKeys.Send to messaging between processes working on different runtimes
SendKeys.Send method is from System.Windows.Forms namespace.
This means it is not a Windows Input simulator, but just a little helper for Windows Forms applications. There is no guarantee this method work with another process on different (not .NET) runtime system.
Despite the fact that SendKeys.Send method uses native Windows API, it send key pressing message only of fixed period of time, so game frame handling may not have time to catch this message to manage it. So you may need for separate commands to send message about key down and key up events.
Do not use SendKeys API for messaging with another processes, especially with games.
Also, games can use protection system to rid of automatic bots that can blocks any messages from operation system programming input
So, what you can use?
First, you can try to use PostMessage of user32.dll system library:
const uint WM_KEYDOWN = 0x0100;
const uint WM_KEYUP = 0x0101;
[DllImport("user32.dll")]
static extern bool PostMessage(IntPtr hWnd, uint msg, int wParam, int lParam);
// hWnd - Window handler (can be getted by using GetForegroundWindow/FindWindow methods)
// msg - Key up/down message (WM_KEYUP / WM_KEYDOWN)
// wParam - Virual key code you need to pass to the window
// lParam - Additional parameter for set up key message behaviour.
All virtual key codes can be found on microsoft docs website:
https://learn.microsoft.com/en-us/windows/win32/inputdev/virtual-key-codes
Don't forget that you need to wait some time before key releasing. This is needed because games cache inputs between frames, and frame has fixed time to catch input. Just write some delay between key down and up messages.
Also you can set up key message behaviour by lParam. See WM_KEYDOWN and WM_KEYUP parameters. Special thing about WM_KEYDOWN message is if you pressing key on real keyboard long time, operation system repeating WM_KEYDOWN message accordingly. Repeating count can be setted up through lParam. Use it if window you messaging does not react on single keydown message.
PostMessage is low-level system command that can be used for messaging between processes. This command has a low probability to be blocked by protection system (but not zero) and high probability to be received by the game/process you working with. Also it provides opportunity to separate key up and key down messages.
What if PostMessage didn't work?
Try to use hardware scan code instead of virtual key code. Detailed explanation how you can do that described in this answer.
If protection system is really good and PostMessage is blocking even if you use hardware scan code, one thing you can try is to use another keyboard input driver or write it yourself. That driver must replace default system keyboard driver. And you can message it to interact with game. This is the 100% guarantee way to interact with other process through keyboard. But if you use public custom keyboard drivers, there is some probability that protection system blocks it. So you need to write your own driver to message between processes.

SendKeys to inactive application

I am trying to figure out how I can make my C# application to send keys to another application window, without focusing on it.
I have a list of words and 3 notepads files.
Imagine I have all 3 notepad windows opened in order, and my program would take the first word in the listbox and write it in the first Notepad window. The second word in the second Notepad window and third one in the third Notepad window. Then start over and continue.
So I want it to post 1 word in each and continue like that over and over.
But I only figured out how to do it with 1 notepad window (while having it active).
int i = 0;
i = listBox1.SelectedIndex;
i = i + 1;
if (i > listBox1.Items.Count - 1)
i = 0;
listBox1.SelectedIndex = i;
string message = listBox1.SelectedItem.ToString();
SendKeys.Send(message);
SendKeys.Send("{ENTER}");
This would require me to first focus on the notepad window, start the timer and then keep focus on the notepad window. It would then just loop through the list and type the word (1 on each line). And it works fine. But I want to be able to do it on 3 windows.
And then I need to get the window title somehow, not the process name.
I would have 3 processes called Notepad then.
But if they are named Note1, Note2, Note3 how would I do that?
I need help to make some kind of list of what programs to write to:
listofprograms = ["Note1", "Note2", "Note3"];
And it would find the application windows opened with those names,
then somehow write the text into those windows.
Could anyone help me out? Haven't found anything about this so far and trust me I've looked around!

Unfortunately there is no great way to do this. SendKeys is a really simple and desirable API but it only applies to the active window. There is no way to make it work on an inactive window nor is there an API with the same ease of access that works on inactive windows.
Generally when people run up against this problem they have to go one of two routes
Physically active the apps you want to send the keys to one at a time
Drop down to a much lower level of SendMessage or PostMessage to send keyboard events
The second option is generally more reliable but harder to implement.

SendKeys is not made for this type of functionality. To do what you're looking for, you're going to need to use some Win32 API calls in your code. See How to send text to Notepad in C#/Win32? for reference.

If you're looking for a way to send keys to an application, using SendKeys.Send(keys) is an option, but you need to bring the window to the top via the SetForegroundWindow API.
So, if you continue using your approach, you could use FindWindow, SetForegroundWindow to force the Notepad windows to be activated and focused, so that you could send the keys.
[DllImportAttribute("User32.dll")]
private static extern int FindWindow(String ClassName, String WindowName);
[DllImport("user32.dll")]
private static extern IntPtr SetForegroundWindow(int hWnd);
public int Activate(int hWnd)
{
if (hWnd > 0) {
SetForegroundWindow(hWnd);
return hWnd;
}
else {
return -1;
}
}
public int GetWindowHwnd(string className, string windowName) {
int hwnd = 0;
string cls = className == string.Empty ? null : className;
string win = windowName == string.Empty ? null : windowName;
hwnd = FindWindow(cls , win );
return hwnd;
}
Although there is also another solution, which could help you out. Here all Notepad processes are handled:
How to send text to Notepad in C#/Win32?
With some adaptions it should work for your case, too (basically you would iterate and loop through the notepad instances found and place a word in each instance).
You might also want to take a look at the following information about FindWindow:
http://www.pinvoke.net/default.aspx/user32.findwindow
SetKeyboardState:
http://www.pinvoke.net/default.aspx/user32/SetKeyboardState.html
As well as SendMessage:
http://www.pinvoke.net/default.aspx/coredll/SendMessage.html
You should find some useful hints in the examples and descriptions.

C# , detect selected text on windows?

I would make a tools like Google toolbar translate function, but it is for desktop.
What i want to do is
highlight the text in any application (word,pdf,live messenger etc) , and translate by google translate api ,return as a tool tips.
I have search msdn about monitoring text, i only found using copy&paste and monitoring clipboard to tick the event.
so, any idea about that?
thanks you.

A starting point would be to get a reference to the current foreground window. The code below will get the currently selected window and the title of that window:
[ DllImport("user32.dll") ]
static extern int GetForegroundWindow();
[ DllImport("user32.dll") ]
static extern int GetWindowText(int hWnd, StringBuilder text, int count);
private void GetActiveWindow()
{
const int nChars = 256;
int handle = 0;
StringBuilder Buff = new StringBuilder(nChars);
handle = GetForegroundWindow();
if ( GetWindowText(handle, Buff, nChars) > 0 )
{
this.captionWindowLabel.Text = Buff.ToString();
this.IDWindowLabel.Text = handle.ToString();
}
}
You could run this code within a timer: i.e give the user 10 seconds to select a window.
I am not sure how you would retrieve selected text within a window, but I will look into it for you.

I think you'll need to start by getting the handle of any window that is activated when your program is active. My guess is you need to look into InteropServices here to do this.
Using Windows API.

It sounds like you need to have your code intercept any window handle of any process, this is where it gets a bit complex as you have to ensure you do have access permissions to access another process.
Speaking of which, I do not think it is a good idea as you could end up crashing another process by poking around under the hood in regards to the winapi calls to trap the text selection event, not too mention the fact that you would have to determine if the process has any text selected. The best direction I can give is this...an article was written on how to spy on a process on CodeProject here, this can be a step in the right direction, bear in mind that the code used was for the .NET 1.0 framework.
Hope this helps and good luck in your coding,
Best regards,
Tom.

Audio feedback using RIL Audio on SmartPhone

We're working on a SIP softphone and we get audio feedback when we call from one phone to the other. However, when we call from a normal SIP Phone (software or hardware) to our app, then it all works fine - it's only when calling from one phone using the app to another one. Here is the code we use to initialize RIL Audio:
public static void InitRILAudio()
{
IntPtr res;
RILRESULTCALLBACK result = new RILRESULTCALLBACK(f_result);
RILNOTIFYCALLBACK notify = new RILNOTIFYCALLBACK(f_notify);
res = RIL_Initialize(1, result, notify, (0x00010000 | 0x00020000 | 0x00080000), 0, out hRil);
if (res != IntPtr.Zero)
return;
RILAUDIODEVICEINFO audioDeviceInfo = new RILAUDIODEVICEINFO();
audioDeviceInfo.cbSize = 16;
audioDeviceInfo.dwParams = 0x00000003; //RIL_PARAM_ADI_ALL;
audioDeviceInfo.dwRxDevice = 0x00000001; //RIL_AUDIO_HANDSET;
audioDeviceInfo.dwTxDevice = 0x00000001; //RIL_AUDIO_HANDSET;
res = RIL_SetAudioDevices(hRil, audioDeviceInfo);
}
We are using SipEk (http://voipengine.googlepages.com/sipeksdk) for the SIP stack. Basically we just use a callback delegate from the SDK for the audio stuff. Has anyone else experienced problems with Audio feedback loops like this? Either with RIL Audio or SipEk? Any suggestions?
Thanks in advance!

Feedback means that you're not using echo cancellation (line and/or acoustic, depending on whether it's working as a speakerphone or not), or if you are, the delay in your system (jitter buffers, network, encode/decode, etc) is greater than the echo canceller can handle. Excessive gain/clipping in the wrong places can also defeat any echo canceller (they don't like non-linear effects).
Sounds like you're just dumping the audio off to some other layers. SipEk is just a wrapper for pjsip, but I assume you're doing audio via the Microsoft RIL/etc stuff, not via pjmedia. You need to have a good understanding of your audio paths - where stuff gets sampled, if/how it's acoustic/line echo-cancelled, what the echo tail is, how it gets encoded and packetized, how it's received, jitter-buffered, loss-concealed, and decoded and played back.