I created a solution in visual studio 2008 on windows 7, 64 bit.
It works.
When I move it onto another machine, also win 7, 64 bit, it crashed with almost no information.
The original issue was this:
call was rejected by callee
I then implemented this solution:
how to properly GetTypeFromProgID for visual studio 2008
However, now my issue is that when I run the executable on a different machine, the program is crashing immediately with the following information:
Description:
Stopped working
Problem signature:
Problem Event Name: APPCRASH
Application Name: EmailSalesVolumeSolution.exe
Application Version: 1.0.0.0
Application Timestamp: 508064dd
Fault Module Name: KERNELBASE.dll
Fault Module Version: 6.1.7601.17932
Fault Module Timestamp: 503285c2
Exception Code: e0434f4d
Exception Offset: 000000000000caed
OS Version: 6.1.7601.2.1.0.256.48
Locale ID: 1033
Read our privacy statement online:
http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0409
If the online privacy statement is not available, please read our privacy statement offline:
C:\Windows\system32\en-US\erofflps.txt
I wrapped the code in a try/catch, and still did not get a proper error message:
static void Main()
{
try
{
EnvDTE80.DTE2 dte;
object obj = null;
System.Type t = null;
// Get the ProgID for DTE 8.0.
t = System.Type.GetTypeFromProgID("VisualStudio.DTE.9.0",
true);
// Create a new instance of the IDE.
obj = System.Activator.CreateInstance(t, true);
// Cast the instance to DTE2 and assign to variable dte.
dte = (EnvDTE80.DTE2)obj;
// Register the IOleMessageFilter to handle any threading
// errors.
MessageFilter.Register();
// Display the Visual Studio IDE.
dte.MainWindow.Activate();
Application.EnableVisualStyles();
Application.SetCompatibleTextRenderingDefault(false);
Application.Run(new Form1());
// For example, get a reference to the solution2 object
// and do what you like with it.
// All done, so shut down the IDE...
dte.Quit();
// and turn off the IOleMessageFilter.
MessageFilter.Revoke();
}
catch (Exception e)
{
MessageBox.Show(e.ToString());
}
}
How can I identify where exactly the exception occurred and WHAT IS the exception?
I do have some unmanaged code:
using System;
using System.Collections.Generic;
using System.Text;
using EnvDTE;
using EnvDTE80;
using EnvDTE90;
using System.Runtime.InteropServices;
namespace EmailSalesVolumeSolution
{
public class MessageFilter : IOleMessageFilter
{
//
// Class containing the IOleMessageFilter
// thread error-handling functions.
// Start the filter.
public static void Register()
{
IOleMessageFilter newFilter = new MessageFilter();
IOleMessageFilter oldFilter = null;
CoRegisterMessageFilter(newFilter, out oldFilter);
}
// Done with the filter, close it.
public static void Revoke()
{
IOleMessageFilter oldFilter = null;
CoRegisterMessageFilter(null, out oldFilter);
}
//
// IOleMessageFilter functions.
// Handle incoming thread requests.
int IOleMessageFilter.HandleInComingCall(int dwCallType,
System.IntPtr hTaskCaller, int dwTickCount, System.IntPtr
lpInterfaceInfo)
{
//Return the flag SERVERCALL_ISHANDLED.
return 0;
}
// Thread call was rejected, so try again.
int IOleMessageFilter.RetryRejectedCall(System.IntPtr
hTaskCallee, int dwTickCount, int dwRejectType)
{
if (dwRejectType == 2)
// flag = SERVERCALL_RETRYLATER.
{
// Retry the thread call immediately if return >=0 &
// <100.
return 99;
}
// Too busy; cancel call.
return -1;
}
int IOleMessageFilter.MessagePending(System.IntPtr hTaskCallee,
int dwTickCount, int dwPendingType)
{
//Return the flag PENDINGMSG_WAITDEFPROCESS.
return 2;
}
// Implement the IOleMessageFilter interface.
[DllImport("Ole32.dll")]
private static extern int
CoRegisterMessageFilter(IOleMessageFilter newFilter, out
IOleMessageFilter oldFilter);
}
[ComImport(), Guid("00000016-0000-0000-C000-000000000046"),
InterfaceTypeAttribute(ComInterfaceType.InterfaceIsIUnknown)]
interface IOleMessageFilter
{
[PreserveSig]
int HandleInComingCall(
int dwCallType,
IntPtr hTaskCaller,
int dwTickCount,
IntPtr lpInterfaceInfo);
[PreserveSig]
int RetryRejectedCall(
IntPtr hTaskCallee,
int dwTickCount,
int dwRejectType);
[PreserveSig]
int MessagePending(
IntPtr hTaskCallee,
int dwTickCount,
int dwPendingType);
}
}
after installing VS 2008 express on the machine that gave me trouble, i am now getting this:
You can try also to subscribe on AppDomain.CurrentDomain.UnhandledException event:
AppDomain.CurrentDomain.UnhandledException +=
(sender, e) => MessageBox.Show(e.ExceptionObject.ToString());
Do this at the top of Main method.
Try printing more from the Exception:
catch(Exception e)
{
MessageBox.Show(e.Message + ":\n" + e.StackTrace);
}
You can use this to catch all Inner Excpetions as well:
string except = "Uncaught Exception: ";
while(e != null)
{
except += e.Message + ";\n";
e = e.InnerException;
}
If program crashes with no exception then you need to try and attach a Debugger,
You'll need Visual Studio installed on the machine or to be able to use Remote Desktop for remote debugging. How to: Attach to a Running Process
To attach to a running process:
On the Debug menu, select Attach to Process.
In the Attach to Process dialog box, find the program that you want
to attach to from the Available Processes list.
If the program that you want to debug is running on another
computer, use the Qualifier list box to select the remote computer.
For more information, see How to: Select a Remote Machine.
If the process is running under a different user account, select the
Show processes from all users check box.
If you are connected through Remote Desktop Connection, select the
Show processes in all sessions check box.
In the Attach to box, make sure that the type of code you will debug
is listed. The default Automatic setting tries to determine what
type of code you want to debug. If the automatic setting is not
appropriate:
Click Select.
In the Select Code Type dialog box, click Debug these code types and
select the types to debug.
Click OK.
Click Attach.
You MUST compile the executable with DEBUG mode
UPDATE
I see you use Type.GetTypeFromProgID that requires .NET 4.5, Please make sure you have .NET 4.5 installed on the PC that is throwing the error !
It seems that t = System.Type.GetTypeFromProgID("VisualStudio.DTE.9.0", true); is throwing COMException that means that "VisualStudio.DTE.9.0" is not registered in the other PC.
You should add Microsoft Visual Studio 2008 DTE as a dependency for your program.
Your DLL may not be registered on the client machine. Check the DLL that you are calling the COM object for and do a Regsvr32 YouComDllName.dll in a administrative command prompt (right click on the command prompt icon and do Run as Administrator. Depending on if you are using a 32 bit or 64 bit application you may need to either use the copy of Regsvr32 in C:\Windows\System32 or in C:\Windows\SysWOW64 (In a backwards way the 64 bit version is in system32 and the 32 bit version is in SysWOW64)
If just regestering the DLL on your machine does not work run Dependancy Walker on the program on your computer. When you get all of the DLLs that your program touches, check the client computer is not missing any. The DLL you are calling may be there, but a DLL it depends on may not be.
The only solution that worked was installing Visual studio. As soon as it was installed, I got the exact exception.
Related
I'm currently coding a windows service (installed as LocalSystem) that monitors several things on a pc/server including processes. For the processes, I'm watching the memory usage and also "try" to get the number of GDI Objects per process (as can be seen in task manager).
Sadly, C# Process objects don't have the gdi count built-in so I'm using the GetGuiResources method from 'user32.dll' as shown in this example:
https://www.pinvoke.net/default.aspx/user32.getguiresources.
Basically I have a list of executable names, for each of them I use GetProcessesByName to retrieve all process instances, and then for each unique process I take the handle and send it to the function to get the Gdi objects count back.
When I try this on my local machine as a simple console app (feeding a name through Console.ReadLine), it works no problem as long as the console app is launched as administrator; i get the same numbers as task manager.
However, when the monitoring service calls this function, I get either 0s (returning error code 87) or worse: processes tied to services (no gui) return me some random numbers (12, 7, 4, etc.) when the task manager actually shows 0 (and last error = 0).
So in summary, every process that shows some GID objects in Task Manager returns 0 (error 87), and each process who has 0 returns me a number (no error, or error 183 for the monitoring service itself).
I've tried this on Windows 10, Windows Server 2012, Windows Server 2008, Windows Server 2003, Windows Server 2016. On windows 10 (my machine) I get 0s everywhere, on other OS I get the mentionned results.
Here's a shortened version of the code I use:
// Monitoring processes exeName example: ssms, sqlbrowser
List<Process> result = Process.GetProcessesByName(exeName).ToList();
if (processes != null)
{
for (int i = 0; i < processes.Count; i++)
{
int gdiCount = processes[i].GetGDIObjectsCount(); // extension method
// logging and doing stuff with gdi count here (but i get 0s or random numbers as I told)
}
}
// Process extension method
public static class CProcessExtensions
{
[DllImport("User32", SetLastError = true)]
extern private static int GetGuiResources(IntPtr hProcess, int uiFlags);
private static int GetGDICount(IntPtr processHandle)
{
if (processHandle == IntPtr.Zero)
{
return -1;
}
int count = GetGuiResources(processHandle, 0);
// Logging Marshal.GetLastWin32Error() here
return count;
}
public static int GetGDIObjectsCount(this Process process)
{
IntPtr handle;
process.Refresh();
try
{
handle = process.Handle;
}
catch (Exception ex)
{
handle = IntPtr.Zero;
}
return GetGDICount(handle);
}
}
I've also tried getting the process handles with the OpenProcess dll method but had the same results.
Anyone faced this kind of problem before?
So, thanks to Jeremy Thompson's comment leading me to info about the session 0, and with further research, I was able to solve my problem.
References:
Application Loader to launch process in another session
Wait for process exit (ProcessWaitHandle)
Get Exit code
What I did is modify the sample code from the first reference to provide a process ID (the one I want the GDI objects count of) and launch my little console app (which takes the same process ID also, and returns the GDI count as exit code) in the same session by duplicating the token of the provided process and call CreateProcessAsUser.
By launching the console app in the same session I was able to retrieve the correct info on GDI objects on every OS I previously tested except Win Server 2003, which I can totally live without.
Why does the following code sometimes causes an Exception with the contents "CLIPBRD_E_CANT_OPEN":
Clipboard.SetText(str);
This usually occurs the first time the Clipboard is used in the application and not after that.
This is caused by a bug/feature in Terminal Services clipboard (and possible other things) and the .NET implementation of the clipboard. A delay in opening the clipboard causes the error, which usually passes within a few milliseconds.
The solution is to try multiple times within a loop and sleep in between.
for (int i = 0; i < 10; i++)
{
try
{
Clipboard.SetText(str);
return;
}
catch { }
System.Threading.Thread.Sleep(10);
}
Actually, I think this is the fault of the Win32 API.
To set data in the clipboard, you have to open it first. Only one process can have the clipboard open at a time. So, when you check, if another process has the clipboard open for any reason, your attempt to open it will fail.
It just so happens that Terminal Services keeps track of the clipboard, and on older versions of Windows (pre-Vista), you have to open the clipboard to see what's inside... which ends up blocking you. The only solution is to wait until Terminal Services closes the clipboard and try again.
It's important to realize that this is not specific to Terminal Services, though: it can happen with anything. Working with the clipboard in Win32 is a giant race condition. But, since by design you're only supposed to muck around with the clipboard in response to user input, this usually doesn't present a problem.
I know this question is old, but the problem still exists. As mentioned before, this exception occurs when the system clipboard is blocked by another process. Unfortunately, there are many snipping tools, programs for screenshots and file copy tools which can block the Windows clipboard. So you will get the exception every time you try to use Clipboard.SetText(str) when such a tool is installed on your PC.
Solution:
never use
Clipboard.SetText(str);
use instead
Clipboard.SetDataObject(str);
I solved this issue for my own app using native Win32 functions: OpenClipboard(), CloseClipboard() and SetClipboardData().
Below the wrapper class I made. Could anyone please review it and tell if it is correct or not. Especially when the managed code is running as x64 app (I use Any CPU in the project options). What happens when I link to x86 libraries from x64 app?
Thank you!
Here's the code:
public static class ClipboardNative
{
[DllImport("user32.dll")]
private static extern bool OpenClipboard(IntPtr hWndNewOwner);
[DllImport("user32.dll")]
private static extern bool CloseClipboard();
[DllImport("user32.dll")]
private static extern bool SetClipboardData(uint uFormat, IntPtr data);
private const uint CF_UNICODETEXT = 13;
public static bool CopyTextToClipboard(string text)
{
if (!OpenClipboard(IntPtr.Zero)){
return false;
}
var global = Marshal.StringToHGlobalUni(text);
SetClipboardData(CF_UNICODETEXT, global);
CloseClipboard();
//-------------------------------------------
// Not sure, but it looks like we do not need
// to free HGLOBAL because Clipboard is now
// responsible for the copied data. (?)
//
// Otherwise the second call will crash
// the app with a Win32 exception
// inside OpenClipboard() function
//-------------------------------------------
// Marshal.FreeHGlobal(global);
return true;
}
}
Actually there could be another issue at hand. The framework call (both the WPF and winform flavors) to something like this (code is from reflector):
private static void SetDataInternal(string format, object data)
{
bool flag;
if (IsDataFormatAutoConvert(format))
{
flag = true;
}
else
{
flag = false;
}
IDataObject obj2 = new DataObject();
obj2.SetData(format, data, flag);
SetDataObject(obj2, true);
}
Note that SetDataObject is always called with true in this case.
Internally that triggers two calls to the win32 api, one to set the data and one to flush it from your app so it's available after the app closes.
I've seen several apps (some chrome plugin, and a download manager) that listen to the clipboard event. As soon as the first call hits, the app will open the clipboard to look into the data, and the second call to flush will fail.
Haven't found a good solution except to write my own clipboard class that uses direct win32 API or to call setDataObject directly with false for keeping data after the app closes.
Use the WinForms version (yes, there is no harm using WinForms in WPF applications), it handles everything you need:
System.Windows.Forms.Clipboard.SetDataObject(yourText, true, 10, 100);
This will attempt to copy yourText to the clipboard, it remains after your app exists, will attempt up to 10 times, and will wait 100ms between each attempt.
Ref. https://learn.microsoft.com/en-us/dotnet/api/system.windows.forms.clipboard.setdataobject?view=netframework-4.7.2#System_Windows_Forms_Clipboard_SetDataObject_System_Object_System_Boolean_System_Int32_System_Int32_
This happen to me in my WPF application. I got OpenClipboard Failed (Exception from HRESULT: 0x800401D0 (CLIPBRD_E_CANT_OPEN)).
i use
ApplicationCommands.Copy.Execute(null, myDataGrid);
solution is to clear the clipboard first
Clipboard.Clear();
ApplicationCommands.Copy.Execute(null, myDataGrid);
The difference between Cliboard.SetText and Cliboard.SetDataObject in WPF is that the text is not copied to the clipboard, only the pointer. I checked the source code. If we call SetDataObject(data, true) Clipoard.Flush() will also be called. Thanks to this, text or data is available even after closing the application. I think Windows applications only call Flush() when they are shutting down. Thanks to this, it saves memory and at the same time gives access to data without an active application.
Copy to clipboard:
IDataObject CopyStringToClipboard(string s)
{
var dataObject = new DataObject(s);
Clipboard.SetDataObject(dataObject, false);
return dataObject;
}
Code when app or window is closed:
try
{
if ((clipboardData != null) && Clipboard.IsCurrent(clipboardData))
Clipboard.Flush();
}
catch (COMException ex) {}
clipboardData is a window class field or static variable.
That's not a solution, just some additional information on how to reproduce it when all solutions work on your PC and fail somewhere else. As mentioned in the accepted answer - clipboard can be busy by some other app. You just need to handle this failure properly, to explain user somehow why it does not work.
So, just create a new console app with few lines below and run it. And while it is running - test your primary app on how it is handles busy clipboard:
using System;
using System.Runtime.InteropServices;
namespace Clipboard
{
class Program
{
[DllImport("user32.dll")]
private static extern bool OpenClipboard(IntPtr hWndNewOwner);
[DllImport("user32.dll")]
private static extern bool CloseClipboard();
static void Main(string[] args)
{
bool res = OpenClipboard(IntPtr.Zero);
Console.Write(res);
Console.Read();
CloseClipboard();
}
}
}
Even though there are command-line commands to start (most of?) the various Control Panel screens in Windows 10, a specific scenario seems to fail:
If the machine starts with Bluetooth turned off (not disabled), running the command which should open the Bluetooth settings screen, simply does nothing. The command could be either ms-settings:bluetooth, bthprops.cpl or ms-settings:Bluetooth.
I've also tried to directly launch the Bluetooth Devices screen (using the command %windir%\explorer.exe shell:::{28803F59-3A75-4058-995F-4EE5503B023C} as described here), but clicking on the "Bluetooth settings" in this window does nothing as well.
The only way to get directly to the Bluetooth settings screen without going through the main Control Panel window and without turning on Bluetooth first, is by right clicking on the relevant tile in Windows Action Center:
Although this seems like a bug on the operating system level, I was wondering if there's any way to know when the launch fails from within C# code. So I've tried using the following code:
try
{
var process = new Process();
process.StartInfo.FileName = "control";
process.StartInfo.Arguments = "bthprops.cpl";
process.Exited += (s, e) =>
{
if (process.ExitCode != 0)
{
TurnOnBt();
}
};
var res = process.Start();
if (!res)
{
TurnOnBt();
}
}
catch (System.Exception ex)
{
int test = 6; // just for breakpoint
}
Problem is, no exception was ever thrown, and most of the time the Process.Exit event was never called.
Further more, calling Windows.Devices.Radios.Radio.GetRadiosAsync() returns an empty list!
Currently the only solution I've found is to manually turn on Bluetooth - it wouldn't change the Process.Start/Exit behavior, but it does allow to successfully lunch the command to directly open Bluetooth Settings window, and to get the list of the machine's Bluetooth/Radio devices. Still, when turning off Bluetooth and restarting the machine, same problem would happen all over again.
Any ideas for a code-based workaround?
note - all this based only on my debugging research, nothing from this is documented
i look how BT-settings window is open via Action Center (win8.1, win 10):
the IApplicationActivationManager interface created and called ActivateApplication method with:
appUserModelId = L"windows.immersivecontrolpanel_cw5n1h2txyewy!microsoft.windows.immersivecontrolpanel"
arguments = L"page=SettingsPagePCSystemBluetooth"
code on c++ can be look like:
if (0 <= CoInitialize(0))
{
IApplicationActivationManager* pAppAct;
if (0 <= CoCreateInstance(__uuidof(ApplicationActivationManager), 0, CLSCTX_ALL, IID_PPV_ARGS(&pAppAct)))
{
ULONG dwProcessId;
pAppAct->ActivateApplication(
L"windows.immersivecontrolpanel_cw5n1h2txyewy!microsoft.windows.immersivecontrolpanel",
L"page=SettingsPagePCSystemBluetooth",
AO_NONE ,
&dwProcessId);
pAppAct->Release();
}
CoUninitialize();
}
the processId (if all ok) reference to "X:\Windows\ImmersiveControlPanel\SystemSettings.exe" -ServerName:microsoft.windows.immersivecontrolpanel
for c# - look IApplicationActivationManager::ActivateApplication in C#?
the "windows.immersivecontrolpanel_cw5n1h2txyewy!microsoft.windows.immersivecontrolpanel" and "page=SettingsPagePCSystemBluetooth" strings not documented anywhere and I not sure are it is "persist" - but currently it used to open manage Bluetooth page in system settings. as is.
in case we run control.exe bthprops.cpl - the process ( control.exe ) launched without any error - as result you and not got any errors when you call this code.
then control.exe bthprops.cpl exec new process rundll32.exe Shell32.dll,Control_RunDLL bthprops.cpl, and exit
the rundll32.dll call Control_RunDLLW(HWND_DESKTOP, (HINSTANCE)&__ImageBase, L"bthprops.cpl", SW_SHOWDEFAULT);
we can and direct call void WINAPI Control_RunDLLW(HWND hwndParent, HINSTANCE hInst, PCWSTR cplName, int nCmdShow ); this api exported from shell32.dll
internally Control_RunDLLW load "bthprops.cpl" (3-rd argument - cplName), find CPlApplet entry point, and send CPL_INIT message. the bthprops.cpl on this message check are bthserv is running via OpenService(L"BTHSERV", ) + QueryServiceStatus (in function BthpIsbluetoothServiceRunning) and if "BTHSERV" not running - it return zero (fail code)
I have a 32 bit dll that I need to be called from a 64 bit .NET Core Application.
My method to do this is to create a 32bit console App that basically looks like this:
[DllImport("x.dll", CharSet = CharSet.Ansi, CallingConvention = CallingConvention.StdCall)]
static extern int x(int var);
static void Main(string[] args)
{
Console.log("I Started!");
int y = x(0);
//do something with y.
}
And then to call that from my 64 bit Core Application using Process.Start("consoleapp.exe")
When I run the consoleapp.exe file, it prints "I Started!" as expected, and the operation of x.dll also executes correctly.
However, when I call the consoleapp.exe from process.Start() I get an SEHException thrown after "I Started!" is printed to the output (i.e. when the DllImport part is hit).
Here is how I'm starting the process within my 64bit App
ProcessStartInfo p = new ProcessStartInfo("consoleapp.exe");
Process process = Process.Start(p);
process.WaitForExit();
I have already tried user impersonation and running the Process object as an administrator, and it isn't working.
The dll file, x.dll is definitely in the working directory of consoleapp.exe, as I have made the process do a console.writeline() on the working directory, and it matches where the x.dll file is located.
The SEHException error code is 0x80004005, which is E_FAIL "Unspecified Failure"
UPDATE: As requested, stack trace is:
at Consoleapp.Consoleapp.x(int32 var)
at Consoleapp.Consoleapp.Main(String[] args)
Not sure how that helps you. If you mean the stack trace of the External Exception, I can't work out how to access that (InnerException is null) - some kind of guide may be good to help me do that.
The message attached to exception is "External component has thrown an exception."
The x.dll file reads a file in a subdirectory under CSIDL_COMMON_APPDATA. This variable is correct when I run program directly, and from web app - used console.log on Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData))
The file contents are read to get database location and credentials to read from a particular database. If any part of that fails, the x.dll returns an error code, not an exception, and I have double checked this is the case by changing credentials in file, database name, record numbers, etc. and no SEHExceptions were thrown when run directly.
Make it
public static extern x(int var);
and/or try to specify it as void
public static extern void x(int var);
I've been working on an application that requires monitoring thread specific mouse activity (WH_MOUSE) on another process and encountered something very curious.
After finding out that this is not possible via exclusively managed code if I don't want to use WH_MOUSE_LL and that I'd need a native DLL export to inject itself in the target process, I set out and created it in C++ according to what scattered documentation I could find on the subject and then tried using it to hook into Notepad.
Although according to GetLastWin32Error the injection succeeded, I was not getting notified of mouse events. After nearly giving up and going for the low level global hook option, I re-read the "Remarks" section of this article which made me suspect that the problem may be because of the "bitness" of my code vs notepad:
A 32-bit DLL cannot be injected into a 64-bit process, and a 64-bit
DLL cannot be injected into a 32-bit process. If an application
requires the use of hooks in other processes, it is required that a
32-bit application call SetWindowsHookEx to inject a 32-bit DLL into
32-bit processes, and a 64-bit application call SetWindowsHookEx to
inject a 64-bit DLL into 64-bit processes.
However, both my native DLL and managed application were compiled as x64, and I was trying to hook into the 64-bit version of notepad, so it should've worked fine, but I took a shot in the dark anyway and went into the SysWOW64 folder and opened the 32-bit Notepad from there, tried hooking in again and this time the hook worked beautifully!
Curiously, I then recompiled both my native DLL and managed app as x86 and tested it it against the 32-bit Notepad and it didn't work, but it worked on my normal 64-bit Notepad!
How am I possibly seem to be able to inject a 32-bit DLL into a 64-bit process and vice versa!
Although my original problem has been solved and I can continue with my app's development, the curiosity as to why I'm observing this strange inverse behavior from SetWindowsHookEx is driving me insane, so I really hope someone will be able to shed some light on this.
I know this a lot of talk and no code, but the code for even a sample app is rather large and comes in both managed and unmanaged flavors, however I'll promptly post any piece of the code you think might be relevant.
I've also created a sample app so you can test this behavior yourself. It's a simple WinForms app that tries to hook into Notepad and displays its mouse events:
http://saebamini.com/HookTest.zip
It contains both an x86 version and an x64 version. On my machine (I'm on a 64-bit Windows 7), the x86 version only works with 64-bit Notepad, and the x64 version only works with 32-bit Notepad (from SysWOW64).
UPDATE - Relevant Bits of Code:
C# call to the unmanaged library:
public SetCallback(HookTypes type)
{
_type = type;
_processHandler = new HookProcessedHandler(InternalHookCallback);
SetCallBackResults result = SetUserHookCallback(_processHandler, _type);
if (result != SetCallBackResults.Success)
{
this.Dispose();
GenerateCallBackException(type, result);
}
}
public void InstallHook()
{
Process[] bsProcesses = Process.GetProcessesByName("notepad");
if(bsProcesses.Length == 0)
{
throw new ArgumentException("No open Notepad instance found.");
}
ProcessThread tmp = GetUIThread(bsProcesses[0]);
if (!InitializeHook(_type, tmp.Id))
{
throw new ManagedHooksException("Hook initialization failed.");
}
_isHooked = true;
}
[System.Runtime.InteropServices.DllImport("user32.dll")]
private static extern int GetWindowThreadProcessId(IntPtr hWnd, IntPtr procid);
// 64-bit version
[DllImport("SystemHookCore64.dll", EntryPoint = "InitializeHook", SetLastError = true,
CharSet = CharSet.Unicode, ExactSpelling = true,
CallingConvention = CallingConvention.Cdecl)]
private static extern bool InitializeHook(HookTypes hookType, int threadID);
[DllImport("SystemHookCore64.dll", EntryPoint = "SetUserHookCallback", SetLastError = true,
CharSet = CharSet.Unicode, ExactSpelling = true,
CallingConvention = CallingConvention.Cdecl)]
private static extern SetCallBackResults SetUserHookCallback(HookProcessedHandler hookCallback, HookTypes hookType);
C++:
HookProc UserMouseHookCallback = NULL;
HHOOK hookMouse = NULL;
HINSTANCE g_appInstance = NULL;
MessageFilter mouseFilter;
bool InitializeHook(UINT hookID, int threadID)
{
if (g_appInstance == NULL)
{
return false;
}
if (hookID == WH_MOUSE)
{
if (UserMouseHookCallback == NULL)
{
return false;
}
hookMouse = SetWindowsHookEx(hookID, (HOOKPROC)InternalMouseHookCallback, g_appInstance, threadID);
return hookMouse != NULL;
}
}
int SetUserHookCallback(HookProc userProc, UINT hookID)
{
if (userProc == NULL)
{
return HookCoreErrors::SetCallBack::ARGUMENT_ERROR;
}
if (hookID == WH_MOUSE)
{
if (UserMouseHookCallback != NULL)
{
return HookCoreErrors::SetCallBack::ALREADY_SET;
}
UserMouseHookCallback = userProc;
mouseFilter.Clear();
return HookCoreErrors::SetCallBack::SUCCESS;
}
return HookCoreErrors::SetCallBack::NOT_IMPLEMENTED;
}
int FilterMessage(UINT hookID, int message)
{
if (hookID == WH_MOUSE)
{
if(mouseFilter.AddMessage(message))
{
return HookCoreErrors::FilterMessage::SUCCESS;
}
else
{
return HookCoreErrors::FilterMessage::FAILED;
}
}
return HookCoreErrors::FilterMessage::NOT_IMPLEMENTED;
}
static LRESULT CALLBACK InternalMouseHookCallback(int code, WPARAM wparam, LPARAM lparam)
{
if (code < 0)
{
return CallNextHookEx(hookMouse, code, wparam, lparam);
}
if (UserMouseHookCallback != NULL && !mouseFilter.IsFiltered((int)wparam))
{
UserMouseHookCallback(code, wparam, lparam);
}
return CallNextHookEx(hookMouse, code, wparam, lparam);
}
My best guess about your problem :
The Windows hook system is able to hook both 32-bit and 64-bit application, from any bitness. The thing is, as you pointed, you can't inject a DLL into an application with the wrong bitness. To make this work, Windows will normally inject the DLL if it can, but if it can't, it will setup a callback that use the hooking application message loop. Since the message loop is handled by the OS, it is used to make a call from different bitness.
In your case, the only thing that work is the message loop way. And there is a good reason for that : your 64-to-64 and 32-to-32 calls have no chance to succeed, because the hook is in the injected DLL, that is, in a different process than your application.
Nothing happens in your case because your UserMouseHookCallback stay to NULL. Indeed, the call to SetUserHookCallback() is done in the application DLL instance, but UserMouseHookCallback is unchanged in the target DLL instance. Once injected, the DLL is in a different process, and should be considered as such. You have to find another way to call back the application (maybe post a message, like in the 32-to-64 case, and/or make use of shared sections).
To test this, put something like MessageBox() in InternalMouseHookCallback(). The box should appear even in 64-to-64 and 32-to-32.