Develop Reference

SpatialCoordinateSystem.TryGetTransformTo() from Webcam to Unity space fails in a non-initial, separated spatial environment on HoloLens 2

I have a quite specific problem regarding to a transformation matrix for transformations from the HoloLens 2 webcam space into the current Unity scene space in a Unity+MRTK+OpenXR app. The goal is to acquire the exact camera pose related to a camera frame, which was acquired through Windows.Media.Capture, in Unity space.
My environment:
Unity 2021.3.8.
MRTK v2.8.2
Mixed Reality OpenXR Plug-In v1.6.0
For obtaining the matrix, I first receive a Windows.Perception.Spatial.SpatialCoordinateSystem instance (unityReferenceCoordinateSystem) representing the Unity Space through the MR OpenXR Plug-In as described HERE:
using Windows.Perception.Spatial;
using Microsoft.MixedReality.OpenXR;
SpatialCoordinateSystem unityReferenceCoordinateSystem = PerceptionInterop.GetSceneCoordinateSystem(Pose.identity) as SpatialCoordinateSystem;
and I obtain the camera space (cameraCoordinateSystem) from the Windows.Media.Capture.Frames.MediaFrameReference camera frame instance acquired from a MediaFrameReader by
MediaFrameReference mediaFrame; // acquired camera frame
SpatialCoordinateSystem cameraCoordinateSystem = mediaFrame.CoordinateSystem;
Finally I obtain the required transformation matrix by using SpatialCoordinateSystem.TryGetTransformTo() as you can see in my complete method:
using Microsoft.MixedReality.Toolkit;
public bool TryGetCameraToUnityMatrix(out Matrix4x4 cameraToUnity)
{
// (obtain MediaFrameReader, acquire a camera frame and obtain
// unityReferenceCoordinateSystem and cameraCoordinateSystem as described above)
System.Numerics.Matrix4x4? camToUnitySysMatrix = cameraCoordinateSystem.TryGetTransformTo(unityReferenceCoordinateSystem);
if (!camToUnitySysMatrix.HasValue)
{
return false;
}
cameraToUnity = camToUnitySysMatrix.Value.ToUnity();
return true;
}
This works all fine so far - until I bring the HoloLens into another spatial environment, which is not connected to the environment, which was present when the app was started.
Describing the following scenario should make clear what I mean by that:
Start the app on HL2
Acquire the cameraToUnity matrix as described --> works fine
Set the HL to stand-by
Go to another room, for which the HL's spatial awareness does not know the connection between these two rooms
Wake up HL and open the (still running) app.
Acquire the cameraToUnity matrix. --> FAILS:
camToUnitySysMatrix.HasValue returns false (even though both arguments unityReferenceCoordinateSystem and cameraCoordinateSystem are not null.)
Set the HL to stand-by again
Go back to the initial environment where the app was originaly started
Wake up HL and open the (still running) app.
Acquire the cameraToUnity matrix as described --> works fine again! (camToUnitySysMatrix has valid value again)
I also made sure that unityReferenceCoordinateSystem = PerceptionInterop.GetSceneCoordinateSystem(Pose.identity) is re-called after I changed the environment and also the MediaFrameReader gets freshly instantiated origining from a new MediaCapture instance.
But obviously a transformation between the two SpatialCoordinateSystems seems to fail if it is attempted in the non-initial spatial environment.
Any ideas on how to solve this?
UPDATE
A minimal Unity sample project for reproducing this problem can be found here:
https://github.com/pjaydev/trygettransformto-so

Usually, users should wear the HoloLens device and keep it turned on while moving and operating, so that the device can fully understand the environment. And HoloLens also has some requirements on the size of the site used, if the room is too small, HoloLens may not work properly. If you have confusion about this issue or there is business impact, you can submit a ticket via aka.ms/HLSupport.

Take screenshot of external OpenGL game C# with BitBlt (CopyFromScreen)

I am trying to create some kind of antycheat for counter strike (hl) game. Of course funcionality of making a screenshot in-game is built-in, but exploited (detected) by antyss applications, so every time screenshot is taken from the game, antyss is disabling the cheats (so that no cheats are visible on the screenshots)
For the last few days, I've read dozens of threads regarding this topic. Most of them are outdated and are using libraries, that are obsolete right now.
I've read about the approach with mirage driver (which is not working on windows 10), about injecting to the application (of course application/game is not part of my code) and using/incjeting some code with OPEN GL/D3D library (to read backbuffer). Probably this could be in the end the only solution.
But right now I have almost a working solution. I write "almost" because it is working but giving me only some kind of "cached" data. It is giving me a correct screenshot, but if I take another screenshot - still the same screenshot is taken as last time. If while being in-game I minimize the application (full-screen mode) and then get back to the game, the new screenshot taken will have up to date screenshot, but then again, the next screenshot would be exactly the same.
I don't know if it is "by design" or is it "some sort of bug" Nevertheless my question is: Can I force somehow this "reloading" without having to programmatically call some kind of "alt+tab" and then focusing on the application once again?
In this topic:
How to take screenshots of a game with OpenGL
#Andon M. Coleman wrote:
Are you on Windows? In fullscreen mode starting with Windows Vista, there is trouble with anything that tries to capture the front-buffer (including the built-in Alt + PrintScreen). The easiest solution is to change your buffer swap behavior to Copy Swap (PFD_SWAP_COPY, slower but guaranteed to work). Often if you Alt+TAB out and back in after making the fullscreen mode switch that will fix it too; though I have never been able to explain that ;) If you did not write the game in the question, then the second solution may be your only choice if you want to use that code.
This is exactly the problem I am facing. As he wrote: "Alt+Tab" is fixing the problem (although he did not know whether it is a feature or a bug) He proposed to change the buffer swap behavior to Copy Swap(PFD_SWAP_COPY) Any tips on how to change my code with that will also be most welcome (I can try this one) But if I understood correctly, this approach is the viable solution only if you can change this in the game (and this is not my case)
Here is my working code (which in topics about such scenarios was claiming that in this approach the screenshot is BLACK. But it is working for me)
private const int SW_RESTORE = 9;
public void TakeScreenShot()
{
var guid = Guid.NewGuid();
string procName = "hl";
Process proc;
try
{
proc = Process.GetProcessesByName(procName)[0];
}
catch (IndexOutOfRangeException e)
{
return;
}
// Focus on the application
SetForegroundWindow(proc.MainWindowHandle);
ShowWindow(proc.MainWindowHandle, SW_RESTORE);
Thread.Sleep(1000);
Rect rect = new Rect();
IntPtr error = GetWindowRect(proc.MainWindowHandle, ref rect);
while (error == (IntPtr)0)
{
error = GetWindowRect(proc.MainWindowHandle, ref rect);
}
int width = rect.right - rect.left;
int height = rect.bottom - rect.top;
using (Bitmap printscreen = new Bitmap(width, height, PixelFormat.Format32bppArgb))
{
using (var graphics = Graphics.FromImage(printscreen))
{
graphics.CopyFromScreen(rect.left,
rect.top,
0,
0,
new Size(width, height),
CopyPixelOperation.SourceCopy);
printscreen.Save($#"{Path.GetTempPath()}\{guid.ToString()}.jpg", ImageFormat.Jpeg);
}
}
}
I want this application to work on Windows7, Windows8, Windows 10. The best would be to cover full screen and windowed mode (but fullscreen is probably more important)
Any advice how to proceed (or why I am getting the "cached" data) would be nice :)
Of course if someone will say (with full authority), that what i want to achieve is impossible with CopyFromScreen (and there is no hack to fix that, apart from minimizing and maximazing the screen) i will consider option with injecting the code. But normally i would want to stay away from this one, as this could be treated as cheat and can lead to VAC ban.
====== UPDATE ======
You can try reproduce the process of taking screenshot by downloading the game (is small one, 260 MB):
https://cssetti.pl/Api/GameDownload.php?GameDownloadId=v43
Then you can copy-paste my code to Linqpad (or any other editor) and run the code. The application after launching will launch the HL process which is then use to try to grab the screenshot.
====== UPDATE 2 ======
In windows mode everything works correctly (the printscreens are ok)

C# how to detect which keyboard is pressed? [duplicate]

I have a barcode scanner (which acts like a keyboard) and of course I have a keyboard too hooked up to a computer. The software is accepting input from both the scanner and the keyboard. I need to accept only the scanner's input. The code is written in C#. Is there a way to "disable" input from the keyboard and only accept input from the scanner?
Note:
Keyboard is part of a laptop...so it cannot be unplugged. Also, I tried putting the following code
protected override Boolean ProcessDialogKey(System.Windows.Forms.Keys keyData)
{
return true;
}
But then along with ignoring the keystrokes from the keyboard, the barcode scanner input is also ignored.
I cannot have the scanner send sentinal characters as, the scanner is being used by other applications and adding a sentinal character stream would mean modifying other code.
Also, I cannot use the timing method of determining if the input came from a barcode scanner (if its a bunch of characters followed by a pause) since the barcodes scanned could potentially be single character barcodes.
Yes, I am reading data from a stream.
I am trying to follow along with the article: Distinguishing Barcode Scanners from the Keyboard in WinForms. However I have the following questions:
I get an error NativeMethods is inaccessible due to its protection level. It seems as though I need to import a dll; is this correct? If so, how do I do it?
Which protected override void WndProc(ref Message m) definition should I use, there are two implementations in the article?
Am getting an error related to [SecurityPermission( SecurityAction.LinkDemand, Flags = SecurityPermissionFlag.UnmanagedCode)] error CS0246: The type or namespace name 'SecurityPermission' could not be found (are you missing a using directive or an assembly reference?). How do I resolve this error?
There is also an error on the line containing: if ((from hardwareId in hardwareIds where deviceName.Contains(hardwareId) select hardwareId).Count() > 0) Error is error CS1026: ) expected.
Should I be placing all the code in the article in one .cs file called BarcodeScannerListener.cs?
Followup questions about C# solution source code posted by Nicholas Piasecki on http://nicholas.piasecki.name/blog/2009/02/distinguishing-barcode-scanners-from-the-keyboard-in-winforms/:
I was not able to open the solution in VS 2005, so I downloaded Visual C# 2008 Express Edition, and the code ran. However, after hooking up my barcode scanner and scanning a barcode, the program did not recognize the scan. I put a break point in OnBarcodeScanned method but it never got hit. I did change the App.config with the id of my Barcode scanner obtained using Device Manager. There seems to be 2 deviceNames with HID#Vid_0536&Pid_01c1 (which is obtained from Device Manager when the scanner is hooked up). I don't know if this is causing the scanning not to work. When iterating over the deviceNames, here is the list of devices I found (using the debugger):
"\??\HID#Vid_0536&Pid_01c1&MI_01#9&25ca5370&0&0000#{4d1e55b2-f16f-11cf-88cb-001111000030}"
"\??\HID#Vid_0536&Pid_01c1&MI_00#9&38e10b9&0&0000#{884b96c3-56ef-11d1-bc8c-00a0c91405dd}"
"\??\HID#Vid_413c&Pid_2101&MI_00#8&1966e83d&0&0000#{884b96c3-56ef-11d1-bc8c-00a0c91405dd}"
"\??\HID#Vid_413c&Pid_3012#7&960fae0&0&0000#{378de44c-56ef-11d1-bc8c-00a0c91405dd}"
"\??\Root#RDP_KBD#0000#{884b96c3-56ef-11d1-bc8c-00a0c91405dd}"
"\??\ACPI#PNP0303#4&2f94427b&0#{884b96c3-56ef-11d1-bc8c-00a0c91405dd}"
"\??\Root#RDP_MOU#0000#{378de44c-56ef-11d1-bc8c-00a0c91405dd}"
"\??\ACPI#PNP0F13#4&2f94427b&0#{378de44c-56ef-11d1-bc8c-00a0c91405dd}"
So there are 2 entries for HID#Vid_0536&Pid_01c1; could that be causing the scanning not to work?
OK so it seems that I had to figure out a way to not depend on the ASCII 0x04 character being sent by the scanner...since my scanner does not send that character. After that, the barcode scanned event is fired and the popup with the barcode is shown. So thanks Nicholas for your help.

You could use the Raw Input API to distinguish between the keyboard and the scanner like I did recently. It doesn't matter how many keyboard or keyboard-like devices you have hooked up; you will see a WM_INPUT before the keystroke is mapped to a device-independent virtual key that you typically see in a KeyDown event.
Far easier is to do what others have recommended and configure the scanner to send sentinel characters before and after the barcode. (You usually do this by scanning special barcodes in the back of the scanner's user manual.) Then, your main form's KeyPreview event can watch those roll end and swallow the key events for any child control if it's in the middle of a barcode read. Or, if you wanted to be fancier, you could use a low-level keyboard hook with SetWindowsHookEx() to watch for those sentinels and swallow them there (advantage of this is you could still get the event even if your app didn't have focus).
I couldn't change the sentinel values on our barcode scanners among other things so I had to go the complicated route. Was definitely painful. Keep it simple if you can!
--
Your update, seven years later: If your use case is reading from a USB barcode scanner, Windows 10 has a nice, friendly API for this built-in in Windows.Devices.PointOfService.BarcodeScanner. It's a UWP/WinRT API, but you can use it from a regular desktop app as well; that's what I'm doing now. Here's some example code for it, straight from my app, to give you the gist:
{
using System;
using System.Linq;
using System.Threading.Tasks;
using System.Windows;
using Windows.Devices.Enumeration;
using Windows.Devices.PointOfService;
using Windows.Storage.Streams;
using PosBarcodeScanner = Windows.Devices.PointOfService.BarcodeScanner;
public class BarcodeScanner : IBarcodeScanner, IDisposable
{
private ClaimedBarcodeScanner scanner;
public event EventHandler<BarcodeScannedEventArgs> BarcodeScanned;
~BarcodeScanner()
{
this.Dispose(false);
}
public bool Exists
{
get
{
return this.scanner != null;
}
}
public void Dispose()
{
this.Dispose(true);
GC.SuppressFinalize(this);
}
public async Task StartAsync()
{
if (this.scanner == null)
{
var collection = await DeviceInformation.FindAllAsync(PosBarcodeScanner.GetDeviceSelector());
if (collection != null && collection.Count > 0)
{
var identity = collection.First().Id;
var device = await PosBarcodeScanner.FromIdAsync(identity);
if (device != null)
{
this.scanner = await device.ClaimScannerAsync();
if (this.scanner != null)
{
this.scanner.IsDecodeDataEnabled = true;
this.scanner.ReleaseDeviceRequested += WhenScannerReleaseDeviceRequested;
this.scanner.DataReceived += WhenScannerDataReceived;
await this.scanner.EnableAsync();
}
}
}
}
}
private void WhenScannerDataReceived(object sender, BarcodeScannerDataReceivedEventArgs args)
{
var data = args.Report.ScanDataLabel;
using (var reader = DataReader.FromBuffer(data))
{
var text = reader.ReadString(data.Length);
var bsea = new BarcodeScannedEventArgs(text);
this.BarcodeScanned?.Invoke(this, bsea);
}
}
private void WhenScannerReleaseDeviceRequested(object sender, ClaimedBarcodeScanner args)
{
args.RetainDevice();
}
private void Dispose(bool disposing)
{
if (disposing)
{
this.scanner = null;
}
}
}
}
Granted, you'll need a barcode scanner that supports the USB HID POS and isn't just a keyboard wedge. If your scanner is just a keyboard wedge, I recommend picking up something like a used Honeywell 4600G off eBay for like $25. Trust me, your sanity will be worth it.

What I did in a similar situation is distinguish between a scan and a user typing by looking at the speed of the input.
Lots of characters very close together then a pause is a scan. Anything else is keyboard input.
I don't know exactly your requirements, so maybe that won't do for you, but it's the best I've got :)

It depends on the way you are interacting with the device. Anyway it wont be a C# solution, it will be some other library. Are you reading data from a stream? If you are just taking keystrokes, there may be nothing you can do about it.

I know this is an old thread, found it by searching barcode scanning in WIN10.
Just a few notes in case someone needs it.
These scanners from Honeywell have several USB interfaces.
One is a keyboard + Hid Point of sales (composite device).
Also there are CDC-ACM (ComPort emulation) and Hid Point of sales (alone) + more.
By default the scanners expose a serial number, so the host can distinguish between many devices (I had once +20 connected). There is a command to disable the serial number though!
The newer models behave the same in this regard.
If you want to see it live, try my terminal program yat3 (free on my site).
It can open all the interfaces mentioned above and is tailored for such devices.
A word to use keyboard interfaces:
Only use them as a last resort. They are slow, less reliable when it comes to exotic characters. The only good use is if you want to enter data into existing applications. If you code anyway, then reading from ComPort/HidPos-Device is easier.

look at this: http://nate.dynalias.net/dev/keyboardredirector.rails (NOT AVAILABLE ANYMORE) works great!
Specify the keyboard and the keys you want to block, and it works like a charm!
Also take a look at this: http://www.oblita.com/interception.html
You can create a C# wrapper for it - it also works like a charm..

I think you might be able to distinguish multiple keyboards through DirectX API, or if that doesn't work, through raw input API.

I have successfully accomplished what you folks are looking for here. I have an application that receives all barcode character data from a Honeywell/Metrologic barcode scanner. No other application on the system receives the data from the scanner, and the keyboard continues to function normally.
My application uses a combination of raw input and the dreaded low-level keyboard hook system. Contrary to what is written here, I found that the wm_input message is received before the keyboard hook function is called. My code to process the wm_input message basically sets a boolean variable to specify whether or not the received character is from the scanner. The keyboard hook function, called immediately after the wm_input is processed, swallows the scanner’s pseudo-keyboard data, preventing the data from being received by other applications.
The keyboard hook function has to be placed in an dll since you want to intercept all system keyboard messages. Also, a memory mapped file has to be used for the wm_input processing code to communicate with the dll.

Unity3D KeyCode.Home issue?

Hello i have a line of code that should quit the game upon pressing the three hardware button on the bottom like Menu, Home, Back
it seems that the back and the home only is working but the menu is not working
I have a Xiaomi Redmi Note 4X phone, how am i suppose to fix this issue?
Is it the same "Menu" as the other phone where when you press it, it'll show list of the opened app
here's the code:
if (Input.GetKey (KeyCode.Home) || Input.GetKey (KeyCode.Escape) || Input.GetKey (KeyCode.Menu))
{
Save ();
System.Diagnostics.Process.GetCurrentProcess ().Kill ();
}

Is it the same "Menu" as the other phone where when you press it,
it'll show list of the opened ap
Yes.
I have a Xiaomi Redmi Note 4X phone, how am i suppose to fix this
issue?
It should work otherwise it is a bug and you should file for a bug request. Before you file for abug request, use the script below to make that the Menu key is not mapped to another key.
public class KeyCodeFinder : MonoBehaviour
{
public Text text;
Array allKeyCodes;
void Start()
{
allKeyCodes = System.Enum.GetValues(typeof(KeyCode));
}
void Update()
{
foreach (KeyCode tempKey in allKeyCodes)
{
if (Input.GetKeyDown(tempKey))
{
text.text = "Pressed: KeyCode." + tempKey;
Debug.Log("Pressed: KeyCode." + tempKey);
}
}
}
}
Meanwhile, you can make a Java plugin that calls your C# function when the Menu keycode is pressed. That's another option you have.

Those buttons are OS related, you don't really control what happens with those. Well, you can track them and kill your app but this is going against what every single application does. That is, if you press the back or home or quit, it gets back to the main OS page, but your app is still on the stack ready to get back.
If you want to save when the user quits, maybe you want to look into
https://docs.unity3d.com/ScriptReference/MonoBehaviour.OnApplicationPause.html
This will make your app behave in a more common way.
On a side note, I would even assume this kind of attempt to over control the app would be rejected for AppStore (iOS) as it would not comply with their generic requirements.

It is allways possible to map a custom KeyCode that is not mapped so far.
First find out what keycode the key has
Event e = Event.current;
if (e.isKey)
{
Debug.Log("Detected key code: " + e.keyCode);
}
Let's say this gives you 10 as keycode (a one that is not mapped in Unities KeyCode so far)
You can than simply define a new one like
KeyCode MY_HOME_BUTTON = (KeyCode) 10;
and than use it like
if(Input.MY_HOME_BUTTON)
{
...
}
If it is completely possible to catch the home button depends on the OS (most of the time those buttons cannot be catched by an app but are handled by the OS first)

Which way I can detect if keyboard and mouse drivers are in broken state (not loaded)?

Since Win 8.1 reimagining happens to me nearly every week, because of that: https://superuser.com/questions/744564/any-solution-appeared-to-fix-win-8-1-drivers-disaster-keyboard-stopped-to-work/747108#747108
I decided to write the program that will run at OS startup, wait for 1 minute (make sure that all drivers should be loaded) and check, if keyboard and mouse in bad state:
And, if bad state found, begins to copy needed for me folders from dick C to backup drive, so I can restore it later, after reimage disk C.
Based on my another question about receiving motherboard ID, I know I can query some info with ManagementObjectSearcher, but it does not give any info about device state.
[Caption][Base board]
[ConfigOptions][NULL]
[CreationClassName][Win32_BaseBoard]
[Depth][NULL]
[Description][Base board]
[Height][NULL]
[HostingBoard][True]
[HotSwappable][False]
[InstallDate][NULL]
[Manufacturer][Gigabyte Technology Co., Ltd.]
[Model][NULL]
[Name][Base board]
[OtherIdentifyingInfo][NULL]
[PartNumber][NULL]
[PoweredOn][True]
[Product][H55M-S2H]
[Removable][False]
[Replaceable][True]
[RequirementsDescription][NULL]
[RequiresDaughterBoard][False]
[SerialNumber][ ]
[SKU][NULL]
[SlotLayout][NULL]
[SpecialRequirements][NULL]
[Status][OK]
[Tag][Base Board]
[Version][x.x]
[Weight][NULL]
[Width][NULL]
So, how I can check if keyboard and mouse is broken?
I prefer .net solution, but of course it can be C or C++