I will try to explain clearly the context of my problem.
I have a server grabbing frames from an IP camera videostreaming. I'm able to process and edit these images but now I have to deliver these images again as a videostream(mjpeg). It is not mandatory that I need to deliver it as streaming to everyone, it would be enought by now if I can send the streaming through TCP to another client.
I don't have any idea what the best way to go about it would be, as I don't think sending the images one by one is a good practice...
Can anyone help me somehow? I've never dealt with this situation and I'm not able to find any solution on Google (probably I'm not doing the right query...)
How would you handle it?
I have found a simple CodeProject which performs a very similar task, streaming your desktop.
I hope it can help you as well!
Here is the simplest way to send Image as video stream over IP using Tcp
This method send and receive video stream.
This is based on Windows form.
Send Stream(Server):
private void Start_Sending_Video_Conference(string remote_IP, int port_number)
{
try
{
ms = new MemoryStream();// Store it in Binary Array as Stream
IDataObject data;
Image bmap;
// Copy image to clipboard
SendMessage(hHwnd, WM_CAP_EDIT_COPY, 0, 0);
// Get image from clipboard and convert it to a bitmap
data = Clipboard.GetDataObject();
if (data.GetDataPresent(typeof(System.Drawing.Bitmap)))
{
bmap = ((Image)(data.GetData(typeof(System.Drawing.Bitmap))));
bmap.Save(ms, ImageFormat.Bmp);
}
picCapture.Image.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);
byte[] arrImage = ms.GetBuffer();
myclient = new TcpClient(remote_IP, 5000);//Connecting with server
myns = myclient.GetStream();
mysw = new BinaryWriter(myns);
mysw.Write(arrImage);//send the stream to above address
ms.Flush();
mysw.Flush();
myns.Flush();
ms.Close();
mysw.Close();
myns.Close();
myclient.Close();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message, "Video Conference Error Message", MessageBoxButtons.OK, MessageBoxIcon.Error);
}
}
Receive Stream(Client):
private void Start_Receiving_Video_Conference()
{
try
{
// Open The Port
mytcpl = new TcpListener(5000);
mytcpl.Start(); // Start Listening on That Port
mysocket = mytcpl.AcceptSocket(); // Accept Any Request From Client and Start a Session
ns = new NetworkStream(mysocket); // Receives The Binary Data From Port
picture_comming.Image = Image.FromStream(ns);
mytcpl.Stop(); // Close TCP Session
if (mysocket.Connected == true) // Looping While Connected to Receive Another Message
{
while (true)
{
Start_Receiving_Video_Conference(); // Back to First Method
}
}
myns.Flush();
}
catch (Exception) { button1.Enabled = true; myth.Abort(); }
}
Unmanaged Imports:
[System.Runtime.InteropServices.DllImport("user32", EntryPoint="SendMessageA")]
static extern int SendMessage(int hwnd, int wMsg, int wParam, [MarshalAs(UnmanagedType.AsAny)]
object lParam);
[System.Runtime.InteropServices.DllImport("user32", EntryPoint="SetWindowPos")]
static extern int SetWindowPos(int hwnd, int hWndInsertAfter, int x, int y, int cx, int cy, int wFlags);
[System.Runtime.InteropServices.DllImport("user32")]
static extern bool DestroyWindow(int hndw);
[System.Runtime.InteropServices.DllImport("avicap32.dll")]
static extern int capCreateCaptureWindowA(string lpszWindowName, int dwStyle, int x, int y, int nWidth, short nHeight, int hWndParent, int nID);
[System.Runtime.InteropServices.DllImport("avicap32.dll")]
static extern bool capGetDriverDescriptionA(short wDriver, string lpszName, int cbName, string lpszVer, int cbVer);
Method To Preview Webcam Image on Server
This method should start first to initialize preview of webcam (On server side)
private void OpenPreviewWindow()
{
int iHeight = 259;
int iWidth = 369;
//
// Open Preview window in picturebox
//
hHwnd = capCreateCaptureWindowA(iDevice.ToString (), (WS_VISIBLE | WS_CHILD), 0, 0, 640, 480, picCapture.Handle.ToInt32(), 0);
//
// Connect to device
//
if (SendMessage(hHwnd, WM_CAP_DRIVER_CONNECT, iDevice, 0) == 1)
{
//
// Set the preview scale
//
SendMessage(hHwnd, WM_CAP_SET_SCALE, 1, 0);
//
// Set the preview rate in milliseconds
//
SendMessage(hHwnd, WM_CAP_SET_PREVIEWRATE, 66, 0);
//
// Start previewing the image from the camera
//
SendMessage(hHwnd, WM_CAP_SET_PREVIEW, 1, 0);
//
// Resize window to fit in picturebox
//
SetWindowPos(hHwnd, HWND_BOTTOM, 0, 0, iWidth,iHeight, (SWP_NOMOVE | SWP_NOZORDER));
}
else
{
//
// Error connecting to device close window
//
DestroyWindow(hHwnd);
}
}
To Close Preview:
private void ClosePreviewWindow()
{
//
// Disconnect from device
//
SendMessage(hHwnd, WM_CAP_DRIVER_DISCONNECT, iDevice, 0);
//
// close window
//
DestroyWindow(hHwnd);
}
WebCam Api
const short WM_CAP = 1024;
const int WM_CAP_DRIVER_CONNECT = WM_CAP + 10;
const int WM_CAP_DRIVER_DISCONNECT = WM_CAP + 11;
const int WM_CAP_EDIT_COPY = WM_CAP + 30;
const int WM_CAP_SET_PREVIEW = WM_CAP + 50;
const int WM_CAP_SET_PREVIEWRATE = WM_CAP + 52;
const int WM_CAP_SET_SCALE = WM_CAP + 53;
const int WS_CHILD = 1073741824;
const int WS_VISIBLE = 268435456;
const short SWP_NOMOVE = 2;
const short SWP_NOSIZE = 1;
const short SWP_NOZORDER = 4;
const short HWND_BOTTOM = 1;
Please ask if any query.
Related
I am doing a WinForms application with image/page scheme (the user is navigating between images by clicking at particular areas of the image). At one page I made rich textbox in which I wanted to load "rules" for the user to accept. Unfortunatelly, the rules are longer than the RTB, so I had to apply some kind of scrolling. I didn't want to let the user use a scrollbar - instead I made two buttons (up & down) by which the user can slide through the rules - user cant select or move text in any other way. Everything works great apart from one thing - after clicking the button, the text slides but part of the 1st and last shown lines are being cut (only part of the height of line is visible in rtb).
Below I list the code used for making such RTB (named "rules"):
rules.Clear();
rules.Cursor = Cursors.Default;
rules.ScrollBars = RichTextBoxScrollBars.Vertical;
rules.ReadOnly = true;
rules.KeyPress += rules_KeyPress;
rules.SelectionChanged += rules_SelectionChanged;
rules.Location = new System.Drawing.Point(450*this.Width/2000, 550*this.Height/1125);
Console.WriteLine(rules.Font);
float size = 16.5F;
rules.Font = new Font("Microsoft Sans Serif", size);
Console.WriteLine(rules.Font);
using (Graphics g = rules.CreateGraphics())
{
rules_line_height = Convert.ToInt32(g.MeasureString("A", rules.Font).Height); //getting the measure of the line
Console.WriteLine(rules_line_height);
}
int rules_h = 400 * this.Height / 1125; //height of the rtb
int lines_n = rules_h / rules_line_height; //max number of lines that can be fit in the height
rules.Size = new System.Drawing.Size(1000 * this.Width / 2000, lines_n*rules_line_height); //height of the rtb adjusted to fill full lines
rules.ForeColor = Color.White;
string fileName = "reg_pl.txt";
string line = "";
StreamReader file = new System.IO.StreamReader(Directory.GetCurrentDirectory() + "/reg/" + fileName, Encoding.GetEncoding("windows-1250"));
while ((line = file.ReadLine()) != null)
{
rules.AppendText(line + "\r\n");
}
file.Close();
rules_max = rules.GetMaxRange(); //getting max range of the rtb
rules_thumb = rules.GetThumb(); //getting size of visible part of the rtb
rules_loc = 0; // current location visible part compared to the whole rtb
rules.ScrollBars = RichTextBoxScrollBars.None;
rules.BorderStyle = BorderStyle.None;
rules.BackColor = Color.FromArgb(18, 25, 56);
rules.BringToFront();
I also used the custom_RTB class for getting max range and thumb of the RTB and scrolling to selected point of the RTB :
public class CustomRTB : System.Windows.Forms.RichTextBox
{
#region API Stuff
public struct SCROLLINFO
{
public int cbSize;
public int fMask;
public int min;
public int max;
public int nPage;
public int nPos;
public int nTrackPos;
}
[DllImport("user32.dll", CharSet = CharSet.Auto)]
public static extern int GetScrollPos(IntPtr hWnd, int nBar);
[DllImport("user32.dll")]
private static extern int SetScrollPos(IntPtr hWnd, int nBar, int nPos, bool bRedraw);
[DllImport("User32.Dll", EntryPoint = "PostMessageA")]
static extern bool PostMessage(IntPtr hWnd, uint msg, int wParam, int lParam);
[DllImport("user32")]
private static extern int GetScrollInfo(IntPtr hwnd, int nBar,
ref SCROLLINFO scrollInfo);
private const int SB_HORZ = 0x0;
private const int SB_VERT = 0x1;
#endregion
public int HorizontalPosition
{
get { return GetScrollPos((IntPtr)this.Handle, SB_HORZ); }
set { SetScrollPos((IntPtr)this.Handle, SB_HORZ, value, true); }
}
public int VerticalPosition
{
get { return GetScrollPos((IntPtr)this.Handle, SB_VERT); }
set { SetScrollPos((IntPtr)this.Handle, SB_VERT, value, true); }
}
public void ScrollTo(int Position)
{
SetScrollPos((IntPtr)this.Handle, 0x1, Position, true);
PostMessage((IntPtr)this.Handle, 0x115, 4 + 0x10000 * Position, 0);
}
public int GetMaxRange()
{
SCROLLINFO scrollInfo = new SCROLLINFO();
scrollInfo.cbSize = Marshal.SizeOf(scrollInfo);
//SIF_RANGE = 0x1, SIF_TRACKPOS = 0x10, SIF_PAGE= 0x2
scrollInfo.fMask = 0x10 | 0x1 | 0x2;
GetScrollInfo((IntPtr)this.Handle, 1, ref scrollInfo);//nBar = 1 -> VScrollbar
return scrollInfo.max;
}
public int GetThumb()
{
SCROLLINFO scrollInfo = new SCROLLINFO();
scrollInfo.cbSize = Marshal.SizeOf(scrollInfo);
//SIF_RANGE = 0x1, SIF_TRACKPOS = 0x10, SIF_PAGE= 0x2
scrollInfo.fMask = 0x10 | 0x1 | 0x2;
GetScrollInfo((IntPtr)this.Handle, 1, ref scrollInfo);//nBar = 1 -> VScrollbar
return scrollInfo.nPage;
}
public int GetCurPos()
{
SCROLLINFO scrollInfo = new SCROLLINFO();
scrollInfo.cbSize = Marshal.SizeOf(scrollInfo);
//SIF_RANGE = 0x1, SIF_TRACKPOS = 0x10, SIF_PAGE= 0x2
scrollInfo.fMask = 0x10 | 0x1 | 0x2;
GetScrollInfo((IntPtr)this.Handle, 1, ref scrollInfo);//nBar = 1 -> VScrollbar
return scrollInfo.nTrackPos;
}
public bool ReachedBottom()
{
SCROLLINFO scrollInfo = new SCROLLINFO();
scrollInfo.cbSize = Marshal.SizeOf(scrollInfo);
//SIF_RANGE = 0x1, SIF_TRACKPOS = 0x10, SIF_PAGE= 0x2
scrollInfo.fMask = 0x10 | 0x1 | 0x2;
GetScrollInfo((IntPtr)this.Handle, 1, ref scrollInfo);//nBar = 1 -> VScrollbar
return scrollInfo.max == scrollInfo.nTrackPos + scrollInfo.nPage;
}
}
And here are the "buttons" for moving the text up:
int lines = rules_thumb / rules_line_height; //getting max number of lines that can be visible
int new_thumb = lines * rules_line_height; //getting the move value so the lines won't be cut
if (rules_loc - (new_thumb) >= 0) //if moving by the move value will be still over 0, then proceed normally
{
rules_loc -= (new_thumb);
rules.ScrollTo(rules_loc);
}
else // if it will pass 0, the move to 0
{
rules_loc = 0;
rules.ScrollTo(rules_loc);
}
And down :
int lines = rules_thumb / rules_line_height;
int new_thumb = lines * rules_line_height;
if (rules_loc + (new_thumb) < rules_max)
{
rules_loc += (new_thumb);
rules.ScrollTo(rules_loc);
}
else
{
rules.ScrollTo(rules_max);
}
What I think I am doing wrong is measuring the lines heigh at given font and setting the good size for the RTB. Does anyone had similar problem and can somehow assist me with it?
Thank you in advance for the answer.
Best regards,
Sarachiel
I need help with HikVision IPCam Video streaming. I searched for 2 hole weeks without luck.
My problem is the IPCamm DLL stream the image into a picturebox using PictureBox.Handle. Its working perfectly fine:
[DllImport("HCNetSDK.dll")]
public static extern int NET_DVR_RealPlay_V30(int lUserID, ref NET_DVR_CLIENTINFO lpClientInfo, RealDataCallBack_V30 fRealDataCallBack_V30, IntPtr pUser, bool bBlocked);
this.realDataCallBack = new RealDataCallBack_V30(RealDataCallback);
this.clientInfo.hPlayWnd = PictureBox.Handle;
this.clientInfo.lChannel = channel;
this.clientInfo.lLinkMode = 0;
this.playHandle = NET_DVR_RealPlay_V30(this.userID, ref this.clientInfo, realDataCallBack, IntPtr.Zero, true);
My Issue is that I need to process the image but I couldn't have any way to capture the image as Bitmap or Image and then display it As I like.
I tried Bitmap.FromHbitmap(PictureBox.Handle), Tried some MemoryMarshel solutions with no luck.
My Only way to get it now is by getting the data from call back functions which is with lower quality, lower frame-count, ...
This snippet draws the data from the handle into a bitmap and then sets the image of the picturebox. The CopyFromScreen line might not be necessary on older systems.
PictureBox.Image = CaptureControl(PictureBox.Handle, PictureBox.Width, PictureBox.Height);
// PictureBox.Image now contains the data that was drawn to it
[DllImport("gdi32.dll")]
private static extern bool BitBlt(IntPtr hdcDest, int nXDest, int nYDest, int nWidth, int nHeight, IntPtr hdcSrc, int nXSrc, int nYSrc, int dwRop);
public Bitmap CaptureControl(IntPtr handle, int width, int height)
{
Bitmap controlBmp;
using (Graphics g1 = Graphics.FromHwnd(handle))
{
controlBmp = new Bitmap(width, height, g1);
using (Graphics g2 = Graphics.FromImage(controlBmp))
{
g2.CopyFromScreen(this.Location.X + PictureBox.Left, this.Location.Y + PictureBox.Top, 0, 0, PictureBox.Size);
IntPtr dc1 = g1.GetHdc();
IntPtr dc2 = g2.GetHdc();
BitBlt(dc2, 0, 0, width, height, handle, 0, 0, 13369376);
g1.ReleaseHdc(dc1);
g2.ReleaseHdc(dc2);
}
}
return controlBmp;
}
You need to set hPlayWnd as zero. Set callback function to work on decoded data. I try to understand Hikvision SDK, a few difficult...
lpPreviewInfo.hPlayWnd = IntPtr.Zero;//预览窗口 live view window
m_ptrRealHandle = RealPlayWnd.Handle;
RealData = new CHCNetSDK.REALDATACALLBACK(RealDataCallBack);//预览实时流回调函数 real-time stream callback function
m_lRealHandle = CHCNetSDK.NET_DVR_RealPlay_V40(m_lUserID, ref lpPreviewInfo, RealData, pUser);
public void RealDataCallBack(Int32 lRealHandle, UInt32 dwDataType, IntPtr pBuffer, UInt32 dwBufSize, IntPtr pUser)
//解码回调函数
private void DecCallbackFUN(int nPort, IntPtr pBuf, int nSize, ref PlayCtrl.FRAME_INFO pFrameInfo, int nReserved1, int nReserved2)
{
// 将pBuf解码后视频输入写入文件中(解码后YUV数据量极大,尤其是高清码流,不建议在回调函数中处理)
if (pFrameInfo.nType == 3) //#define T_YV12 3
{
// FileStream fs = null;
// BinaryWriter bw = null;
// try
// {
// fs = new FileStream("DecodedVideo.yuv", FileMode.Append);
// bw = new BinaryWriter(fs);
// byte[] byteBuf = new byte[nSize];
// Marshal.Copy(pBuf, byteBuf, 0, nSize);
// bw.Write(byteBuf);
// bw.Flush();
// }
// catch (System.Exception ex)
// {
// MessageBox.Show(ex.ToString());
// }
// finally
// {
// bw.Close();
// fs.Close();
// }
}
}
See the source code
I am loading the location of the windows last position from a config file (it is unnecessary to know why I am doing that and not using App Settings). The values being read in are correct and sent to the SetWindowPos P/Invoke function. However, on returning from SetWindowPos the values have changed.
Here is the code that gets the X Y positions and calls SetWindowPos
private void LoadSettings()
{
if (File.Exists(GetSettingsPath()))
{
string p = GetSettingsPath();
FileStream fs = null;
StreamReader sr = null;
try
{
fs = new FileStream(p, FileMode.Open, FileAccess.Read, FileShare.Read);
sr = new StreamReader(fs);
XPos = int.Parse(sr.ReadLine());
YPos = int.Parse(sr.ReadLine());
thisWidth = int.Parse(sr.ReadLine());
thisHeight = int.Parse(sr.ReadLine());
// Now throw the window to back and prevent it interacting.
SetWindowPos(Handle, HWND_BOTTOM, XPos, YPos, thisWidth, thisHeight, SWP_NOMOVE | SWP_NOSIZE | SWP_NOREPOSITION | SWP_NOACTIVATE); // Set Form1 as BottomMost
sr.Close();
fs.Close();
}
catch (Exception)
{
if (sr != null) sr.Close();
if (fs != null) fs.Close();
}
}
else
{
SaveDefaultSettings();
}
}
Here is the P/Invoke definition:
[DllImport("user32.dll")]
static extern bool SetWindowPos(IntPtr hWnd, IntPtr hWndInsertAfter, int X, int Y, int cx, int cy, uint uFlags);
static readonly IntPtr HWND_BOTTOM = new IntPtr(1);
const UInt32 SWP_NOSIZE = 0x0001;
const UInt32 SWP_NOMOVE = 0x0002;
const UInt32 SWP_NOACTIVATE = 0x0010;
const UInt32 SWP_NOREPOSITION = 0x0200;
private int XPos = 0;
private int YPos = 0;
private int thisWidth = 0;
private int thisHeight = 0;
Obviously the XPos, YPos, thisWidth and thisHeight are not part of SetWindoPos but are here because that is were they are used.
So to the Questions:
Why is the X and Y Coordinates being changed by SetWindowsPos (The X value is < 1600 and the Y value is around 4)?
Can I stop this from happening?
Can I stop the window Flashing when being created and set to the BottomMost position?
Could it be that the window has a min size defined?
I'm currently working on creating an Ambilight for my computer monitor with C#, an arduino, and an Ikea Dioder. Currently the hardware portion runs flawlessly; however, I'm having a problem with detecting the average color of a section of screen.
I have two issues with the implementations that I'm using:
Performance - Both of these algorithms add a somewhat noticeable stutter to the screen. Nothing showstopping, but it's annoying while watching video.
No Fullscreen Game Support - When a game is in fullscreen mode both of these methods just return white.
public class DirectxColorProvider : IColorProvider
{
private static Device d;
private static Collection<long> colorPoints;
public DirectxColorProvider()
{
PresentParameters present_params = new PresentParameters();
if (d == null)
{
d = new Device(new Direct3D(), 0, DeviceType.Hardware, IntPtr.Zero, CreateFlags.SoftwareVertexProcessing, present_params);
}
if (colorPoints == null)
{
colorPoints = GetColorPoints();
}
}
public byte[] GetColors()
{
var color = new byte[4];
using (var screen = this.CaptureScreen())
{
DataRectangle dr = screen.LockRectangle(LockFlags.None);
using (var gs = dr.Data)
{
color = avcs(gs, colorPoints);
}
}
return color;
}
private Surface CaptureScreen()
{
Surface s = Surface.CreateOffscreenPlain(d, Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height, Format.A8R8G8B8, Pool.Scratch);
d.GetFrontBufferData(0, s);
return s;
}
private static byte[] avcs(DataStream gs, Collection<long> positions)
{
byte[] bu = new byte[4];
int r = 0;
int g = 0;
int b = 0;
int i = 0;
foreach (long pos in positions)
{
gs.Position = pos;
gs.Read(bu, 0, 4);
r += bu[2];
g += bu[1];
b += bu[0];
i++;
}
byte[] result = new byte[3];
result[0] = (byte)(r / i);
result[1] = (byte)(g / i);
result[2] = (byte)(b / i);
return result;
}
private Collection<long> GetColorPoints()
{
const long offset = 20;
const long Bpp = 4;
var box = GetBox();
var colorPoints = new Collection<long>();
for (var x = box.X; x < (box.X + box.Length); x += offset)
{
for (var y = box.Y; y < (box.Y + box.Height); y += offset)
{
long pos = (y * Screen.PrimaryScreen.Bounds.Width + x) * Bpp;
colorPoints.Add(pos);
}
}
return colorPoints;
}
private ScreenBox GetBox()
{
var box = new ScreenBox();
int m = 8;
box.X = (Screen.PrimaryScreen.Bounds.Width - m) / 3;
box.Y = (Screen.PrimaryScreen.Bounds.Height - m) / 3;
box.Length = box.X * 2;
box.Height = box.Y * 2;
return box;
}
private class ScreenBox
{
public long X { get; set; }
public long Y { get; set; }
public long Length { get; set; }
public long Height { get; set; }
}
}
You can find the file for the directX implmentation here.
public class GDIColorProvider : Form, IColorProvider
{
private static Rectangle box;
private readonly IColorHelper _colorHelper;
public GDIColorProvider()
{
_colorHelper = new ColorHelper();
box = _colorHelper.GetCenterBox();
}
public byte[] GetColors()
{
var colors = new byte[3];
IntPtr hDesk = GetDesktopWindow();
IntPtr hSrce = GetDC(IntPtr.Zero);
IntPtr hDest = CreateCompatibleDC(hSrce);
IntPtr hBmp = CreateCompatibleBitmap(hSrce, box.Width, box.Height);
IntPtr hOldBmp = SelectObject(hDest, hBmp);
bool b = BitBlt(hDest, box.X, box.Y, (box.Width - box.X), (box.Height - box.Y), hSrce, 0, 0, CopyPixelOperation.SourceCopy);
using(var bmp = Bitmap.FromHbitmap(hBmp))
{
colors = _colorHelper.AverageColors(bmp);
}
SelectObject(hDest, hOldBmp);
DeleteObject(hBmp);
DeleteDC(hDest);
ReleaseDC(hDesk, hSrce);
return colors;
}
// P/Invoke declarations
[DllImport("gdi32.dll")]
static extern bool BitBlt(IntPtr hdcDest, int xDest, int yDest, int
wDest, int hDest, IntPtr hdcSource, int xSrc, int ySrc, CopyPixelOperation rop);
[DllImport("user32.dll")]
static extern bool ReleaseDC(IntPtr hWnd, IntPtr hDc);
[DllImport("gdi32.dll")]
static extern IntPtr DeleteDC(IntPtr hDc);
[DllImport("gdi32.dll")]
static extern IntPtr DeleteObject(IntPtr hDc);
[DllImport("gdi32.dll")]
static extern IntPtr CreateCompatibleBitmap(IntPtr hdc, int nWidth, int nHeight);
[DllImport("gdi32.dll")]
static extern IntPtr CreateCompatibleDC(IntPtr hdc);
[DllImport("gdi32.dll")]
static extern IntPtr SelectObject(IntPtr hdc, IntPtr bmp);
[DllImport("user32.dll")]
private static extern IntPtr GetDesktopWindow();
[DllImport("user32.dll")]
private static extern IntPtr GetWindowDC(IntPtr ptr);
[DllImport("user32.dll")]
private static extern IntPtr GetDC(IntPtr ptr);
}
You Can Find the File for the GDI implementation Here.
The Full Codebase Can be Found Here.
Updated Answer
The problem of slow screen capture performance most likely is caused by BitBlt() doing a pixel conversion when the pixel formats of source and destination don't match. From the docs:
If the color formats of the source and destination device contexts do not match, the BitBlt function converts the source color format to match the destination format.
This is what caused slow performance in my code, especially in higher resolutions.
The default pixel format seems to be PixelFormat.Format32bppArgb, so that's what you should use for the buffer:
var screen = new Bitmap(bounds.Width, bounds.Height, PixelFormat.Format32bppArgb);
var gfx = Graphics.FromImage(screen);
gfx.CopyFromScreen(bounds.Location, new Point(0, 0), bounds.Size);
The next source for slow performance is Bitmap.GetPixel() which does boundary checks. Never use it when analyzing every pixel. Instead lock the bitmap data and get a pointer to it:
public unsafe Color GetAverageColor(Bitmap image, int sampleStep = 1) {
var data = image.LockBits(
new Rectangle(Point.Empty, Image.Size),
ImageLockMode.ReadOnly,
PixelFormat.Format32bppArgb);
var row = (int*)data.Scan0.ToPointer();
var (sumR, sumG, sumB) = (0L, 0L, 0L);
var stride = data.Stride / sizeof(int) * sampleStep;
for (var y = 0; y < data.Height; y += sampleStep) {
for (var x = 0; x < data.Width; x += sampleStep) {
var argb = row[x];
sumR += (argb & 0x00FF0000) >> 16;
sumG += (argb & 0x0000FF00) >> 8;
sumB += argb & 0x000000FF;
}
row += stride;
}
image.UnlockBits(data);
var numSamples = data.Width / sampleStep * data.Height / sampleStep;
var avgR = sumR / numSamples;
var avgG = sumG / numSamples;
var avgB = sumB / numSamples;
return Color.FromArgb((int)avgR, (int)avgG, (int)avgB);
}
This should get you well below 10 ms, depending on the screen size. In case it is still too slow you can increase the sampleStep parameter of GetAverageColor().
Original Answer
I recently did the same thing and came up with something that worked surprisingly good.
The trick is to create an additional bitmap that is 1x1 pixels in size and set a good interpolation mode on its graphics context (bilinear or bicubic, but not nearest neighbor).
Then you draw your captured bitmap into that 1x1 bitmap exploiting the interpolation and retrieve that pixel to get the average color.
I'm doing that at a rate of ~30 fps. When the screen shows a GPU rendering (e.g. watching YouTube full screen with enabled hardware acceleration in Chrome) there is no visible stuttering or anything. In fact, CPU utilization of the application is way below 10%. However, if I turn off Chrome's hardware acceleration then there is definitely some slight stuttering noticeable if you watch close enough.
Here are the relevant parts of the code:
using var screen = new Bitmap(width, height);
using var screenGfx = Graphics.FromImage(screen);
using var avg = new Bitmap(1, 1);
using var avgGfx = Graphics.FromImage(avg);
avgGfx.InterpolationMode = InterpolationMode.HighQualityBicubic;
while (true) {
screenGfx.CopyFromScreen(left, top, 0, 0, screen.Size);
avgGfx.DrawImage(screen, 0, 0, avg.Width, avg.Height);
var color = avg.GetPixel(0, 0);
var bright = (int)Math.Round(Math.Clamp(color.GetBrightness() * 100, 1, 100));
// set color and brightness on your device
// wait 1000/fps milliseconds
}
Note that this works for GPU renderings, because System.Drawing.Common uses GDI+ nowadays. However, it does not work when the content is DRM protected. So it won't work with Netflix for example :(
I published the code on GitHub. Even though I abandoned the project due to Netflix' DRM protection it might help someone else.
Here is my code that I use to simulate a tab-keypress in a certain process:
[DllImport("user32.dll")]
static extern bool PostMessage(IntPtr hWnd, UInt32 Msg, int wParam, int lParam);
public Form1()
{
PostMessage(MemoryHandler.GetMainWindowHandle(),
(int)KeyCodes.WMessages.WM_KEYDOWN,
(int)KeyCodes.VKeys.VK_TAB, 0);
InitializeComponent();
}
Is there any way to extend it so that it presses the key for (example) 1 second, instead of just tapping it?
Please note that I'm not interested in a Thread.Sleep() solution that blocks the UI thread.
The repeating of a keystroke when you hold it down is a feature that's built into the keyboard controller. A microprocessor that's built into the keyboard itself. The 8042 microcontroller was the traditional choice, the keyboard device driver still carries its name.
So, no, this is not done by Windows and PostMessage() is not going to do it for you. Not exactly a problem, you can simply emulate it with a Timer.
If you want to simulate what Windows does with messages, you likely want to find out how fast the key repeat rate is. That can be found at HKEY_CURRENT_USER\Control Panel\Keyboard\KeyboardSpeed. there's also the KeyboardDelay value.
What Windows does is send a WM_KEYDOWN and a WM_CHAR when a key is initially pressed. Then, if the key is still pressed after KeyboardDelay time span, the WM_KEYDOWN and WM_CHAR pair are repeated every KeyboardSpeed until the key is depressed--at which point WM_KEYUP is sent.
I would suggest using a Timer to send the messages at a specific frequencies.
Update:
for example:
int keyboardDelay, keyboardSpeed;
using (var key = Registry.CurrentUser.OpenSubKey(#"Control Panel\Keyboard"))
{
Debug.Assert(key != null);
keyboardDelay = 1;
int.TryParse((String)key.GetValue("KeyboardDelay", "1"), out keyboardDelay);
keyboardSpeed = 31;
int.TryParse((String)key.GetValue("KeyboardSpeed", "31"), out keyboardSpeed);
}
maxRepeatedCharacters = 30; // repeat char 30 times
var startTimer = new System.Windows.Forms.Timer {Interval = keyboardSpeed};
startTimer.Tick += startTimer_Tick;
startTimer.Start();
var repeatTimer = new System.Windows.Forms.Timer();
repeatTimer.Interval += keyboardDelay;
repeatTimer.Tick += repeatTimer_Tick;
//...
private static void repeatTimer_Tick(object sender, EventArgs e)
{
PostMessage(MemoryHandler.GetMainWindowHandle(),
(int)KeyCodes.WMessages.WM_KEYDOWN,
(int)KeyCodes.VKeys.VK_TAB, 0);
PostMessage(MemoryHandler.GetMainWindowHandle(),
(int)KeyCodes.WMessages.WM_CHAR,
(int)KeyCodes.VKeys.VK_TAB, 0);
counter++;
if (counter > maxRepeatedCharacters)
{
Timer timer = sender as Timer;
timer.Stop();
}
}
private static void startTimer_Tick(object sender, EventArgs eventArgs)
{
Timer timer = sender as Timer;
timer.Stop();
PostMessage(MemoryHandler.GetMainWindowHandle(),
(int)KeyCodes.WMessages.WM_KEYDOWN,
(int)KeyCodes.VKeys.VK_TAB, 0);
PostMessage(MemoryHandler.GetMainWindowHandle(),
(int)KeyCodes.WMessages.WM_CHAR,
(int)KeyCodes.VKeys.VK_TAB, 0);
}
When holding a key down on a physical keyboard, repeated keystrokes are passed to the active window. This is not built into the keyboard, but is a Windows feature.
You can simulate this by doing the following steps in order:
Send a keydown message.
Run a timer at 30 ms intervals (the default in Windows, changeable through Ease of Access settings), sending a keypress message to the window at each tick.
Send a keyup message.
I would do it in a thread, for sleeping and not-blocking the UI thread.
Look at this:
System.Threading.Thread KeyThread = new System.Threading.Thread(() => {
//foreach process
// press key now
PostMessage(MemoryHandler.GetMainWindowHandle(),
(int)KeyCodes.WMessages.WM_KEYDOWN,
(int)KeyCodes.VKeys.VK_TAB, 0);
System.Threading.Thread.Sleep(1000); // wait 1 second
//foreach process
// release keys again
PostMessage(MemoryHandler.GetMainWindowHandle(),
(int)KeyCodes.WMessages.WM_KEYUP,
(int)KeyCodes.VKeys.VK_TAB, 0);
});
Ofcourse, you have to start it.
I'm not sure exactly what you're trying to achieve, but below is the function I use to simulate text input using SendInput.
If you alter this slightly to make the final call to SendInput from a new thread, and then separate out the down and up events with a timer, does that achieve what you need?
[DllImport("user32.dll", SetLastError = true)]
static extern UInt32 SendInput(UInt32 numberOfInputs, INPUT[] inputs, Int32 sizeOfInputStructure);
public enum InputType : uint
{
MOUSE = 0,
KEYBOARD = 1,
HARDWARE = 2,
}
struct INPUT
{
public UInt32 Type;
public MOUSEKEYBDHARDWAREINPUT Data;
}
struct KEYBDINPUT
{
public UInt16 Vk;
public UInt16 Scan;
public UInt32 Flags;
public UInt32 Time;
public IntPtr ExtraInfo;
}
public enum KeyboardFlag : uint // UInt32
{
EXTENDEDKEY = 0x0001,
KEYUP = 0x0002,
UNICODE = 0x0004,
SCANCODE = 0x0008,
}
public static void SimulateTextEntry(string text)
{
if (text.Length > UInt32.MaxValue / 2) throw new ArgumentException(string.Format("The text parameter is too long. It must be less than {0} characters.", UInt32.MaxValue / 2), "text");
var chars = UTF8Encoding.ASCII.GetBytes(text);
var len = chars.Length;
INPUT[] inputList = new INPUT[len * 2];
for (int x = 0; x < len; x++)
{
UInt16 scanCode = chars[x];
var down = new INPUT();
down.Type = (UInt32)InputType.KEYBOARD;
down.Data.Keyboard = new KEYBDINPUT();
down.Data.Keyboard.Vk = 0;
down.Data.Keyboard.Scan = scanCode;
down.Data.Keyboard.Flags = (UInt32)KeyboardFlag.UNICODE;
down.Data.Keyboard.Time = 0;
down.Data.Keyboard.ExtraInfo = IntPtr.Zero;
var up = new INPUT();
up.Type = (UInt32)InputType.KEYBOARD;
up.Data.Keyboard = new KEYBDINPUT();
up.Data.Keyboard.Vk = 0;
up.Data.Keyboard.Scan = scanCode;
up.Data.Keyboard.Flags = (UInt32)(KeyboardFlag.KEYUP | KeyboardFlag.UNICODE);
up.Data.Keyboard.Time = 0;
up.Data.Keyboard.ExtraInfo = IntPtr.Zero;
// Handle extended keys:
// If the scan code is preceded by a prefix byte that has the value 0xE0 (224),
// we need to include the KEYEVENTF_EXTENDEDKEY flag in the Flags property.
if ((scanCode & 0xFF00) == 0xE000)
{
down.Data.Keyboard.Flags |= (UInt32)KeyboardFlag.EXTENDEDKEY;
up.Data.Keyboard.Flags |= (UInt32)KeyboardFlag.EXTENDEDKEY;
}
inputList[2*x] = down;
inputList[2*x + 1] = up;
}
var numberOfSuccessfulSimulatedInputs = SendInput((UInt32)len*2, inputList, Marshal.SizeOf(typeof(INPUT)));
}
Not sure what you're doing with the PostMessage but modifying some code from here:
SendKey.Send() Not working
[DllImport("user32.dll", SetLastError = true)]
static extern void keybd_event(byte bVk, byte bScan, uint dwFlags, UIntPtr dwExtraInfo);
public static void PressKey(Keys key, bool up)
{
const int KEYEVENTF_EXTENDEDKEY = 0x1;
const int KEYEVENTF_KEYUP = 0x2;
if (up)
{
keybd_event((byte)key, 0x45, KEYEVENTF_EXTENDEDKEY | KEYEVENTF_KEYUP, (UIntPtr)0);
}
else
{
keybd_event((byte)key, 0x45, KEYEVENTF_EXTENDEDKEY, (UIntPtr)0);
}
}
void TestProc()
{
PressKey(Keys.Tab, false);
Thread.Sleep(1000);
PressKey(Keys.Tab, true);
}
Maybe this could work for you. Its just a key down and then a key up with a sleep in between. You could even add to this further and pass a time value for how long you want the key to stay down.