Develop Reference

How to Capture Multiple Images Using Direct Show [duplicate]

I have a C# desktop application in which one thread that I create continously gets an image from a source(it's a digital camera actually) and puts it on a panel(panel.Image = img) in the GUI(which must be another thread as it is the code-behind of a control.
The application works but on some machines I get the following error at random time intervals(unpredictable)
************** Exception Text **************
System.InvalidOperationException: The object is currently in use elsewhere.
Then the panel turns into a red cross, red X - i think this is the invalid picture icon that is editable from the properties. The application keeps working but the panel is never updated.
From what I can tell this error comes from the control's onpaint event where I draw something else on the picture.
I tried using a lock there but no luck :(
The way I call the function that puts the image on the panel is as follows:
if (this.ReceivedFrame != null)
{
Delegate[] clients = this.ReceivedFrame.GetInvocationList();
foreach (Delegate del in clients)
{
try
{
del.DynamicInvoke(new object[] { this,
new StreamEventArgs(frame)} );
}
catch { }
}
}
this is the delegate:
public delegate void ReceivedFrameEventHandler(object sender, StreamEventArgs e);
public event ReceivedFrameEventHandler ReceivedFrame;
and this is how the function inside the control code-behind registers to it:
Camera.ReceivedFrame +=
new Camera.ReceivedFrameEventHandler(camera_ReceivedFrame);
I also tried
del.Method.Invoke(del.Target, new object[] { this, new StreamEventArgs(b) });
instead of
del.DynamicInvoke(new object[] { this, new StreamEventArgs(frame) });
but no luck
Does anyone know how I could fix this error or at least catch the error somehow and make the thread put the images on the panel once again?

This is because Gdi+ Image class is not thread safe. Hovewer you can avoid InvalidOperationException by using lock every time when you need to Image access, for example for painting or getting image size:
Image DummyImage;
// Paint
lock (DummyImage)
e.Graphics.DrawImage(DummyImage, 10, 10);
// Access Image properties
Size ImageSize;
lock (DummyImage)
ImageSize = DummyImage.Size;
BTW, invocation is not needed, if you will use the above pattern.

I had a similar problem with the same error message but try as I might, locking the bitmap didn't fix anything for me. Then I realized I was drawing a shape using a static brush. Sure enough, it was the brush that was causing the thread contention.
var location = new Rectangle(100, 100, 500, 500);
var brush = MyClass.RED_BRUSH;
lock(brush)
e.Graphics.FillRectangle(brush, location);
This worked for my case and lesson learned: Check all the reference types being used at the point where thread contention is occurring.

Seems to me, that the same Camera object is used several times.
E.g. try to use a new buffer for each received frame. It seems to me, that while the picture box is drawing the new frame, your capture library fills that buffer again. Therefore on faster machines this might not be an issue, with slower machines it might be an issue.
I've programmed something similar once, after each received frame, we had to request to receive the next frame and set the NEW frame receive buffer in that request.
If you can not do that, copy the received frame from the camera first to a new buffer and append that buffer to a queue, or just use 2 alternating buffers and check for overruns. Either use myOutPutPanel.BeginInvoke to call the camera_ReceivedFrame method, or better have a thread running, which checks the queue, when it has a new entry it calls mnyOutPutPanel.BeginInvoke to invoke your method to set the new buffer as image on the panel.
Furthermore, once you received the buffer, use the Panel Invoke Method to invoke the setting of the image (guarantee that it runs in the window thread and not the thread from your capture library).
The example below can be called from any thread (capture library or other separate thread):
void camera_ReceivedFrame(object sender, StreamEventArgs e)
{
if(myOutputPanel.InvokeRequired)
{
myOutPutPanel.BeginInvoke(
new Camera.ReceivedFrameEventHandler(camera_ReceivedFrame),
sender,
e);
}
else
{
myOutPutPanel.Image = e.Image;
}
}

I think this is multithreading problem
Use windows golden rule and update the panel in the main thread use panel.Invoke
This should overcome cross threading exception

UWP - How to use WriteableBimap in a background task? Alternative?

The following code reads an image, and centers (cuts and copies) that image into a resulting image with a black frame, returns the new image.
Can i do it without using a writeablebitmap class?
private async Task<WriteableBitmap> FrameToCenter(StorageFile image, Size newSize)
{
WriteableBitmap result = new WriteableBitmap((int) newSize.Width,(int) newSize.Height);
WriteableBitmap bitmapToFrame = await StorageFileToSoftwareBitmap(image);
Rect sourceRect = new Rect(0, 0, bitmapToFrame.PixelWidth, bitmapToFrame.PixelHeight);
int a = (int) Math.Min(newSize.Height, newSize.Width);
Rect destRect = new Rect((int) ((_screenResolution.Width - _screenResolution.Height)/2), 0, a, a);
result.Blit(destRect, bitmapToFrame, sourceRect);
return result;
}
Everything worked fine, but then I tried to put that code into a BackgroundTask in UWP, that results in an exception due to the fact that I can not use WriteableBitmap on a different thread then the UI thread.
Is there anything else I can do? I need to run it in a backgroundtask though.
As well the line:
result.Blit(destRect, bitmapToFrame, sourceRect);
is actually using a WriteableBitmapEx extension which I won't be able to use. Any other way? I really dont see a way how to manipulate a bitmap on a background task thread. I am quiet lost here. I've been even thinking about using openCV library and just do it in C++, would that solve the problem?
There is as well this SO Q&A, which mentions I should derive my background task from XamlRenderingBackgroundTask class. How can I do that?

You use the XamlBackgroundTask the same way you would a regular BackgroundTask
public sealed class MyBackgroundTask : XamlRenderingBackgroundTask
{
protected override async void OnRun(IBackgroundTaskInstance taskInstance)
{
var deferral = taskInstance.GetDeferral();
//your code here
deferral.Complete();
}
}
The difference is that this will have access to the UI thread so you can use your WriteableBitmap.
I would just keep an eye on the important recommendation in this article.
Important To keep the memory footprint of the background task as low as possible, this task should be implemented in a C++ Windows Runtime Component for Windows Phone. The memory footprint will be higher if written in C# and will cause out of memory exceptions on low-memory devices which will terminate the background task. For more information on memory constraints, see Support your app with background tasks.

Accessing System.Drawing.Bitmap from GTK# Thread throws Object Currently in use elsewhere exception

Im trying to manipulate an image using system.drawing in GTk#.I want the UI to update the image on screen as soon as the user updates a textbox.To implement this i tried using the background worker from winforms,it worked but when the textbox is updated at a higher speed the application becomes stuck with no error.
So i took a look at multithreading in GTK here http://www.mono-project.com/docs/gui/gtksharp/responsive-applications/ and created a thread .
void textboxchanged()
{
Thread thr = new Thread (new ThreadStart (ThreadRoutine));
thr.Start ();
}
static void ThreadRoutine ()
{
LargeComputation ();
}
static void LargeComputation ()
{
image=new Bitmap(backupimage);
//Long image processing
}
It works poorly than the background worker throws up object currently in use elsewhere error here image=new Bitmap(backupimage); when the speed of entry in textbox is even a little fast.What im i doing wrong ?
Update 1 :
Im not processing the same image using 2 different threads that does 2 different operations at the same time.Im calling the thread that does the same operation before the old thread is complete.As in background worker i need a way to check if the old thread has completed working before launching the new one.So basically what im looking for is a way to check if an instance of the same thread
is running.In winforms i used to do if(backgroundworker.isbusy==false) then do stuff
Update 2
Solution with performance degradation
As suggested by #voo Replacing the global bitmap helped solve the issue.What i did was instead of using global bitmap.I created a global string(filename).Now i use img=new Bitmap(filename).Tried executing fast as i can no error came up.So inorder to update the GUI i used the invoke as suggested here mono-project.com/docs/gui/gtksharp/responsive-applications/.The thing is no error comes up and image gets updated,but when the typing operation is fast enough there a wait involved. Performance got degraded.This was not the case with background worker.Is there a way to improve performance.
At end of the large image processing operation method i added this to update GUI
Gtk.Application.Invoke (delegate {
MemoryStream istream=new MemoryStream();
img.Save (istream, System.Drawing.Imaging.ImageFormat.Png);
istream.Position = 0;
workimagepixbuff = new Gdk.Pixbuf (istream);
image1.Pixbuf = workimagepixbuff.ScaleSimple (400, 300, Gdk.InterpType.Bilinear);
});
// cannot directly convert Bitmap to Pixbuff,so doing this

The problem here is that you are processing an image at two places (two threads) on the same time and the image operation in .Net (GDI speaking) does not allow this.
Because you did not provided very much information I'm just guessing here.
When manipulating bitmap images in GDI, there are BitmapData behind the scene that needs to be locked and unlocked. This mechanism just make the picture available in memory for read/write. But AFAIK when you lock a BitmapData that is already locked you get a similar exception : System.InvalidOperationException, Bitmap region is already locked.
To me it sounds like you are getting this kind of error, but with other words because you are not explicitly locking bitmap data bits. GDI just tell you : "I would have to lock bitmap data bits but I can't because the object is already used (locked) elsewhere."
The solution here may be to try to synchronize bitmap usage (where bits locking may be involved) between thread whenever they may occur. So you may want to use the lock keyword or a similar mechanism:
So try something that looks like the following:
private static object _imageLock = new object();
static void LargeComputation ()
{
lock(_imageLock)
{
image=new Bitmap(backupimage);
//Long image processing ...
}
}
static void AnotherImageOperationSomewhereElse()
{
lock(_imageLock)
{
//Another image processing on backupImage or something derived from it...
}
}

User created pixel byte array does not appear to update correctly (WPF)

I have a system where I am able to collect 8-bit Gray images from a camera, place the data into a WriteableBitmap and display the images on a WPF Image object. This work happens in a camera thread. I used this article to help me: How to create a BitmapImage from a pixel byte array (live video display)
What I am trying to to do is make a copy of a subset of the pixel data of the image data. During the frame update in the camera thread I am trying to create the copy of the data in a separate byte array. My code appears to work at first, but then after a few iterations my buffer variable goes from having a range of grey levels (0-255) to only having values of 255 in every array element. The variable appears to accumulate data and max out as opposed to reset each time the background worker is called. I will present my code below.
Can anyone see and describe to me what I am doing wrong? Thank you.
public partial class MainWindow : Window
{
[DllImport("Kernel32.dll",EntryPoint="RtlMoveMemory")]
public static extern void CopyMemory(IntPtr Destination, IntPtr Source,
uint Length);
// Declarations
var pData = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(byte))*FrameSize);
var pFocusData = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(byte))
*FrameSize);
BackgroundWorker bw = new BackgroundWorker();
static CameraWorker cWorker;
static Thread cThread;
WriteableBitmap wbm;
public IntPtr wbmBackBuffer;
const int FrameSize = 1944*2592;
// CameraWorker Event Handler
void CW_FrameUpdated(object sender, CameraWorkerEventArgs e)
{
if (!e.Updated) return;
// e.pData is an IntPtr containing the camera frame data
CopyMemory(this.wbmBackBuffer, e.pData, FrameSize);
this.Dispatcher.Invoke(wbm.Lock);
this.Dispatcher.Invoke(()=>{ wbm.AddDirtyRect(
new Int32Rect(0,0,wbm.PixelWidth,wbm.PixelHeight)); });
this.Dispatcher.Invoke(wbm.Unlock);
// The above works and I get streaming data to my view port.
// Now I want to make a copy of the pixel data to send to another thread
// for processing. This is where I am having trouble.
if (bw.IsBusy) return;
CopyMemory(pFocusData, e.pData, FrameSize);
var args = new List<object>();
args.Add(pFocusData);
bw.RunWorkerAsync(args);
}
// BackgroundWorker event handlers
void bw_DoWork(object sender, DoWorkEventArgs e)
{
// This is where I see the result of the problem when debugging.
List<object> argu = e.Argument as List<object>;
var pData = (IntPtr) argu[0];
var fullFrame = new byte[FrameSize];
Marshal.Copy(pData,fullFrame,0,FrameSize);
// Perform operations on the byte array data.
// I extract a subregion from the byte array to process, however after a
// couple of iterations, all values in fullFrame equal 255. The pData that
// is coming in should be a copy of the pixel data that is being displayed
// on the screen. While the screen keeps updating with a live video image,
// the frameData variable appears to keep accumulating rather than resetting
// with each backgroundworker call.
}
void bw_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
{
// Update UI elements using Dispatcher with data obtained during DoWork.
}
// Window event handlers
private void MainWindow_Initialized(object sender, EventArgs e)
{
// Set up the WBM
wbm = new WriteableBitmap(width,height,96d,96d,PixelFormats.Gray8,null);
this.wbmBackBuffer = wbm.BackBuffer;
// Set up the camera to grab frames in another thread
cWorker = new CameraWorker(camera);
cWorker.CameraFrameUpdated += CW_FrameUpdated;
cThread = new Thread(new ThreadStart(cWorker.ThreadRun));
cThread.Start();
while(!cThread.IsAlive);
// Set up the background worker for processing image data
bw.DoWork += bw_DoWork;
bw.RunWorkerCompleted += bw_RunWorkerCompleted;
// Bind the image data to the Image object that has the name "viewer"
viewer.Source = wbm;
}
private void MainWindow_Closing(object sender,
System.ComponentModel.CancelEventArgs e)
{
Marshal.FreeHGlobal(pData);
}
}
EDIT: I corrected the typo that Erti-Chris Eelmaa pointed out. It was just a transcription error on my part in showing the relevant portion of my code.
EDIT #2:
1) What happens if you do your BW stuff after if (!e.Updated) return; line? Will the wbm start having this accumulation error and your BW will be fine?
There is no difference in behavior. The wbm is still fine and my BW variable accumulates.
2) BackgroundWorker.IsBusy property will be false after bw_DoWork is finished. Are you okay with this behavior?
My intent was to only process a new frame when the BW was finished. I thought that IsBusy lasts until RunWorkerCompleted is run? I don't need to process every frame.
I tried to simply create a new BW for every call to CW_FrameUpdated, but that did not solve the problem either.
3) What does the MoveMemory do? Does it copy memory from SOURCE to DESTINATION without changing anything in SOURCE? Does it even copy anything?
RtlMoveMemory (CopyMemory) is supposed to copy bytes from one area of memory to the other. I thought that by having allocated two separate spaces of equal size by the AllocHGlobal function I could do a rapid copy of the 8MB of data from one variable to the other. From what I have been reading it appears that I am doing something that the managed memory does not like. I need a fast deep copy of the data. I will try System.Buffer.BlockCopy again in case I missed something the first time.
4) What buffer variable are you speaking about? On every stage, verify ALL BUFFERS and nail down the EXACT place where the buffers differ. You need to create function DumpMemory(IntPtr unmanagedMemory). You can cast IntPtr to byte and use fixed() statement to do so.*
I will set up the DumpMemory function and let you know what I find. In the meantime, here is the data flow and roster of relevant variables.
pData (IntPtr): unmanaged memory containing 1944*2592 bytes of image data in Gray8 format with no header
wbm (WriteableBitmap): WPF variable that is bound to an Image object on the main window
wbmBackBuffer (IntPtr): local variable that points to the same location as the wbm.BackBuffer
pData is copied to wbmBackBuffer using CopyMemory, and because wbm is bound to the Image object, the current image frame is updated on the main window
pFocusData (IntPtr): this is a local pointer that is allocated memory in the size of a full frame of data
pData is copied to pFocusData through CopyMemory.
fullFrame (byte []): this is a byte array copy of pFocusData. This is where I see the accumulation occur.
EDIT #3: I finally solved the issue. It turns out there was a logic error in the sub-arrays I was selecting. My view port is about 800*400 while the image array is about 3.5 times larger. I did not scale the coordinates properly, so my sample regions were all looking at a small and similar region of the frame data. I was able to notice this by following the 4th suggestion and using a memory dump to see exactly what was going on at each step. It helped me see that nothing was wrong with the code ultimately and what I thought was accumulation was actually just the camera saturating in a small region.
The good news is that the code posted above is correct and works. I ultimately ended up using Buffer.BlockCopy and created a duplicate frame that I pass through the camera worker event args.
Thank you very much for your help Erti-Chris Eelmaa!

You're getting tired, it's time for a Christmas now :P
if (bw.IsBusy) return;
// replace this.wbmBackBuffer with pFocusData
CopyMemory(this.wbmBackBuffer, e.pData, FrameSize);
var args = new List<object>();
args.Add(pFocusData);
bw.RunWorkerAsync(args);
// edit, so looking at your code, I have few questions / suggestions.
1) What happens if you do your BW stuff after if (!e.Updated) return; line? Will the wbm start having this accumulation error and your BW will be fine?
2) BackgroundWorker.IsBusy property will be false after bw_DoWork is finished. Are you okay with this behavior?
3) What does the MoveMemory do? Does it copy memory from SOURCE to DESTINATION without changing anything in SOURCE? Does it even copy anything?
4) What buffer variable are you speaking about? On every stage, verify ALL BUFFERS and nail down the EXACT place where the buffers differ. You need to create function DumpMemory(IntPtr unmanagedMemory). You can cast IntPtr to byte* and use fixed() statement to do so.

how to preload images in a background thread?

In my WPF app i need to load some images. I only need to display one image at a time. If i load the image when it's needed, there is a slightly delay. So i thought to myself: "Hey, why not do some preloading in a background thread? Can't be that hard." I have some experience with threads, but not enough to know that this thought was wrong. I started programming and run into some problems. I fixed some of the problems and i probably could fix the other problems too, but that would result in spaghetti code. So, I think starting from the scratch would be the best. What initial planing is needed to build a nice and little preloading thread? Is there a pattern or something like that?
Here's my current setup:
LinkedList<string> to stores pathes to the pictures and navigate to the next picture
Dictionary<string, BitmapImage> to store the preloaded images

I'd use something like this:
class ImageManager
{
private Dictionary<string, Image> images=
new Dictionary<string,Image>();
public Image get(string s) { // blocking call, returns the image
return load(s);
}
private Image load(string s) { // internal, thread-safe helper
lock(images) {
if(!images.ContainsKey(s)) {
Image img=// load the image s
images.Add(s,img);
return img;
}
return images[s];
}
}
public void preload(params string[] imgs) { // non-blocking preloading call
foreach(string img in imgs) {
BackgroundWorker bw=new BackgroundWorker();
bw.DoWork+=(s,e)=>{ load(img); } // discard the actual image return
bw.RunWorkerAsync();
}
}
}
// in your main function
{
ImageManager im=new ImageManager();
im.preload("path1", "path2", "path3", "path4"); // non-blocking call
// then you just request images based on their path
// they'll become available as they are loaded
// or if you request an image before it's queued to be loaded asynchronously
// it will get loaded synchronously instead, thus with priority because it's needed
}

That certainly sounds like a good use for a background thread. Also, as your unit of work is quite large, there shouldn't be too much contention for the synchronization on your collections. You might find examples of similar algorithms, but I think you'll have to roll your own implementation - it's not that complicated.
One thing springs to mind, though: you will either have to keep a record of which images are currently in the process of being loaded, or tolerate multiple loads of the same image.
For example, if your UI requires an image that has not yet been loaded, you will probably want to load that image as a priority. If you know that the background thread is in the process of loading that image, you could just wait for it to become available. If instead you decide to just do the load on the UI thread, there is a possibility that the background thread will try to add a loaded image that is already present.
So, there will have to be some synchronization, but it shouldn't be too complicated.
Nick

Marcel,
WPF provides us already with great mechanisms of BackgroundWorker and Dispatcher to make you forget about writing your own thread mechanisms. However your problem doesn't seem to be that obvious for me. How many images do you need/where do you get these from? Please give us more info.

I was looking in to this yesterday and couldn't find much on the subject. There's actually quite a simple solution to the problem. Use WebClient to load the images asynchronously into a stream and then add that stream to a BitmapImage. Bellow is an example of how I implemented the preloading of a list of images. the example uses the Reactive Extensions Library(Rx) but it can easily be implemented in a non Rx way (Rx makes the code a lot more succinct and hides a lot of state).
public IEnumerable<BitmapImage> BitmapImages { get; private set }
private void PreloadImages(IEnumerbale<Uri> uriCollection)
{
var bitmapImages= new List<BitmapImage>();
uriCollection.ToObservable()
.SelectMany(LoadImageAsync)
.Catch(Observable.Empty<BitmapImage>())
.Subscribe(bitmapImages.Add,
() =>
{
BitmapImages = bitmapImages;
});
}
private IObservable<BitmapImage> LoadImageAsync(Uri uri)
{
return Observable.CreateWithDisposable<BitmapImage>(observer =>
{
var downloader = new WebClient();
downloader.OpenReadCompleted += (s, e) =>
{
if (e.Error != null)
{
observer.OnError(e.Error);
}
var bitmapImage = new BitmapImage();
bitmapImage.BeginInit();
bitmapImage.StreamSource = e.Result;
bitmapImage.EndInit();
observer.OnNext(bitmapImage);
observer.OnCompleted();
};
downloader.OpenReadAsync(uri);
return downloader;
});
}