I am looking for some help with my MultiSourceFrameArrived event in the Kinect v2 SDK.
The following is the method in question:
private async void _reader_MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
{
MultiSourceFrame multiSourceFrame = e.FrameReference.AcquireFrame();
using (var colorFrame = multiSourceFrame.ColorFrameReference.AcquireFrame())
{
if (colorFrame != null)
{
_writeableBitmap.Lock();
colorFrame.CopyConvertedFrameDataToIntPtr(
_writeableBitmap.BackBuffer,
(uint)(_colorFrameDescription.Width * _colorFrameDescription.Height * _colorFrameDescription.BytesPerPixel),
ColorImageFormat.Bgra);
_writeableBitmap.AddDirtyRect(new Int32Rect(0, 0, _writeableBitmap.PixelWidth, _writeableBitmap.PixelHeight));
_writeableBitmap.Unlock();
reflectionOverlayControl.ReflectionImageSource = _writeableBitmap;
}
}
using (var bodyFrame = multiSourceFrame.BodyFrameReference.AcquireFrame())
{
if (bodyFrame != null)
{
Body body = JointHelpers.FindClosestBody(bodyFrame);
if (body != null)
{
if (body.IsTracked)
{
Dictionary<BodyComponentType, BodyComponent> bodyComponentDictionary = BuildBodyComponentDictionary(body);
foreach (BodyComponent bodyComponent in bodyComponentDictionary.Values.OrderBy(x => x.BodyComponentType))
{
bodyComponent.Generate(_writeableBitmap, _coordinateMapper, FrameType.Color, 25);
if (!_isRunningFiltering)
{
_isRunningFiltering = true;
try
{
await Task.Run(() =>
{
bodyComponent.RunFunFiltering();
});
}
finally
{
_isRunningFiltering = false;
}
}
}
reflectionOverlayControl.UpdateValues(
bodyComponentDictionary,
GetFullBodyComponent(body));
}
}
}
}
}
Now, allow me to explain:
The method runs when a particular kind of frame arrives from the Kinect, this is acquired and I can extract the ColorFrame and BodyFrame out of it in the using blocks.
The first "using" block turns the ColorFrame into a WPF WriteableBitmap (declared in the constructor) and sets a user control's ReflectionImageSource set equal to this WriteableBitmap. If this were the only using block, I would see a very smooth feed on the screen!
The second BodyFrame using determines the closest body, if it is tracked and then creates a Dictionary populated with a persons BodyComponents (hands, feet, head etc.)
The foreach loop here runs the "Generate" function on each BodyComponent, which sets a few of it's properties. For example, it sets an EncompassingRectangle property which is an Int32Rect object designed to encompass the component.
The next bit is where I need help!
The method RunFunFiltering is a heavily intensive processing method which, when run, would create a blocking statement that freezes up my UI. This would have the effect of making my color frame video feed very jumpy! This RunFunFiltering method needs to set some of the BodyComponent class's properties, such as the colour that the rectangle should be displayed, the number of white pixels in it's ReflectionImageSource and to set another writeable bitmap with the part of the first ReflectionImageSource which is contained in the rectangle.
Since this object is now complete, with all properties set (and this has been done for each of the BodyComponent's in the dictionary) I run an UpdateValues method on the view, which displays the interesting stuff in the BodyComponent class on the screen for me.
Following some advice from #sstan in this post: Async Await to Keep Event Firing
I threw in a Task.Run() block. However, this doesn't seem to be releasing my UI and I still see a jumpy image. The weird thing is in that timer example, that it works perfectly! I'm at a bit of a loss here to know what to do.
I'm a bit of a beginner with asynchronous functions but I would really like to understand your solutions. If you can provide an explanation with your code I'd be extremely grateful!
Update
I have been able to identify that the using statement which acquires the frame blocks the UI when it is placed outside of the Task.Run call.
I can't just make the whole BodyFrame using block run asynchronously because I need the first "Generate" function to always happen and not be part of the heavy processing thread. Two using blocks seems inelegant and is rather pushing my question under the carpet...
From your comment I understand the following:
You have an async function that is called when a frame arrives
If no RunFunFiltering task is running start one
If such a task is running, don't start a new one
If RunFunFiltering is finished Process the result
.
Task taskFunFiltering = null;
private async Task ProcessFrame(...)
{ // a new frame is arrived
DoSomeProcessing(...);
// only start a new run fun filtering if previous one is finished
if (taskFunFiltering == null || taskFunFiltering.IsCompleted)
{ // start a new fun filtering
// don't wait for the result
taskFunFiltering = Task.Run( () => ...);
}
}
private async Task RunFunFiltering(...)
{
// do the filtering and wait until finished
var filterResult = await DoFiltering(...);
DisplayResult(filterResult);
}
Related
I'm making a system to balance calls inside of the OnGUI method of a EditorWindow.
I'm doing the following:
public void Update()
{
Repaint();
}
Inside of my OnGUI method I'm calling this Balancer. I have one list with the callbacks (List).
So the idea is simple, some callvaxc
I'm skipping some repaint frames for the callback that has the complete GUI, and calling on each repaint for other callbacks (for example, a marquee label or dispalying gifs).
By some reason, this error happens "Getting control 0's position in a group with only 0 controls when doing repaint"
private int m_repaintCounter;
public void Draw()
{
Event e = Event.current;
try
{
foreach (var action in m_actions)
{
try
{
// Test 1
// MainAction is a class that inherits from Action (class MainAction : Action)
if (action is MainAction)
{
bool isDesignedType = e.rawType == EventType.Repaint || e.rawType == EventType.Layout;
if (isDesignedType)
++m_repaintCounter;
if (!(m_repaintCounter == 20 && isDesignedType))
continue;
else
m_repaintCounter = 0;
}
// Test 2
action.Value();
}
catch (Exception ex)
{
Debug.LogException(ex);
}
}
}
catch
{
// Due to recompile the collection will modified, so we need to avoid the exception
}
}
But if I comment the "Test 1" everythings works fine.
On the ctor of the class we need to specify a callback to a GUI method, for example:
public Balancer(Action drawAction)
{
m_actions = new List<Action>();
m_actions.Add(drawAction);
}
So we could do easily (inside the EditorWindow):
private Balancer m_balancer;
public void OnEnable()
{
m_balancer = new Balancer(Draw);
}
public void Draw()
{
// This block will be called every 20 repaints as specified on the if statment
GUILayout.BeginHorizontal("box");
{
GUILayout.Button("I'm the first button");
GUILayout.Button("I'm to the right");
// This marquee will be called on each repaint
m_balancer.AddAction(() => CustomClass.DisplayMarquee("example"));
}
GUILayout.EndHorizontal();
}
// Inside of the Balancer class we have
// We use System.Linq.Expressions to identify actions that were added already
private HashSet<string> m_alreadyAddedActions = new HashSet<string>();
public void AddAction(Expression<Action> callback)
{
if(!m_alreadyAddedActions.Add(callback.ToString()))
return;
m_actions.Add(callback.Compile());
}
I can't figure this out. I couldn't find any information on the internet. Can anyone help me?
Ok, so, OnGui (IMGui) is awful to work with. If you aren't using it for an editor script, use the new 4.6 UI (UGui) instead.
Now then. The problem.
OnGui is called at least twice every frame. One of those is to calculate layouts and the other is to actually draw stuff ("repaint").
If the number of things, size of things, or anything else changes between these two calls then Unity will error with "Getting control 0's position in a group with only 0 controls when doing repaint."
That is: you cannot change UI state in the IMGui system at any point after Layout and before Repaint.
Only, only, only change state (and thereby which objects are being drawn) during Event.current == EventType.Repaint and only, only, only change state for the next frame (alternatively, do the changes during Event.current == EventType.Layout, provided that this same, new state will result in the same code path during Repaint). Do not, under any circumstances, make changes during Repaint that were not present during the previous Layout.
PROBLEM
I am using a secondary view to run my media files, but When I close my secondary view with close button on it ( while media is still playing ) the secondary view/window closes but the media somehow keeps playing because I can hear the sound and source of sound seems to be the primary view ( main app window ). how can I completely terminate the secondary window when I close it?
TRIED
I followed windows samples multiple views and was able to complete all steps, I copied the ViewLifetimeControl.cs file from the sample and used it in my project. the code runs fine until it reaches Windows.Current.Close() in released event of the secondary view.
Then it gives an exception when it tries "Window.Current.Close()" with in the released event. according to documentation exception occurs due to any on going changes ( which might be because of media file playing ), but I need to force close the window even when media file is playing how can I do that? btw here is the exception :
Message = "COM object that has been separated from its underlying RCW cannot be used."
Code to Create and Show secondary view
internal static async Task CompactOpen(string Title, string caption)
{
ViewLifetimeControl viewControl = null;
await CoreApplication.CreateNewView().Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
{
viewControl = ViewLifetimeControl.CreateForCurrentView();
viewControl.Title = Title;
viewControl.StartViewInUse();
var frame = new Frame();
frame.MinHeight = 200;
frame.MinWidth = 200;
frame.Navigate(typeof(CompactNowPlayingPage), new object[] { viewControl,caption});
Window.Current.Content = frame;
Window.Current.Activate();
ApplicationView.GetForCurrentView().Title = viewControl.Title;
});
((App)App.Current).SecondaryViews.Add(viewControl);
var selectedView = viewControl;
var sizePreference = new SizePreferenceString() { Title = "SizePreference", Preference = ViewSizePreference.Default };
var anchorSizePreference = new SizePreferenceString() { Title = "AnchorSizePreference", Preference = ViewSizePreference.Default };
if (selectedView != null && sizePreference != null && anchorSizePreference != null)
{
try
{
selectedView.StartViewInUse();
var viewShown = await ApplicationViewSwitcher.TryShowAsStandaloneAsync(
selectedView.Id,
sizePreference.Preference,
ApplicationView.GetForCurrentView().Id,
anchorSizePreference.Preference);
if (!viewShown)
{
// The window wasn't actually shown, so release the reference to it
// This may trigger the window to be destroyed
}
// Signal that switching has completed and let the view close
selectedView.StopViewInUse();
}
catch (InvalidOperationException)
{
// The view could be in the process of closing, and
// this thread just hasn't updated. As part of being closed,
// this thread will be informed to clean up its list of
// views (see SecondaryViewPage.xaml.cs)
}
}
}
Released Event
private async void ViewLifetimeControl_Released(Object sender, EventArgs e)
{
((ViewLifetimeControl)sender).Released -= ViewLifetimeControl_Released;
// The ViewLifetimeControl object is bound to UI elements on the main thread
// So, the object must be removed from that thread
await mainDispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
{
((App)App.Current).SecondaryViews.Remove(thisViewControl);
});
// The released event is fired on the thread of the window
// it pertains to.
//
// It's important to make sure no work is scheduled on this thread
// after it starts to close (no data binding changes, no changes to
// XAML, creating new objects in destructors, etc.) since
// that will throw exceptions
Window.Current.Close(); //this is where that exception occurs
}
Note : both of above methods and even all the related variables, all of them I have followed the guidelines within the uwp sample for multiple views.
Thanks in advance, any help would be really appreciated, I only want to force close the secondary view ( If that's possible )
Is this in the editor or the app? If it's in your debug or build of the app, the secondary view is most likely still open but hidden. You may be using a custom close button which doesn't perform its job well enough. Instead of putting down SecondaryViews.Remove you should do what you had originally written and try StopViewInUse. It may not work, I'm not used to this kind of thing.
I have a structural problem resulting in an System.ObjectDisposedException with a dll.
More specifically it's a µEye Camera driver that is advised to capture data from a camera. I got two events from this camera that are fired asynchronously:
OnFrameEvent
OnSequenceEvent
The latter tells me that my capture is beeing completed and I can continue in saving my images.
This is the code that does the work:
private void onSequenceEvent(object sender, EventArgs e)
{
uEye.Camera Camera = sender as uEye.Camera;
SequenceCount++;
Camera.Acquisition.Stop();
int s32SeqID;
statusRet = Camera.Memory.Sequence.GetLast(out s32SeqID);
Invoke((MethodInvoker)delegate ()
{
lblStatus.Text = "Save Images...";
this.pbCapture.Value = 0;
});
Rectangle src = new Rectangle();
Rectangle dst = new Rectangle();
src.X = AOI_Size.X;
src.Y = AOI_Size.Y;
src.Width = AOI_Size.Width;
src.Height = AOI_Size.Height;
dst.X = 0; dst.Y = 0; dst.Width = AOI_Size.Width; dst.Height = AOI_Size.Height;
Bitmap bitmap_source = new Bitmap(MySensorInfo.MaxSize.Width, MySensorInfo.MaxSize.Height);;
Bitmap bitmap_destination = new Bitmap(dst.Width, dst.Height);;
Graphics g = Graphics.FromImage(bitmap_destination);
for (int i = 1; i < s32SeqID; i++)
{
Camera.Memory.ToBitmap(i, out bitmap_source);
g.DrawImage(bitmap_source, dst, src, GraphicsUnit.Pixel);
bitmap_destination.Save(PathToSave + i.ToString() + ".bmp");
this?.Invoke((MethodInvoker)delegate ()
{
pbOverallProgress.Value++;
pbCapture.Value++;
});
}
bitmap_source.Dispose();
g.Dispose();
this.CloseForm?.Invoke(1);
}
However - as you may expect the Invoke call will cause the ObjectDisposedException as the "Cancel"-Button is beeing pressed:
this?.Invoke((MethodInvoker)delegate ()
{
pbOverallProgress.Value++;
pbCapture.Value++;
});
Here's the code of the Cancel-Button:
private void btn_Exit_Click(object sender, EventArgs e)
{
if (MessageBox.Show("Do you really want to cancel?", "Abort", MessageBoxButtons.YesNo, MessageBoxIcon.Question) == DialogResult.Yes)
{
this.CloseForm?.Invoke(0);
}
}
private void UEye_Dialog_Form_CloseForm(int exitCode)
{
this?.Invoke((MethodInvoker)delegate ()
{
if (Camera != null)
{
Camera.EventFrame -= onFrameEvent;
Camera.EventSequence -= onSequenceEvent;
}
Camera?.Acquisition.Stop();
Camera = null;
ReturnCode = exitCode;
this.Close();
});
}
UEye_Dialog_Form_CloseForm(int exitCode) is a event where CloseForm is it's delegate.
I hope this was not too much information :)
This exception only occure if the image is beeing saved, not if I'm waiting for the sequence to be captured.
Sure I could pack the UI Update code within a try-catch-block or check if the forms state is Disposed/Disposing. But for my little programming skills it looks like a structural problem.
Thank you for your help :)
The tricky part is that you're doing multi-threading without synchronization.
Invoke presents one such synchronization point, which is fine. However, as you've found out, it doesn't work after the handle is disposed - this makes perfect sense; Invoke simply sends a window message to the given handle, and when the handle no longer exists there's noöne to deal with the message (not to mention what would Close (which just sends WM_CLOSE etc.) do when the window no longer exists).
Solving this is quite tricky, actually. Even if you check if the form is disposed before trying the Invoke, it might still be disposed between the check and the Invoke itself. locking would allow you to handle the synchronization, but you'd need to lock in DestroyHandle or perhaps an event like Closing - the key is to make sure that whatever signal you send is safely tied to whether Invoke is safe right now. Of course, you don't really want to use Invoke - you need BeginInvoke, otherwise you're guaranteed to have a deadlock when Invoke needs to wait for the UI thread which is currently waiting for the lock to be released. Not good :)
Being prepared for an ObjectDisposedException might be the best solution. But looking through the reference source code, it doesn't look like it's 100% correct either - it's consistent on a single thread, but you're not calling Invoke on the UI thread (obviously).
I have a ListBox, where my SelectedValue is set to a class DefaultStrediska which has IEditableObject implemented. What I am doing every time user selects a new item under this particular ListBox (SelectedValue changes), I first check if any change has been made, and if yes; then I ask user if he wants to save temporary changes (otherwise I discard them and return back to the original values).
I am using Mahapps.Metro async method for displaying a message (rather than using traditional System.Windows.MessageBox) and getting the result. The problem is, that this is an asynchronous method that I have to call from my property. Here it is how I do it:
private async Task<bool> GetResult()
{
if (await Window.ShowMessageAsync("Zmena v údajoch", "Pozor! Nastala zmena v údajoch. Prajete si ich dočasne uložiť zmeny?", MessageDialogStyle.AffirmativeAndNegative) == MessageDialogResult.Affirmative)
_SelectedStredisko.EndEdit();
return true;
}
private DefaultStrediska _SelectedStredisko;
public DefaultStrediska SelectedStredisko
{
get { return _SelectedStredisko; }
set
{
//check if any changes have been made
if (value != null && _SelectedStredisko != null)
{
if (_SelectedStredisko.WasChangeMade())
{
var x = GetResult().Result;
}
}
_SelectedStredisko = value;
//create backup of current data
_SelectedStredisko.BeginEdit();
OnPropertyChanged("SelectedStredisko");
}
}
However the problem is, that now my var x = GetResult().Result completely blocks the UI thread and I neither get the messagebox, nor can do anything else. If I remove .Result, then the code first goes to _SelectedStredisko = value and only afterwards calls the GetResult() method, which is unacceptable.
What am I doing wrong in here?
There are a number of ways to avoid the deadlock, I go through a few of them here. I think in your case it might be best to use ConfigureAwait(false) when you are showing the message, but I haven't used that API myself.
await Window.ShowMessageAsync(..).ConfigureAwait(false)
I currently have an application I'm writing in c# (using .NET) that requires me to start a timer as soon as a user sees an image on screen up until they respond with a key press.
Now I realise that practically this is very difficult given the monitor input lag and response time, the time the keyboard takes to physically send the message, the OS to process it, etc.
But I'm trying my best to reduce it down to mostly a constant error (the response time results will be used to compare one user to the next so a constant error isn't really an issue). However annoying hurdle is the variable caused by the monitor refresh rate, as I gather when my onPaint message is called and done with, it doesn't mean the image has actually been processed and sent from the graphics buffer?
Unfortunately time restrictions and other commitments would realistically restrict me to continuing this task in c# for windows.
So what I was wondering was if either handling all the drawing in OpenGL or DirectX or better still for me if it is possible to just using either OpenGL or DirectX to create an event when the screen is updated?
Another suggestion given to me previously was regarding V-Sync, if I switch this off is the image sent as soon as it is drawn? as opposed to sending images at a set rate synchronised to the monitor refresh rate?
You must render your graphic in a separate thread in order to:
Use vertical synchronisation to have a precise timing of the effective display of your image.
Get the precise timing of your user input (since user interface is not on the same thread than the render loop.
Initialise Direct3D to enable the VSync during render :
// DirectX example
presentParams.SwapEffect = SwapEffect.Discard;
presentParams.BackBufferCount = 1;
presentParams.PresentationInterval = PresentInterval.One;
device = new Device(...
Perform the render in a separate thread:
Thread renderThread = new Thread(RenderLoop);
renderThread.Start();
shouldDisplayImageEvent = new AutoResetEvent();
Then use the following render loop:
void RenderLoop()
{
while(applicationActive)
{
device.BeginScene();
// Other rendering task
if (shouldDisplayImageEvent.WaitOne(0))
{
// Render image
// ...
userResponseStopwatch = new Stopwatch();
userResponseStopwatch.Start();
}
device.EndScene();
device.Present();
}
}
Then handle the user input :
void OnUserInput(object sender, EventArgs e)
{
if (userResponseStopwatch != null)
{
userResponseStopwatch.Stop();
float userResponseDuration = userResponseStopwatch.ElapsedMillisecond - 1000 / device.DisplayMode.RefreshRate - displayDeviceDelayConstant;
userResponseStopwatch = null;
}
}
You now use the shouldDisplayImageEvent.Set() event trigger to display the image as needed and start the stop watch.
First enable the VSync on your application idle loop :
// DirectX example
presentParams.SwapEffect = SwapEffect.Discard;
presentParams.BackBufferCount = 1;
presentParams.PresentationInterval = PresentInterval.One;
device = new Device(...
Application.Idle += new EventHandler(OnApplicationIdle);
// More on this here : http://blogs.msdn.com/tmiller/archive/2005/05/05/415008.aspx
internal void OnApplicationIdle(object sender, EventArgs e)
{
Msg msg = new Msg();
while (true)
{
if (PeekMessage(out msg, IntPtr.Zero, 0, 0, 0))
break;
}
// Clearing render
// ...
if (displayImage)
{
// Render image
// ...
renderTime = DateTime.now();
}
device.Present();
}
With the vsync enabled, the device.Present function block until the next frame synchronisation, so if you compute the time between renderTime and the user input time and remove the display device delay + 16.67ms you should get your user response delay.