Need SlimDX to display images quickly - c#

I need to display images very quickly (about 60 FPS). Picturebox/Panel doesn't do the job at higher resolutions, so I am now turning to SlimDX, which I hope is the right move.
As SlimDX uses Directx, and DirectX uses the GPU, I should be able to do it very quickly. From my understanding, the GPU works with images a lot faster than with the CPU.
I am doing this:
MessagePump.Run(form, () =>
{
device.BeginScene();
sprite.Begin(SlimDX.Direct3D9.SpriteFlags.None);
tx = SlimDX.Direct3D9.Texture.FromStream(device, (new MemoryStream(reader.ReadBytes(reader.ReadInt32()))), SlimDX.Direct3D9.Usage.None, SlimDX.Direct3D9.Pool.Managed);
sprite.Draw(tx, Color.Transparent);
sprite.End();
device.EndScene();
device.Present();
});
And initializing everything:
var form = new RenderForm("Test");
form.Width = 1280;
form.Height = 720;
SlimDX.Direct3D9.PresentParameters presentParams = new SlimDX.Direct3D9.PresentParameters
{
BackBufferWidth = form.Width,
BackBufferHeight = form.Height,
DeviceWindowHandle = form.Handle,
PresentFlags = SlimDX.Direct3D9.PresentFlags.None,
BackBufferCount = 0,
PresentationInterval = SlimDX.Direct3D9.PresentInterval.Immediate,
SwapEffect = SlimDX.Direct3D9.SwapEffect.Discard,
BackBufferFormat = SlimDX.Direct3D9.Format.A8R8G8B8,
Windowed = true,
};
device = new SlimDX.Direct3D9.Device(new SlimDX.Direct3D9.Direct3D(), 0, SlimDX.Direct3D9.DeviceType.Hardware, form.Handle, SlimDX.Direct3D9.CreateFlags.HardwareVertexProcessing, presentParams);
device.Viewport = new SlimDX.Direct3D9.Viewport(0, 0, form.Width, form.Height);
SlimDX.Direct3D9.Sprite sprite;
SlimDX.Direct3D9.Texture tx;
sprite = new SlimDX.Direct3D9.Sprite(device);
tx = new SlimDX.Direct3D9.Texture(device, form.Width, form.Height, 0, SlimDX.Direct3D9.Usage.None, SlimDX.Direct3D9.Format.X8R8G8B8, SlimDX.Direct3D9.Pool.Managed);
There are two problems with this:
it's extremely slow; picturebox is much faster
it shows the image incorrectly (it's zoomed in)


You need to cache the textures, as making them every frame is slow. E.g.
public class CacheObjects
{
static List cacheSCBrushes = new List();
public static SolidColorBrush GetColorBrush(Color4 Color, Direct2D.RenderTarget renderTGT)
{
// ERROR: Not supported in C#: OnErrorStatement
SolidColorBrush returnBrush = null;
bool found = false;
foreach (void br_loopVariable in cacheSCBrushes) {
br = br_loopVariable;
if (br.Color.Red == Color.Red) {
if (br.Color.Green == Color.Green) {
if (br.Color.Blue == Color.Blue) {
if (br.Color.Alpha == Color.Alpha) {
found = true;
returnBrush = br;
exit for;
}
}
}
}
}
if (!found) {
returnBrush = new SolidColorBrush(renderTGT, Color);
cacheSCBrushes.Add(returnBrush);
}
return returnBrush;
}
public static void ClearCache()
{
foreach (void Brush_loopVariable in cacheSCBrushes) {
Brush = Brush_loopVariable;
Brush.Dispose();
}
}
}
If that is impossible, [In a seperate thread], load the next 5 textures...Rendering is fast, object creation is slow.
public void LoadTexture () {
//Load the next 5 here
}

Related

How to draw GIFs better in DrawingVisual?

I am trying to draw giftu in DrawingVisual.
Here are some codes that I have tried,But the performance is very poor.
Used up to 22% of cpu (AMD Ryzen 7 5800H) and 50% of gpu(rtx 3060) and 4GB of RAM when drawing a 1924*934, 434 fps gif.
And it produces a very strange trailing shadow:
Screenshots
public void InitImage(string path)
{
isAnima = false;
uriBitmap = BitmapDecoder.Create(
new Uri(path, UriKind.Relative),
BitmapCreateOptions.None,
BitmapCacheOption.Default);
if (uriBitmap.Frames.Count > 1)
{
isAnima = true;
//Get the information of each frame of the gif here
frameInfos = new List<FrameInfo>();
for (int i = 0; i < uriBitmap.Frames.Count; i++)
{
frameInfos.Add(GetFrameInfo(uriBitmap.Frames[i]));
}
frameIndex = 0;
try
{
if (animationThread == null)
{
animationThread = new Thread(ChangeGifFrame);
animationThread.Start();
}
}
catch (Exception)
{
Debug.WriteLine("线程关闭");
}
}
}
private void DrawImage(Point location, Size size)
{
var drawing = this.dvc.drawingVisual.RenderOpen();
drawing.PushTransform(rotate);
if (!isAnima)
{
drawing.DrawImage(baseSource, new Rect(location, size));
}
else
{
//Since the FrameDisposalMethod is Combine, I need to draw all the frames before this frame
for (int i = 0; i < frameIndex; i++)
{
var frame = uriBitmap.Frames[i];
var info = frameInfos[i];
drawing.DrawImage(frame, new Rect(new Point(location.X + info.Left * zoom, location.Y + info.Top * zoom), new Size(info.Width * zoom, info.Height * zoom)));
}
}
drawing.Close();
}
public void ChangeGifFrame()
{
while (true)
{
if (isAnima)
{
this.Dispatcher.Invoke(new Action(() =>
{
frameIndex++;
if (frameIndex >= uriBitmap.Frames.Count)
{
frameIndex = 0;
}
DrawImage(drawPoint, drawSize);
}));
}
Thread.Sleep(30);
}
}
I also tried mixing BitmapSource using the following code, but the performance is worse than drawing directly
public BitmapSource MixBitmapSource(BitmapSource bs1, BitmapSource bs2)
{
DrawingVisual dv = new DrawingVisual();
RenderTargetBitmap render = new RenderTargetBitmap(bs1.PixelWidth, bs2.PixelHeight, bs1.DpiX, bs2.DpiY, PixelFormats.Default);
DrawingContext dc = dv.RenderOpen();
dc.DrawImage(bs1, new Rect(0, 0, bs1.PixelWidth, bs1.PixelHeight));
dc.DrawImage(bs2, new Rect(0, 0, bs1.PixelWidth, bs1.PixelHeight));
dc.Close();
render.Render(dv);
return render;
}
Is there a better way to draw gifs in DrawingVisual?
Thanks for all your help!

UWP - How to create a CompositionSurfaceBrush with a repeated pattern

I would like to have a Panel with a Background that shows a repeated pattern (e.g. dots, evenly separated of 30 pixels).
So far, I managed to create a subclass of XamlCompositionBrushBase that allows we to create my own shape (e.g. a single dot). but I am failing to understand how to repeat this pattern.
This is my custom Brush:
public sealed class DottedBackgroundBrush : XamlCompositionBrushBase
{
public DottedBackgroundBrush()
{
}
protected override void OnConnected()
{
// Delay creating composition resources until they're required.
if (CompositionBrush == null)
{
var compositor = Window.Current.Compositor;
// Actual Width/Height are going to be returned in effective pixels which
// is going to differ from the size of the bitmap that we'll render from the XAML.
var width = 400;
var height = 400;
// Make our visual:
var spriteVisual = compositor.CreateSpriteVisual();
spriteVisual.Size = new Vector2(width, height);
CanvasDevice device = CanvasDevice.GetSharedDevice();
var graphicsDevice = CanvasComposition.CreateCompositionGraphicsDevice(compositor, device);
CompositionSurfaceBrush drawingBrush = compositor.CreateSurfaceBrush();
var drawingSurface = graphicsDevice.CreateDrawingSurface(
new Size(width, height),
DirectXPixelFormat.B8G8R8A8UIntNormalized,
DirectXAlphaMode.Premultiplied);
using (var ds = CanvasComposition.CreateDrawingSession(drawingSurface))
{
ds.Clear(Colors.Transparent);
ds.DrawCircle(new Vector2(10, 10), 5, Colors.Black, 3);
}
drawingBrush.Surface = drawingSurface;
CompositionBrush = drawingBrush;
}
}
protected override void OnDisconnected()
{
// Dispose of composition resources when no longer in use.
if (CompositionBrush != null)
{
CompositionBrush.Dispose();
CompositionBrush = null;
}
}
}
How can I enable the circle to be replicated indefinitely, instead of having as single instance?

For this, you want to create a CompositionEffectBrush as your main brush, using a Win2D BorderEffect - which does the actual tiling - and set it's source to be your SurfaceBrush.
Example (adapted from a repo of mine so it might be a bit roundabouts)
public class TilingBrush : XamlCompositionBrushBase
{
protected Compositor _compositor => Window.Current.Compositor;
protected CompositionBrush _imageBrush = null;
protected IDisposable _surfaceSource = null;
protected override void OnConnected()
{
base.OnConnected();
if (CompositionBrush == null)
{
CreateEffectBrush();
Render();
}
}
protected override void OnDisconnected()
{
base.OnDisconnected();
this.CompositionBrush?.Dispose();
this.CompositionBrush = null;
ClearResources();
}
private void ClearResources()
{
_imageBrush?.Dispose();
_imageBrush = null;
_surfaceSource?.Dispose();
_surfaceSource = null;
}
private void UpdateBrush()
{
if (CompositionBrush != null && _imageBrush != null)
{
((CompositionEffectBrush)CompositionBrush).SetSourceParameter(nameof(BorderEffect.Source), _imageBrush);
}
}
protected ICompositionSurface CreateSurface()
{
double width = 20;
double height = 20;
CanvasDevice device = CanvasDevice.GetSharedDevice();
var graphicsDevice = CanvasComposition.CreateCompositionGraphicsDevice(_compositor, device);
var drawingSurface = graphicsDevice.CreateDrawingSurface(
new Size(width, height),
DirectXPixelFormat.B8G8R8A8UIntNormalized,
DirectXAlphaMode.Premultiplied);
/* Create Drawing Session is not thread safe - only one can ever be active at a time per app */
using (var ds = CanvasComposition.CreateDrawingSession(drawingSurface))
{
ds.Clear(Colors.Transparent);
ds.DrawCircle(new Vector2(10, 10), 5, Colors.Black, 3);
}
return drawingSurface;
}
private void Render()
{
ClearResources();
try
{
var src = CreateSurface();
_surfaceSource = src as IDisposable;
var surfaceBrush = _compositor.CreateSurfaceBrush(src);
surfaceBrush.VerticalAlignmentRatio = 0.0f;
surfaceBrush.HorizontalAlignmentRatio = 0.0f;
surfaceBrush.Stretch = CompositionStretch.None;
_imageBrush = surfaceBrush;
UpdateBrush();
}
catch
{
// no image for you, soz.
}
}
private void CreateEffectBrush()
{
using (var effect = new BorderEffect
{
Name = nameof(BorderEffect),
ExtendY = CanvasEdgeBehavior.Wrap,
ExtendX = CanvasEdgeBehavior.Wrap,
Source = new CompositionEffectSourceParameter(nameof(BorderEffect.Source))
})
using (var _effectFactory = _compositor.CreateEffectFactory(effect))
{
this.CompositionBrush = _effectFactory.CreateBrush();
}
}
}
For the longest time I've meant to add it to the WindowsCommunityToolkit, but for the longest time I've been slamming into bugs with the Visual Layer that stop me. This particular case should work fine however.

Memory Leak Issue. Eye-tracking in Unity with OpenCVSharp

I've been working on this project for a few months now, where I'm trying to integrate eye-tracking into Unity using OpenCVSharp. I've managed to get everything working, including the actual tracking of the pupil etc, however I've got a memory leak. Basically after 20-30seconds of the program running it freezes and the console errors saying "Unable to allocate (insert number here) bits". After looking at the memory usage during running of the program, you can see its use steadily climb until it maxes then crashes.
Now I've spent quite a while trying to fix the issue, and read a lot of help posts about releasing images/storage etc correctly. Despite the fact I'm doing this, it doesn't appear to be releasing them correctly. I tried using the garbage collector to force it to reclaim the memory however that didn't seem to work either. Am I just doing something fundamentally wrong with the images and how I reclaim them? Or is having the creation of new images each frame (even though I'm releasing them) causing the problem.
Any help would be greatly appreciated. Here's the code below, you can ignore a lot of the stuff within the update function as its to do with the actual tracking section and calibration. I realise the code is pretty messy, sorry about that! The main section to worry about is EyeDetection().
using UnityEngine;
using System.Collections;
using System;
using System.IO;
using OpenCvSharp;
using OpenCvSharp.Blob;
//using System.Xml;
//using System.Threading;
//using AForge;
//using OpenCvSharp.Extensions;
//using System.Windows.Media;
//using System.Windows.Media.Imaging;
public class CaptureScript2 : MonoBehaviour
{
//public GameObject planeObj;
public WebCamTexture webcamTexture; //Texture retrieved from the webcam
//public Texture2D texImage; //Texture to apply to plane
public string deviceName;
private int devId = 1;
private int imWidth = 800; //camera width
private int imHeight = 600; //camera height
private string errorMsg = "No errors found!";
private static IplImage camImage; //Ipl image of the converted webcam texture
//private static IplImage yuv;
//private static IplImage dst;
private CvCapture cap; //Current camera capture
//private IplImage eyeLeft;
//private IplImage eyeRight;
//private IplImage eyeLeftFinal;
//private IplImage eyeRightFinal;
private double leftEyeX;
private double leftEyeY;
private double rightEyeX;
private double rightEyeY;
private int calibState;
private double LTRCPx;
private double LTLCPx;
private double LBLCPy;
private double LTLCPy;
private double RTRCPx;
private double RTLCPx;
private double RBLCPy;
private double RTLCPy;
private double gazeWidth;
private double gazeHeight;
private double gazeScaleX;
private double gazeScaleY;
public static CvMemStorage storageFace;
public static CvMemStorage storage;
public static double gazePosX;
public static double gazePosY;
private bool printed = true;
//private CvRect r;
//private IplImage smallImg;
CvColor[] colors = new CvColor[]
{
new CvColor(0,0,255),
new CvColor(0,128,255),
new CvColor(0,255,255),
new CvColor(0,255,0),
new CvColor(255,128,0),
new CvColor(255,255,0),
new CvColor(255,0,0),
new CvColor(255,0,255),
};
//scale for small image
const double Scale = 1.25;
const double scaleEye = 10.0;
const double ScaleFactor = 2.5;
//must show 2 eyes on the screen
const int MinNeighbors = 2;
const int MinNeighborsFace = 1;
// Use this for initialization
void Start ()
{
//Webcam initialisation
WebCamDevice[] devices = WebCamTexture.devices;
Debug.Log ("num:" + devices.Length);
for (int i=0; i<devices.Length; i++)
{
print (devices [i].name);
if (devices [i].name.CompareTo (deviceName) == 1)
{
devId = i;
}
}
if (devId >= 0)
{
//mainImage = new IplImage (imWidth, imHeight, BitDepth.U8, 3);
}
//create capture from current device
cap = Cv.CreateCameraCapture(devId);
//set properties of the capture
Cv.SetCaptureProperty(cap, CaptureProperty.FrameWidth, imWidth);
Cv.SetCaptureProperty(cap, CaptureProperty.FrameHeight, imHeight);
//create window to display capture
//Cv.NamedWindow("Eye tracking", WindowMode.AutoSize);
Cv.NamedWindow ("EyeLeft", WindowMode.AutoSize);
Cv.NamedWindow ("EyeRight", WindowMode.AutoSize);
Cv.NamedWindow ("Face", WindowMode.AutoSize);
calibState = 1;
}
void Update ()
{
if(Input.GetKeyDown(KeyCode.Space) && calibState < 3)
{
calibState++;
}
if(Input.GetMouseButtonDown(0) && calibState < 4)
{
printed = false;
calibState++;
Cv.DestroyAllWindows();
Cv.ReleaseCapture(cap);
cap = Cv.CreateCameraCapture(devId);
}
//if device is connected
if (devId >= 0)
{
//cap = Cv.CreateCameraCapture(devId);
//Cv.Release
//retrieve the current frame from camera
camImage = Cv.QueryFrame(cap);
//detect eyes and apply circles
//
EyeDetection();
Cv.ReleaseImage(camImage);
//PupilTracking();
switch(calibState)
{
case 1:
LTRCPx = leftEyeX;
RTRCPx = rightEyeX;
break;
case 2:
LTLCPx = leftEyeX;
LTLCPy = leftEyeY;
RTLCPx = rightEyeX;
RTLCPy = rightEyeY;
break;
case 3:
LBLCPy = leftEyeY;// + rightEyeY) /2 ;
RBLCPy = rightEyeY;
break;
case 4:
//gazeWidth = (((LTRCPx - LTLCPx) + (RTRCPx - RTLCPx)) / 2) * -1;
//gazeHeight = ((LBLCPy - LTLCPy) + (RBLCPy - RTLCPy)) /2;
gazeWidth = LTLCPx -LTRCPx;
gazeHeight = LBLCPy - LTLCPy;
gazeScaleX = (Screen.width/gazeWidth);
gazeScaleY = Screen.height/gazeHeight;
gazePosX = gazeScaleX *(leftEyeX - LTRCPx);
gazePosY = gazeScaleY *(leftEyeY - LTLCPy);
break;
}
//Cv.ReleaseCapture(cap);
}
else
{
Debug.Log ("Can't find camera!");
}
//print (calibState);
if(printed == false)
{
print ("Gaze pos x = " + gazePosX);
print ("Gaze pos Y = " + gazePosY);
print ("Scale x = " + gazeScaleX);
print ("Scale y = " + gazeScaleY);
print ("Gaze width = " + gazeWidth);
print ("Gaze Height = " + gazeHeight);
print ("left eye x = " + leftEyeX);
print ("left eye Y = " + leftEyeY);
print ("calib state = " + calibState);
printed = true;
}
//Cv.ShowImage("Eye tracking", mainImage);
//Cv.ShowImage ("EyeLeft", grayEyeLeft);
//Cv.ShowImage ("EyeRight", grayEyeRight);
}
void EyeDetection()
{
IplImage mainImage = new IplImage (imWidth, imHeight, BitDepth.U8, 3);
IplImage smallImg = new IplImage(mainImage.Width, mainImage.Height ,BitDepth.U8, 1);
Cv.Resize (camImage, mainImage, Interpolation.Linear);
IplImage gray = new IplImage(mainImage.Size, BitDepth.U8, 1);
Cv.CvtColor (mainImage, gray, ColorConversion.BgrToGray);
Cv.Resize(gray, smallImg, Interpolation.Linear);
Cv.EqualizeHist(smallImg, smallImg);
Cv.ReleaseImage (gray);
//IplImage hack = Cv.LoadImage("\\Users\\User\\Desktop\\Honours Projects\\Project10\\Project\\Assets\\bug.jpeg");
//Cv.Erode (hack, hack);
//Cv.ReleaseImage (hack);
//uint sizeStore = 2877212;
CvHaarClassifierCascade cascadeFace = CvHaarClassifierCascade.FromFile("\\Users\\User\\Documents\\opencv\\sources\\data\\haarcascades\\haarcascade_frontalface_alt2.xml");
CvMemStorage storageFace = new CvMemStorage();
storageFace.Clear ();
CvSeq<CvAvgComp> faces = Cv.HaarDetectObjects(smallImg, cascadeFace, storageFace, ScaleFactor, MinNeighborsFace, 0, new CvSize(30,30));
for(int j = 0; j < faces.Total; j++)
{
CvRect face = faces[j].Value.Rect;
CvHaarClassifierCascade cascadeEye = CvHaarClassifierCascade.FromFile ("\\Users\\User\\Documents\\opencv\\sources\\data\\haarcascades\\haarcascade_eye.xml");
IplImage faceImg = new IplImage(face.Width, face.Height, BitDepth.U8, 1);
IplImage faceImgColour = new IplImage(face.Width, face.Height, BitDepth.U8, 3);
CvMemStorage storage = new CvMemStorage();
storage.Clear ();
Cv.SetImageROI(smallImg, face);
Cv.Copy (smallImg, faceImg);
Cv.ResetImageROI(smallImg);
Cv.SetImageROI(mainImage, face);
Cv.Copy (mainImage, faceImgColour);
Cv.ResetImageROI(mainImage);
Cv.ShowImage ("Face", faceImgColour);
CvSeq<CvAvgComp> eyes = Cv.HaarDetectObjects(faceImg, cascadeEye, storage, ScaleFactor, MinNeighbors, 0, new CvSize(30, 30));
for(int i = 0; i < eyes.Total; i++)
{
CvRect r = eyes[i].Value.Rect;
Cv.SetImageROI(faceImgColour, r);
if(i == 1)
{
IplImage eyeLeft = new IplImage(new CvSize(r.Width, r.Height), BitDepth.U8, 3);
Cv.Copy(faceImgColour, eyeLeft);
IplImage yuv = new IplImage(eyeLeft.Size, BitDepth.U8, 3);
IplImage dst = new IplImage(eyeLeft.Size, BitDepth.U8, 3);
IplImage grayEyeLeft = new IplImage(eyeLeft.Size, BitDepth.U8, 1);
IplImage eyeLeftFinal = new IplImage(Cv.Round(grayEyeLeft.Width * scaleEye), Cv.Round(grayEyeLeft.Height * scaleEye), BitDepth.U8, 1);
Cv.CvtColor(eyeLeft, yuv, ColorConversion.BgrToCrCb);
Cv.Not(yuv, dst);
Cv.CvtColor(dst,eyeLeft,ColorConversion.CrCbToBgr);
Cv.CvtColor(eyeLeft, grayEyeLeft, ColorConversion.BgrToGray);
Cv.Resize (grayEyeLeft, eyeLeftFinal, Interpolation.Linear);
Cv.Threshold(eyeLeftFinal, eyeLeftFinal, 230, 230, ThresholdType.Binary);
CvBlobs b1 = new CvBlobs(eyeLeftFinal);
if(b1.Count > 0)
{
leftEyeX = b1.LargestBlob().Centroid.X;
leftEyeY = b1.LargestBlob().Centroid.Y;
}
Cv.ShowImage ("EyeLeft", eyeLeftFinal);
Cv.ReleaseImage (yuv);
Cv.ReleaseImage (dst);
Cv.ReleaseImage (grayEyeLeft);
Cv.ReleaseImage (eyeLeftFinal);
b1.Clear();
Cv.ReleaseImage (eyeLeft);
}
if(i == 0)
{
IplImage eyeRight = new IplImage(new CvSize(r.Width, r.Height), BitDepth.U8, 3);
Cv.Copy(faceImgColour, eyeRight);
IplImage yuv2 = new IplImage(eyeRight.Size, BitDepth.U8, 3);
IplImage dst2 = new IplImage(eyeRight.Size, BitDepth.U8, 3);
IplImage grayEyeRight = new IplImage(eyeRight.Size, BitDepth.U8, 1);
IplImage eyeRightFinal = new IplImage(Cv.Round(grayEyeRight.Width * scaleEye), Cv.Round(grayEyeRight.Height * scaleEye), BitDepth.U8, 1);
Cv.CvtColor(eyeRight, yuv2, ColorConversion.BgrToCrCb);
Cv.Not(yuv2, dst2);
Cv.CvtColor(dst2,eyeRight,ColorConversion.CrCbToBgr);
Cv.CvtColor(eyeRight, grayEyeRight, ColorConversion.BgrToGray);
Cv.Resize (grayEyeRight, eyeRightFinal, Interpolation.Linear);
Cv.Threshold(eyeRightFinal, eyeRightFinal, 230, 230, ThresholdType.Binary);
CvBlobs b2 = new CvBlobs(eyeRightFinal);
if(b2.Count > 0)
{
rightEyeX = b2.LargestBlob().Centroid.X;
rightEyeY = b2.LargestBlob().Centroid.Y;
}
Cv.ShowImage ("EyeRight", eyeRightFinal);
Cv.ReleaseImage (yuv2);
Cv.ReleaseImage (dst2);
Cv.ReleaseImage (grayEyeRight);
Cv.ReleaseImage (eyeRightFinal);
b2.Clear ();
Cv.ReleaseImage (eyeRight);
}
Cv.ResetImageROI(faceImgColour);
}
//Cv.ShowImage("Eye tracking", mainImage);
Cv.ReleaseImage (faceImg);
Cv.ReleaseImage (faceImgColour);
Cv.ReleaseMemStorage(storage);
Cv.ReleaseHaarClassifierCascade(cascadeEye);
}
Cv.ReleaseMemStorage(storageFace);
Cv.ReleaseHaarClassifierCascade(cascadeFace);
//PupilTracking ();
Cv.ReleaseImage(smallImg);
Cv.ReleaseImage (mainImage);
GC.Collect();
}
void OnGUI ()
{
GUI.Label (new Rect (200, 200, 100, 90), errorMsg);
}
void OnDestroy()
{
Cv.DestroyAllWindows();
Cv.ReleaseCapture(cap);
}

I am not familiar with OpenCV, but as a general rule:
I would limit instantiation in the Update loop, like new CvMemStorage()
Don't load data in the Update loop: CvHaarClassifierCascade.FromFile("\\Users\\User\\Documents\\opencv\\sources\\data\\haarcascades\\haarcascade_frontalface_alt2.xml"); That should be loaded once on start and assigned to a class variable.
Allocate on start and Release only if needed.
I find that in most situations there's plenty of RAM to go around. I allocate on Start() what is going to be used over and over, especially 60 times per second in the Update() loop!
But Loading XML data, Allocating and releasing variables like storage or cascadeEye, is bound to create issues when the app is trying to do so 60 times a second.
Creating and destroying objects is a very, very, very expensive. So do so wisely and sparingly, especially when dealing with complex data structures like the OpenCV objects, bitmaps or loaders.
hth.

SlimDX low performance with many vertices

So I've been playing around with SlimDX for quite a while now,
and experienced some issues with big STL files.
While on OpenGL they load without flinching I get down to 1-2 FPS an soon as I load files of about 100mb (same issues with multiple files) in SharpGL. Did I miss anything, or is there anything I am simply doing not right at all ?
Edit: Just to specify my question: is performance with SlimDX on 1.000.000+ vertices always that poor ?
Edit: I know, that using an index buffer would be more efficient, as well as I know that CullingMode.None isn't really a FPS-Saver, but in the OpenGL test I've even used two sided lighting and a bit of smoothing, which should be as hard as creating (in the worst case) 3 times as many points as necessary.
Edit: Out of curiosity I modified the code to include an indexBuffer, and it really did have some impact on the FPS, I am validating this right now
BasicFramework.cs
#region Using Statements
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Text;
using SlimDX;
using SlimDX.Direct3D11;
using SlimDX.DXGI;
using SlimDX.Windows;
using Device = SlimDX.Direct3D11.Device;
using Texture2D = SlimDX.Direct3D11.Texture2D;
#endregion
namespace SlimDX_Evaluation
{
public abstract class BasicFramework : IDisposable
{
#region Membervariables
//Objects
private RenderForm renderForm;
private SwapChain swapChain;
private Factory factory;
private Device device;
private DeviceContext deviceContext;
private Texture2D backBufffer;
private Texture2D depthBuffer;
private RenderTargetView renderTargetView;
private DepthStencilView depthStencilView;
private TimeSpan lastFrameTime;
private Stopwatch clock;
//Variables
private bool userResized;
private bool isResizing;
#endregion
#region Constructors
/**
* The Constructor initializes the default behavior of the Framework.
* It is not supposed to be replaced, the customization should be done in the Constructor
*/
public BasicFramework() : this("My Title") { }
public BasicFramework(string title)
{
//Create the winForm
renderForm = new RenderForm(title);
renderForm.ClientSize = new System.Drawing.Size(800, 480);
renderForm.MaximizeBox = true;
renderForm.FormBorderStyle = System.Windows.Forms.FormBorderStyle.Sizable;
//Hook into Windows.Forms Event
renderForm.ClientSizeChanged += HandleClientSizeChanged;
//Generate SwapChain
var desc = new SwapChainDescription()
{
BufferCount = 1,
ModeDescription = new ModeDescription(renderForm.ClientSize.Width,
renderForm.ClientSize.Height,
new Rational(60, 1),
Format.B8G8R8A8_UNorm),
IsWindowed = true,
OutputHandle = renderForm.Handle,
SampleDescription = new SampleDescription(1, 0),
SwapEffect = SwapEffect.Discard,
Usage = Usage.RenderTargetOutput,
};
Device.CreateWithSwapChain(
DriverType.Hardware,
DeviceCreationFlags.None,
desc,
out device,
out swapChain
);
//Set DeviceContext
deviceContext = device.ImmediateContext;
// prevent DXGI handling of alt+enter,prt scrn, etc which doesn't work properly with Winforms
using (var factory = swapChain.GetParent<Factory>())
factory.SetWindowAssociation(renderForm.Handle, WindowAssociationFlags.IgnoreAll);
//Generate Backbuffer
backBufffer = Texture2D.FromSwapChain<Texture2D>(swapChain, 0);
renderTargetView = new RenderTargetView(device, backBufffer);
//Generate Depthbuffer and DepthBufferView
depthBuffer = new Texture2D(device, new Texture2DDescription()
{
Format = Format.D16_UNorm,
ArraySize = 1,
MipLevels = 1,
Width = renderForm.ClientSize.Width,
Height = renderForm.ClientSize.Height,
SampleDescription = new SampleDescription(1, 0),
Usage = ResourceUsage.Default,
BindFlags = BindFlags.DepthStencil,
CpuAccessFlags = CpuAccessFlags.None,
OptionFlags = ResourceOptionFlags.None,
});
depthStencilView = new DepthStencilView(device, depthBuffer);
//Define Rasterizer
RasterizerStateDescription rasterizerDescription = new RasterizerStateDescription()
{
CullMode = CullMode.None,
FillMode = FillMode.Solid,
IsAntialiasedLineEnabled = true,
IsFrontCounterclockwise = true,
IsMultisampleEnabled = true,
IsDepthClipEnabled = true,
IsScissorEnabled = false
};
deviceContext.Rasterizer.State = RasterizerState.FromDescription(device, rasterizerDescription);
//Set ViewPort
deviceContext.Rasterizer.SetViewports(new Viewport(
0,
0,
renderForm.Width,
renderForm.Height));
deviceContext.OutputMerger.SetTargets(depthStencilView, renderTargetView);
//Force recalibration on first load
userResized = true;
}
#endregion
#region Run
public void Run()
{
clock = new Stopwatch();
clock.Start();
this.lastFrameTime = clock.Elapsed;
Initialize();
LoadContent();
MessagePump.Run(renderForm, () =>
{
if (userResized)
{
backBufffer.Dispose();
RenderTargetView.Dispose();
depthBuffer.Dispose();
depthStencilView.Dispose();
//Resize the buffers
swapChain.ResizeBuffers(
0,
renderForm.ClientSize.Width,
renderForm.ClientSize.Height,
Format.Unknown,
SwapChainFlags.None
);
//Get the new Backbuffer
backBufffer = Texture2D.FromSwapChain<Texture2D>(swapChain, 0);
//Renew RenderTargetView
renderTargetView = new RenderTargetView(device, backBufffer);
//Create the new DepthBuffer
depthBuffer = new Texture2D(device, new Texture2DDescription()
{
Format = Format.D32_Float_S8X24_UInt,
ArraySize = 1,
MipLevels = 1,
Width = renderForm.ClientSize.Width,
Height = renderForm.ClientSize.Height,
SampleDescription = new SampleDescription(1, 0),
Usage = ResourceUsage.Default,
BindFlags = BindFlags.DepthStencil,
CpuAccessFlags = CpuAccessFlags.None,
OptionFlags = ResourceOptionFlags.None
});
//Create DepthBufferView
depthStencilView = new DepthStencilView(device, depthBuffer);
//SetUp Targets and Viewports for Rendering
deviceContext.Rasterizer.SetViewports(new Viewport(0, 0, renderForm.Width, renderForm.Height));
deviceContext.OutputMerger.SetTargets(depthStencilView, renderTargetView);
//finished resizing
isResizing = userResized = false;
}
TimeSpan timeSinceLastFrame = clock.Elapsed - this.lastFrameTime;
this.lastFrameTime = clock.Elapsed;
Update(clock.Elapsed, timeSinceLastFrame);
BeginFrame();
Draw(clock.Elapsed, timeSinceLastFrame);
EndFrame();
});
UnloadContent();
}
#endregion
#region MethodsToOverride
public virtual void Update(TimeSpan totalRunTime, TimeSpan timeSinceLastFrame)
{
}
public virtual void Draw(TimeSpan totalRunTime, TimeSpan timeSinceLastFrame)
{
}
public virtual void BeginFrame()
{
}
public void EndFrame()
{
swapChain.Present(0, PresentFlags.None); //Presents the image to the user
}
public virtual void Initialize()
{
}
public virtual void LoadContent()
{
}
public virtual void UnloadContent()
{
}
public virtual void Dispose()
{
renderForm.Dispose();
backBufffer.Dispose();
deviceContext.ClearState();
deviceContext.Flush();
device.Dispose();
deviceContext.Dispose();
depthBuffer.Dispose();
depthStencilView.Dispose();
swapChain.Dispose();
}
#endregion
#region Handlers
private void HandleResize(object sender, EventArgs e)
{
backBufffer.Dispose();
RenderTargetView.Dispose();
depthBuffer.Dispose();
depthStencilView.Dispose();
//Resize the buffers
swapChain.ResizeBuffers(
0,
renderForm.ClientSize.Width,
renderForm.ClientSize.Height,
Format.Unknown,
SwapChainFlags.None
);
//Get the new Backbuffer
backBufffer = Texture2D.FromSwapChain<Texture2D>(swapChain, 0);
//Renew RenderTargetView
renderTargetView = new RenderTargetView(device, backBufffer);
//Create the new DepthBuffer
depthBuffer = new Texture2D(device, new Texture2DDescription()
{
Format = Format.D32_Float_S8X24_UInt,
ArraySize = 1,
MipLevels = 1,
Width = renderForm.ClientSize.Width,
Height = renderForm.ClientSize.Height,
SampleDescription = new SampleDescription(1, 0),
Usage = ResourceUsage.Default,
BindFlags = BindFlags.DepthStencil,
CpuAccessFlags = CpuAccessFlags.None,
OptionFlags = ResourceOptionFlags.None
});
//Create DepthBufferView
depthStencilView = new DepthStencilView(device, depthBuffer);
//SetUp Targets and Viewports for Rendering
deviceContext.Rasterizer.SetViewports(new Viewport(0, 0, renderForm.Width, renderForm.Height));
deviceContext.OutputMerger.SetTargets(depthStencilView, renderTargetView);
//finished resizing
isResizing = userResized = false;
TimeSpan timeSinceLastFrame = clock.Elapsed - this.lastFrameTime;
this.lastFrameTime = clock.Elapsed;
Update(clock.Elapsed, timeSinceLastFrame);
BeginFrame();
Draw(clock.Elapsed, timeSinceLastFrame);
EndFrame();
}
private void HandleClientSizeChanged(object sender, EventArgs e)
{
userResized = true;
}
#endregion
#region GetAndSet
public Device Device
{
get
{
return this.device;
}
}
public DeviceContext DeviceContext
{
get
{
return this.deviceContext;
}
}
public RenderTargetView RenderTargetView
{
get
{
return this.renderTargetView;
}
}
public RenderForm RenderForm
{
get
{
return this.renderForm;
}
}
public DepthStencilView DepthStencilView
{
get
{
return this.depthStencilView;
}
}
#endregion
}
}
SimpleIntegration.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Drawing;
using SlimDX;
using SlimDX.D3DCompiler;
using SlimDX.Direct3D11;
using Buffer = SlimDX.Direct3D11.Buffer;
using System.Diagnostics;
namespace SlimDX_Evaluation
{
class SampleIntegration : BasicFramework
{
#region Members
private VertexShader vertexShader;
private PixelShader pixelShader;
private Buffer constantBuffer;
private VertexBufferBinding vertexBufferBinding_model;
private int vertCount;
private Stopwatch timer;
private long lastFrame;
private int frameCount;
private Matrix view;
private Matrix proj;
private Matrix viewProj;
Matrix worldViewProj;
#endregion
public override void Draw(TimeSpan totalRunTime, TimeSpan timeSinceLastFrame)
{
//Output FPS
frameCount++;
if (timer.ElapsedMilliseconds - lastFrame >= 1000)
{
Console.WriteLine("FPS: " + frameCount);
lastFrame = timer.ElapsedMilliseconds;
frameCount = 0;
}
worldViewProj = Matrix.Multiply(Matrix.RotationAxis(Vector3.UnitY, timer.ElapsedMilliseconds / 1000.0f), viewProj);
//Update ConstantBuffer
var buffer = new MyConstantBuffer();
buffer.worldViewProj = worldViewProj;
var data = new DataStream(System.Runtime.InteropServices.Marshal.SizeOf(new MyConstantBuffer()), true, true);
data.Write(buffer);
data.Position = 0;
DeviceContext.UpdateSubresource(new DataBox(0, 0, data),constantBuffer,0);
//Clear
Device.ImmediateContext.ClearRenderTargetView(RenderTargetView, Color.WhiteSmoke);
Device.ImmediateContext.ClearDepthStencilView(DepthStencilView, DepthStencilClearFlags.Depth, 1.0f, 0);
//Draw
DeviceContext.InputAssembler.PrimitiveTopology = PrimitiveTopology.TriangleList;
DeviceContext.InputAssembler.SetVertexBuffers(0, vertexBufferBinding_model);
Device.ImmediateContext.Draw(vertCount, 0);
base.Draw(totalRunTime, timeSinceLastFrame);
}
public override void LoadContent()
{
//Initialize the timer
timer = new Stopwatch();
timer.Start();
//Initialize Matrices
view = Matrix.LookAtLH(new Vector3(0, 100, -500), new Vector3(0, 0, 0), Vector3.UnitY);
proj = Matrix.PerspectiveFovLH((float)Math.PI / 4.0f, RenderForm.ClientSize.Width / RenderForm.ClientSize.Height, 0.1f, 10000.0f);
viewProj = Matrix.Multiply(view, proj);
//Load Shaders
ShaderBytecode vertexShaderByteCode;
ShaderBytecode pixelShaderByteCode;
try
{
vertexShaderByteCode = ShaderBytecode.CompileFromFile("Shaders/shader.hlsl", "VShader", "vs_4_0",ShaderFlags.None,EffectFlags.None);
pixelShaderByteCode = ShaderBytecode.CompileFromFile("Shaders/shader.hlsl", "PShader", "ps_4_0",ShaderFlags.None,EffectFlags.None);
}
catch (System.Exception ex)
{
throw ex;
}
vertexShader = new VertexShader(Device, vertexShaderByteCode);
pixelShader = new PixelShader(Device, pixelShaderByteCode);
DeviceContext.VertexShader.Set(vertexShader);
DeviceContext.PixelShader.Set(pixelShader);
var signature = ShaderSignature.GetInputSignature(vertexShaderByteCode);
//Define first 16 floats as Position, next 16 as Color, next 12 normal (4 cords, 4 Color, 3 normal parts)
InputElement[] elements = new InputElement[]
{
new InputElement("POSITION", 0, SlimDX.DXGI.Format.R32G32B32A32_Float, 0, 0),
new InputElement("COLOR" , 0, SlimDX.DXGI.Format.R32G32B32A32_Float, 16, 0),
new InputElement("NORMAL" , 0, SlimDX.DXGI.Format.R32G32B32_Float, 32, 0),
};
//Define Layout for the InputAssembler
DeviceContext.InputAssembler.InputLayout = new InputLayout(Device, signature, elements);
//Generate and link constant buffers
constantBuffer = new Buffer(Device, System.Runtime.InteropServices.Marshal.SizeOf(new Matrix()), ResourceUsage.Default, BindFlags.ConstantBuffer, CpuAccessFlags.None, ResourceOptionFlags.None, 0);
DeviceContext.VertexShader.SetConstantBuffer(constantBuffer,0);
//load STL and generate Vertices from it
ModuleWorks.Meshf meshf = ModuleWorks.MeshHelper.ReadSTLf(#"C:\ModuleWorks\STL\Homer.stl", ModuleWorks.Unit.Metric);
try
{
vertCount = meshf.TriangleCount * 3;
var vertices_model = new DataStream(vertCount * System.Runtime.InteropServices.Marshal.SizeOf(typeof(Vertex)), true, true);
var stopWatch = new Stopwatch();
stopWatch.Start();
for (int x = 0; x < meshf.TriangleCount; x++)
{
var triangle = meshf.GetTriangle(x);
var normal = triangle.Normal;
vertices_model.Write(new Vertex(meshf.GetPoint(triangle.Idx1).X, meshf.GetPoint(triangle.Idx1).Y, meshf.GetPoint(triangle.Idx1).Z, 1.0f, 0.0f, 0.0f, 1.0f, normal.X, normal.Y, normal.Z));
vertices_model.Write(new Vertex(meshf.GetPoint(triangle.Idx2).X, meshf.GetPoint(triangle.Idx2).Y, meshf.GetPoint(triangle.Idx2).Z, 1.0f, 0.0f, 0.0f, 1.0f, normal.X, normal.Y, normal.Z));
vertices_model.Write(new Vertex(meshf.GetPoint(triangle.Idx3).X, meshf.GetPoint(triangle.Idx3).Y, meshf.GetPoint(triangle.Idx3).Z, 1.0f, 0.0f, 0.0f, 1.0f, normal.X, normal.Y, normal.Z));
}
vertices_model.Position = 0;
//Generate VertexBufferBinding
var sizeInBytes = vertCount * System.Runtime.InteropServices.Marshal.SizeOf(typeof(Vertex));
var stride = System.Runtime.InteropServices.Marshal.SizeOf(typeof(Vector4)) * 2 + System.Runtime.InteropServices.Marshal.SizeOf(typeof(Vector3));
var vertexBuffer_model = new Buffer(Device, vertices_model, sizeInBytes, ResourceUsage.Default, BindFlags.VertexBuffer, CpuAccessFlags.None, ResourceOptionFlags.None, 0);
vertexBufferBinding_model = new VertexBufferBinding(vertexBuffer_model, stride, 0);
vertices_model.Close();
}
catch (System.Exception ex)
{
Console.WriteLine(ex);
return;
}
}
public override void Dispose()
{
vertexShader.Dispose();
pixelShader.Dispose();
constantBuffer.Dispose();
base.Dispose();
}
}
}
shader.hlsl
cbuffer matrixBuffer : register(b0)
{
float4x4 worldViewProj;
};
struct VOut
{
float4 position : SV_POSITION;
float4 color : COLOR;
float3 normal : NORMAL;
};
VOut VShader(float4 position : POSITION, float4 color : COLOR, float3 normal : NORMAL)
{
VOut output = (VOut)0;
output.position = mul(worldViewProj, position);
output.normal = normalize(mul((float3x3)worldViewProj,normal));
output.color = color;
return output;
}
float4 PShader(VOut vout) : SV_TARGET
{
return vout.color;
}
Thanks in advance

I solved it.
The issue was based on an inperformant approach to adding the vertices.
For further information, checkout this nice paper I've found

transition between images for picturebox in c# [duplicate]

This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
Transition of images in Windows Forms Picture box
I use the code below to change images in a picturebox every 5 seconds, but it's not looking good when changing an image: I want a transition effect between images.
Used Code
private void backgroundWorker1_DoWork(object sender, DoWorkEventArgs e)
{
Bitmap[] pictures = new Bitmap[9];
pictures[0] = new Bitmap(#"Library Images\cf3.jpg");
pictures[1] = new Bitmap(#"Library Images\cf4.jpg");
pictures[2] = new Bitmap(#"Library Images\l1.JPG");
pictures[3] = new Bitmap(#"Library Images\l2.JPG");
pictures[4] = new Bitmap(#"Library Images\l3.JPG");
pictures[5] = new Bitmap(#"Library Images\l4.JPG");
pictures[6] = new Bitmap(#"Library Images\l5.JPG");
pictures[7] = new Bitmap(#"Library Images\l6.JPG");
pictures[8] = new Bitmap(#"Library Images\l7.JPG");
Random random = new Random();
while (true)
{
int attempt = random.Next(0, pictures.Length);
pictureBox1.Image = pictures[attempt];
System.Threading.Thread.Sleep(5000);
}
}
example code greatly appreciated thanks in advance...

Simply take new code file and paste below code in it
an original answer for the similar question, answer taken from another question
Answer
using System;
using System.Drawing;
using System.Drawing.Imaging;
using System.Windows.Forms;
public class BlendPanel : Panel
{
private Image mImg1;
private Image mImg2;
private float mBlend;
public BlendPanel()
{
SetStyle(ControlStyles.AllPaintingInWmPaint | ControlStyles.UserPaint | ControlStyles.OptimizedDoubleBuffer, true);
}
public Image Image1
{
get { return mImg1; }
set { mImg1 = value; Invalidate(); }
}
public Image Image2
{
get { return mImg2; }
set { mImg2 = value; Invalidate(); }
}
public float Blend
{
get { return mBlend; }
set { mBlend = value; Invalidate(); }
}
protected override void OnPaint(PaintEventArgs e)
{
if (mImg1 == null || mImg2 == null)
e.Graphics.FillRectangle(new SolidBrush(this.BackColor), new Rectangle(0, 0, this.Width, this.Height));
else
{
Rectangle rc = new Rectangle(0, 0, this.Width, this.Height);
ColorMatrix cm = new ColorMatrix();
ImageAttributes ia = new ImageAttributes();
cm.Matrix33 = mBlend;
ia.SetColorMatrix(cm);
e.Graphics.DrawImage(mImg2, rc, 0, 0, mImg2.Width, mImg2.Height, GraphicsUnit.Pixel, ia);
cm.Matrix33 = 1F - mBlend;
ia.SetColorMatrix(cm);
e.Graphics.DrawImage(mImg1, rc, 0, 0, mImg1.Width, mImg1.Height, GraphicsUnit.Pixel, ia);
}
base.OnPaint(e);
}
}
Build your project. You can now drop a BlendPanel from the top of the toolbox onto your form. Here's a sample program that uses it:
private float mBlend;
private int mDir = 1;
public int count = 0;
public Bitmap[] pictures;
public void myPhoto()
{
pictures = new Bitmap[9];
pictures[0] = new Bitmap(#"Library Images\cf3.jpg");
pictures[1] = new Bitmap(#"Library Images\cf4.jpg");
pictures[2] = new Bitmap(#"Library Images\l1.JPG");
pictures[3] = new Bitmap(#"Library Images\l2.JPG");
pictures[4] = new Bitmap(#"Library Images\l3.JPG");
pictures[5] = new Bitmap(#"Library Images\l4.JPG");
pictures[6] = new Bitmap(#"Library Images\l5.JPG");
pictures[7] = new Bitmap(#"Library Images\l6.JPG");
pictures[8] = new Bitmap(#"Library Images\l7.JPG");
timer1.Interval = 50; //time of transition
timer1.Tick += BlendTick;
try
{
blendPanel1.Image1 = pictures[count];
blendPanel1.Image2 = pictures[++count];
}
catch
{
}
timer1.Enabled = true;
}
private void BlendTick(object sender, EventArgs e)
{
mBlend += mDir * 0.02F;
if (mBlend > 1)
{
mBlend = 0.0F;
if ((count + 1) < pictures.Length)
{
blendPanel1.Image1 = pictures[count];
blendPanel1.Image2 = pictures[++count];
}
else
{
blendPanel1.Image1 = pictures[count];
blendPanel1.Image2 = pictures[0];
count = 0;
}
}
blendPanel1.Blend = mBlend;
}
You'll need to modify the new Bitmap(#"yourimagePath"); calls. Build and run. You should see the displayed image smoothly morph from your first image to your second image without any flickering.
I hope it helps for other...

Categories

Resources