I want to play a video (mostly .mov with Motion JPEG) in frame by frame mode with changing framerate. I have a function who gives me a framenumber and then I have to jump there. It will be mostly in one direction but can skip a few frames from time to time; also the velocity is not constant.
So I have a timer asking every 40ms about a new framenumber and setting the new position.
My first approach now is with DirectShow.Net (Interop.QuartzTypeLib). Therefore I render and open the video and set it to pause to draw the picture in the graph
FilgraphManagerClass media = new FilgraphManagerClass();
media.RenderFile(FileName);
media.pause();
Now I will just set a new position
media.CurrentPosition = framenumber * media.AvgTimePerFrame;
Since the video is in pause mode it will then draw every requested new position (frame). Works perfectly fine but really slow... the video keeps stuttering and lagging and its not the video source; there are enough frames recorded to play a fluent video.
With some performance tests I found out that the LAV-Codec is the bottleneck here. This is not included directly in my project since its a DirectShow-Player it will be cast through my codec pack I installed on my PC.
Ideas:
Using the LAV-Codec by myself directly in C#. I searched but everyone is using DirectShow it seems, building their own filters and not using existing ones directly in the project.
Instead of seeking or setting the time, can I get single frames just by the framenumber and draw them simply?
Is there a complete other way to archive what I want to do?
Background:
This project has to be a train simulator. We recorded real time videos of trains driving from inside the cockpit and know which frame is what position. Now my C# programm calculates the position of the train in dependence of time and acceleration, gives back the appropriate framenumber and draw this frame.
Additional Information:
There is another project (not written by me) in C/C++ who uses DirectShow and the avcodec-LAV directly with a similar way I do and it works fine! Thats because I had the idea to use a codec / filter like the avrcodec-lav by myself. But I can't find an interop or interface to work with C#.
Thanks everyone for reading this and trying to help! :)
Obtaining specific frame by seeking filter graph (the entire pipeline) is pretty slow since every seek operation involves the following on its backyard: flushing everything, possibly re-creating worker threads, seeking to first key frame/splice point/clean point/I-Frame before the requested time, start of decoding starting from found position skipping frames until originally requested time is reached.
Overall, the method works well when you scrub paused video, or retrieve specific still frames. When however you try to play this as smooth video, it eventually causes significant part of the effort to be wasted and spent on seeking within video stream.
Solutions here are:
re-encode video to remove or reduce temporal compression (e.g. Motion JPEG AVI/MOV/MP4 files)
whenever possible prefer to skip frames and/or re-timestamp them according to your algorithm instead of seeking
have a cached of decoded video frames and pick from there, populate them as necessary in worker thread
The latter two are unfortunately hard to achieve without advanced filter development (where continuous decoding without interruption by seeking operations is the key to achieving decent performance). With basic DirectShow.Net you only have basic control over streaming and hence the first item from the list above.
Wanted to post a comment instead of an answer, but don't have the reputation. I think your heading in the wrong direction with Direct Show. I've been messing with motion-jpeg for a few years now between C# & Android, and have gotten great performance with built-in .NET code (for converting byte-array to Jpeg frame) and a bit of multi-threading. I can easily achieve over 30fps from multiple devices with each device running in it's own thread.
Below is an older version of my motion-jpeg parser from my C# app 'OmniView'. To use, just send the network stream to the constructor, and receive the OnImageReceived event. Then you can easily save the frames to the hard-drive for later use (perhaps with the filename set to the timestamp for easy lookup). For better performance though, you will want to save all of the images to one file.
using OmniView.Framework.Helpers;
using System;
using System.IO;
using System.Text;
using System.Windows.Media.Imaging;
namespace OmniView.Framework.Devices.MJpeg
{
public class MJpegStream : IDisposable
{
private const int BUFFER_SIZE = 4096;
private const string tag_length = "Content-Length:";
private const string stamp_format = "yyyyMMddHHmmssfff";
public delegate void ImageReceivedEvent(BitmapImage img);
public delegate void FrameCountEvent(long frames, long failed);
public event ImageReceivedEvent OnImageReceived;
public event FrameCountEvent OnFrameCount;
private bool isHead, isSetup;
private byte[] buffer, newline, newline_src;
private int imgBufferStart;
private Stream data_stream;
private MemoryStream imgStreamA, imgStreamB;
private int headStart, headStop;
private long imgSize, imgSizeTgt;
private bool useStreamB;
public volatile bool EnableRecording, EnableSnapshot;
public string RecordPath, SnapshotFilename;
private string boundary_tag;
private bool tagReadStarted;
private bool enableBoundary;
public volatile bool OututFrameCount;
private long FrameCount, FailedCount;
public MJpegStream() {
isSetup = false;
imgStreamA = new MemoryStream();
imgStreamB = new MemoryStream();
buffer = new byte[BUFFER_SIZE];
newline_src = new byte[] {13, 10};
}
public void Init(Stream stream) {
this.data_stream = stream;
FrameCount = FailedCount = 0;
startHeader(0);
}
public void Dispose() {
if (data_stream != null) data_stream.Dispose();
if (imgStreamA != null) imgStreamA.Dispose();
if (imgStreamB != null) imgStreamB.Dispose();
}
//=============================
public void Process() {
if (isHead) processHeader();
else {
if (enableBoundary) processImageBoundary();
else processImage();
}
}
public void Snapshot(string filename) {
SnapshotFilename = filename;
EnableSnapshot = true;
}
//-----------------------------
// Header
private void startHeader(int remaining_bytes) {
isHead = true;
headStart = 0;
headStop = remaining_bytes;
imgSizeTgt = 0;
tagReadStarted = false;
}
private void processHeader() {
int t = BUFFER_SIZE - headStop;
headStop += data_stream.Read(buffer, headStop, t);
int nl;
//
if (!isSetup) {
byte[] new_newline;
if ((nl = findNewline(headStart, headStop, out new_newline)) >= 0) {
string tag = Encoding.UTF8.GetString(buffer, headStart, nl - headStart);
if (tag.StartsWith("--")) boundary_tag = tag;
headStart = nl+new_newline.Length;
newline = new_newline;
isSetup = true;
return;
}
} else {
while ((nl = findData(newline, headStart, headStop)) >= 0) {
string tag = Encoding.UTF8.GetString(buffer, headStart, nl - headStart);
if (!tagReadStarted && tag.Length > 0) tagReadStarted = true;
headStart = nl+newline.Length;
//
if (!processHeaderData(tag, nl)) return;
}
}
//
if (headStop >= BUFFER_SIZE) {
string data = Encoding.UTF8.GetString(buffer, headStart, headStop - headStart);
throw new Exception("Invalid Header!");
}
}
private bool processHeaderData(string tag, int index) {
if (tag.StartsWith(tag_length)) {
string val = tag.Substring(tag_length.Length);
imgSizeTgt = long.Parse(val);
}
//
if (tag.Length == 0 && tagReadStarted) {
if (imgSizeTgt > 0) {
finishHeader(false);
return false;
}
if (boundary_tag != null) {
finishHeader(true);
return false;
}
}
//
return true;
}
private void finishHeader(bool enable_boundary) {
int s = shiftBytes(headStart, headStop);
enableBoundary = enable_boundary;
startImage(s);
}
//-----------------------------
// Image
private void startImage(int remaining_bytes) {
isHead = false;
imgBufferStart = remaining_bytes;
Stream imgStream = getStream();
imgStream.Seek(0, SeekOrigin.Begin);
imgStream.SetLength(imgSizeTgt);
imgSize = 0;
}
private void processImage() {
long img_r = (imgSizeTgt - imgSize - imgBufferStart);
int bfr_r = Math.Max(BUFFER_SIZE - imgBufferStart, 0);
int t = (int)Math.Min(img_r, bfr_r);
int s = data_stream.Read(buffer, imgBufferStart, t);
int x = imgBufferStart + s;
appendImageData(0, x);
imgBufferStart = 0;
//
if (imgSize >= imgSizeTgt) processImageData(0);
}
private void processImageBoundary() {
int t = Math.Max(BUFFER_SIZE - imgBufferStart, 0);
int s = data_stream.Read(buffer, imgBufferStart, t);
//
int nl, start = 0;
int end = imgBufferStart + s;
while ((nl = findData(newline, start, end)) >= 0) {
int tag_length = boundary_tag.Length;
if (nl+newline.Length+tag_length > BUFFER_SIZE) {
appendImageData(start, nl+newline.Length - start);
start = nl+newline.Length;
continue;
}
//
string v = Encoding.UTF8.GetString(buffer, nl+newline.Length, tag_length);
if (v == boundary_tag) {
appendImageData(start, nl - start);
int xstart = nl+newline.Length + tag_length;
int xsize = shiftBytes(xstart, end);
processImageData(xsize);
return;
} else {
appendImageData(start, nl+newline.Length - start);
}
start = nl+newline.Length;
}
//
if (start < end) {
int end_x = end - newline.Length;
if (start < end_x) {
appendImageData(start, end_x - start);
}
//
shiftBytes(end - newline.Length, end);
imgBufferStart = newline.Length;
}
}
private void processImageData(int remaining_bytes) {
if (EnableSnapshot) {
EnableSnapshot = false;
saveSnapshot();
}
//
try {
BitmapImage img = createImage();
if (EnableRecording) recordFrame();
if (OnImageReceived != null) OnImageReceived.Invoke(img);
FrameCount++;
}
catch (Exception) {
// output frame error ?!
FailedCount++;
}
//
if (OututFrameCount && OnFrameCount != null) OnFrameCount.Invoke(FrameCount, FailedCount);
//
useStreamB = !useStreamB;
startHeader(remaining_bytes);
}
private void appendImageData(int index, int length) {
Stream imgStream = getStream();
imgStream.Write(buffer, index, length);
imgSize += (length - index);
}
//-----------------------------
private void recordFrame() {
string stamp = DateTime.Now.ToString(stamp_format);
string filename = RecordPath+"\\"+stamp+".jpg";
//
ImageHelper.Save(getStream(), filename);
}
private void saveSnapshot() {
Stream imgStream = getStream();
//
imgStream.Position = 0;
Stream file = File.Open(SnapshotFilename, FileMode.Create, FileAccess.Write);
try {imgStream.CopyTo(file);}
finally {file.Close();}
}
private BitmapImage createImage() {
Stream imgStream = getStream();
imgStream.Position = 0;
return ImageHelper.LoadStream(imgStream);
}
//-----------------------------
private Stream getStream() {return useStreamB ? imgStreamB : imgStreamA;}
private int findNewline(int start, int stop, out byte[] data) {
for (int i = start; i < stop; i++) {
if (i < stop-1 && buffer[i] == newline_src[0] && buffer[i+1] == newline_src[1]) {
data = newline_src;
return i;
} else if (buffer[i] == newline_src[1]) {
data = new byte[] {newline_src[1]};
return i;
}
}
data = null;
return -1;
}
private int findData(byte[] data, int start, int stop) {
int data_size = data.Length;
for (int i = start; i < stop-data_size; i++) {
if (findInnerData(data, i)) return i;
}
return -1;
}
private bool findInnerData(byte[] data, int buffer_index) {
int count = data.Length;
for (int i = 0; i < count; i++) {
if (data[i] != buffer[buffer_index+i]) return false;
}
return true;
}
private int shiftBytes(int start, int end) {
int c = end - start;
for (int i = 0; i < c; i++) {
buffer[i] = buffer[end-c+i];
}
return c;
}
}
}
Related
I'm dealing with c# concurrent-queue and multi-threading in socket-programming tcp/ip
First, I've already done with socket-programming itself. That means, I've already finished coding about client, server and stuffs about communication itself
basic structure is pipe-lined(producer-consumer problem) and now I'm doing with bit conversion
below is brief summary about my code
client-socket ->server-socket -> concurrent_queue_1(with type byte[65536],Thread_1 process this) -> concurrent_queue_2(with type double[40,3500], Thread_2 process this) -> display-data or other work(It can be gpu-work)
*(double[40,3500] can be changed to other size)
Till now,I've implemented putting_data into queue1(Thread1) and just dequeuing all(Thread2) and, its speed is about 700Mbps
The reason I used two concurrent_queue is, I want communication,and type conversion work to be processed in background regardless of main procedure about control things.
Here is the code about my own concurrent_queue with Blocking
public class BlockingConcurrentQueue<T> : IDisposable
{
private readonly ConcurrentQueue<T> _internalQueue;
private AutoResetEvent _autoResetEvent;
private long _consumed;
private long _isAddingCompleted = 0;
private long _produced;
private long _sleeping;
public BlockingConcurrentQueue()
{
_internalQueue = new ConcurrentQueue<T>();
_produced = 0;
_consumed = 0;
_sleeping = 0;
_autoResetEvent = new AutoResetEvent(false);
}
public bool IsAddingCompleted
{
get
{
return Interlocked.Read(ref _isAddingCompleted) == 1;
}
}
public bool IsCompleted
{
get
{
if (Interlocked.Read(ref _isAddingCompleted) == 1 && _internalQueue.IsEmpty)
return true;
else
return false;
}
}
public void CompleteAdding()
{
Interlocked.Exchange(ref _isAddingCompleted, 1);
}
public void Dispose()
{
_autoResetEvent.Dispose();
}
public void Enqueue(T item)
{
_internalQueue.Enqueue(item);
if (Interlocked.Read(ref _isAddingCompleted) == 1)
throw new InvalidOperationException("Adding Completed.");
Interlocked.Increment(ref _produced);
if (Interlocked.Read(ref _sleeping) == 1)
{
Interlocked.Exchange(ref _sleeping, 0);
_autoResetEvent.Set();
}
}
public bool TryDequeue(out T result)
{
if (Interlocked.Read(ref _consumed) == Interlocked.Read(ref _produced))
{
Interlocked.Exchange(ref _sleeping, 1);
_autoResetEvent.WaitOne();
}
if (_internalQueue.TryDequeue(out result))
{
Interlocked.Increment(ref _consumed);
return true;
}
return false;
}
}
My question is here
As I mentioned above, concurrent_queue1's type is byte[65536] and 65536 bytes = 8192 double data.
(40 * 3500=8192 * 17.08984375)
I want merge multiple 8192 double data into form of double[40,3500](size can be changed)and enqueue to concurrent_queue2 with Thread2
It's easy to do it with naive-approach(using many complex for loop) but it's slow cuz, It copys all the
data and expose to upper class or layer.
I'm searching method automatically enqueuing with matched size like foreach loop automatically iterates through 2D-array in row-major way, not yet found
Is there any fast way to merge 1D-byte array into form of 2D-double array and enqueue it?
Thanks for your help!
I try to understand your conversion rule, so I write this conversion code. Use Parallel to speed up the calculation.
int maxSize = 65536;
byte[] dim1Array = new byte[maxSize];
for (int i = 0; i < maxSize; ++i)
{
dim1Array[i] = byte.Parse((i % 256).ToString());
}
int dim2Row = 40;
int dim2Column = 3500;
int byteToDoubleRatio = 8;
int toDoubleSize = maxSize / byteToDoubleRatio;
double[,] dim2Array = new double[dim2Row, dim2Column];
Parallel.For(0, toDoubleSize, i =>
{
int row = i / dim2Column;
int col = i % dim2Column;
int originByteIndex = row * dim2Column * byteToDoubleRatio + col * byteToDoubleRatio;
dim2Array[row, col] = BitConverter.ToDouble(
dim1Array,
originByteIndex);
});
need to process data received on a serial port in the background so that the decoded data can be used/displayed on the user form as it is received. Serial data message blocks may sometimes be received in multiple buffers. Therefore, the serial data buffers must be processed in the order they are received in order to insure that all message blocks are complete.
I am trying to process multiple serial data buffers in the order they are received using delegates in c#.
The method "public static void ReceiveSerialData(byte[] byteBuffer, int length)" is called from my ClassSerialPort when data is received.
I get the exception "System.NullReferenceException: 'Object reference not set to an instance of an object.'" on the line "caller.EndInvoke(out threadID, result);"
I have two questions:
How do I correct the System.NullReferenceException exception?
Will using delegates this way process multiple serial data buffers in the order they are received?
My code:
using ClassSerialPort;
using TheUserForm;
using System;
using System.Threading;
using System.Diagnostics;
using System.Text;
namespace TheUserForm
{
class ClsReceiveSerialData
{
// SerialPort property
private ClsSerialPort serialPort;
public ClsSerialPort SerialPor`enter code here`t
{
get
{
return serialPort;
}
set
{
serialPort = value;
}
}
public static void ReceiveSerialData(byte[] byteBuffer, int length)
{
int i;
int threadID = 0;
#if (TEST)
StringBuilder sb = new StringBuilder(byteBuffer.Length * 2);
i = 0;
foreach (byte b in byteBuffer)
{
//convert a byte to a hex string
sb.AppendFormat("{0:x2}", b);
i++;
}
//write the debuf string to the output window
Debug.WriteLine("byteBuffer[" + (i - 1) + "] = " + sb.ToString());
#endif
// create an instance of a delegate class
ClsProcessData ad = new ClsProcessData();
// Create the delegate.
AsyncProcessData caller =
AsyncProcessData(ad.BeginProcessingData);
// Initiate the asychronous call.
IAsyncResult result =
caller.BeginInvoke(byteBuffer, length, out threadID, null, null);
// Call EndInvoke to wait for the asynchronous call to complete,
// and to retrieve the results.
caller.EndInvoke(out threadID, result);
}
}
class ClsProcessData
{
bool gotFirstFEcode = false;
bool gotSecondFEcode = false;
bool gotCtrlrCIV = false;
bool okToSaveCommandBytes = false;
bool gotEndOfMessageCode = true;
byte[] commandData;
int commandIndex = 0;
public void BeginProcessingData(byte[] data, int byteCount,
out int threadId)
{
lock (this)
{
int i;
threadId = Thread.CurrentThread.ManagedThreadId;
#if (TEST)
Debug.WriteLine(
String.Concat("BeginProcessingData threadID = ",
Convert.ToString(threadId)));
#endif
//find a preamble
i = 0;
while (i < byteCount)
{
// have we completed processing the current message?
if (gotEndOfMessageCode)
{
//reset the preamble detection booleans
gotFirstFEcode = false;
gotSecondFEcode = false;
gotCtrlrCIV = false;
okToSaveCommandBytes = false;
gotEndOfMessageCode = false;
} // If
//can we save a command byte now?
if (okToSaveCommandBytes)
{
//go save a command byte
i = ExtractCommand(data, i, byteCount);
//reset the preamble detection booleans
gotFirstFEcode = false;
gotSecondFEcode = false;
gotCtrlrCIV = false;
okToSaveCommandBytes = false;
gotEndOfMessageCode = false;
} // If
//have we found the radio//s civ address?
if (gotCtrlrCIV && (data[i] ==
ClsConstants.CTRLR_DEFAULT_CIV_ADDR))
{
//we are ok to start saving command bytes
okToSaveCommandBytes = true;
} // If
//do we have both FE codes of the preamble and not
another extraneous FE code in the buffer
if (gotSecondFEcode && (data[i] ==
ClsConstants.CTRLR_DEFAULT_CIV_ADDR))
{
gotCtrlrCIV = true;
} // If
if (gotFirstFEcode && (data[i] ==
ClsConstants.PREAMBLE_CODE))
{
gotSecondFEcode = true;
} // If
//do we have the first preamble code?
if ((data[i] == ClsConstants.PREAMBLE_CODE) &&
(!gotSecondFEcode))
{
gotFirstFEcode = true;
//reset } // of message boolean
gotEndOfMessageCode = false;
} // If
//increment the array index
i++;
} // while
}
}
// ExrtractCommand returns an updated index value
private int ExtractCommand(byte[] data, int index, int byteCount)
{
int i = index;
while ((i < byteCount) && (!gotEndOfMessageCode))
{
if (data[i] == ClsConstants.END_OF_MESSAGE_CODE)
{
//set the end of message flag
gotEndOfMessageCode = true;
//Process the command
ProcessCommand(commandData, commandIndex);
}
else
{
//save a command byte
commandData[commandIndex] = data[i];
//increment to the next command index
commandIndex++;
} // If
//increment the data array index
i++;
} // while
return i;
} // void
}
CSCore (https://github.com/filoe/cscore) seems to be a very good audio library for C# but it lacks documentation and good examples.
I've been playing with Bass.Net for a long time and the architecture of CSCore is not like the Bass library so it is really hard to find out the right way to do some common tasks.
I'm trying to capture microphone input from WasapiCapture device and output the recorded data to WasapiOut device but I could not succeed it.
The following is the code I could find after googling but it does not work.
MMDeviceEnumerator deviceEnum = new MMDeviceEnumerator();
MMDeviceCollection devices = deviceEnum.EnumAudioEndpoints(DataFlow.Capture, DeviceState.Active);
using (var capture = new WasapiCapture())
{
capture.Device = deviceEnum.GetDefaultAudioEndpoint(DataFlow.Capture, Role.Multimedia);
capture.Initialize();
using (var source = new SoundInSource(capture))
{
using (var soundOut = new WasapiOut())
{
capture.Start();
soundOut.Device = deviceEnum.GetDefaultAudioEndpoint(DataFlow.Render, Role.Multimedia);
soundOut.Initialize(source);
soundOut.Play();
}
}
}
What I am trying to do is write an application like this one:
http://www.pitchtech.ch/PitchBox/
I have my own DSP functions which I want to apply to recorded data.
Does anybody have examples of directing WasapiCapture to WasapiOut and writing a custom DSP?
I found the solution with the help of the CSCore library creator Florian Rosmann (filoe).
Here is a sample DSP class which amplifies the provided audio data.
class DSPGain: ISampleSource
{
ISampleSource _source;
public DSPGain(ISampleSource source)
{
if (source == null)
throw new ArgumentNullException("source");
_source = source;
}
public int Read(float[] buffer, int offset, int count)
{
float gainAmplification = (float)(Math.Pow(10.0, GainDB / 20.0));
int samples = _source.Read(buffer, offset, count);
for (int i = offset; i < offset + samples; i++)
{
buffer[i] = Math.Max(Math.Min(buffer[i] * gainAmplification, 1), -1);
}
return samples;
}
public float GainDB { get; set; }
public bool CanSeek
{
get { return _source.CanSeek; }
}
public WaveFormat WaveFormat
{
get { return _source.WaveFormat; }
}
public long Position
{
get
{
return _source.Position;
}
set
{
_source.Position = value;
}
}
public long Length
{
get { return _source.Length; }
}
public void Dispose()
{
}
}
And you can use this class as in the sample below:
WasapiCapture waveIn;
WasapiOut soundOut;
DSPGain gain;
private void StartFullDuplex()
{
try
{
MMDeviceEnumerator deviceEnum = new MMDeviceEnumerator();
MMDeviceCollection devices = deviceEnum.EnumAudioEndpoints(DataFlow.Capture, DeviceState.Active);
waveIn = new WasapiCapture(false, AudioClientShareMode.Exclusive, 5);
waveIn.Device = deviceEnum.GetDefaultAudioEndpoint(DataFlow.Capture, Role.Multimedia);
waveIn.Initialize();
waveIn.Start();
var source = new SoundInSource(waveIn) { FillWithZeros = true };
soundOut = new WasapiOut(false, AudioClientShareMode.Exclusive, 5);
soundOut.Device = deviceEnum.GetDefaultAudioEndpoint(DataFlow.Render, Role.Multimedia);
gain = new DSPGain(source.ToSampleSource());
gain.GainDB = 5;
soundOut.Initialize(gain.ToWaveSource(16));
soundOut.Play();
}
catch (Exception ex)
{
Debug.WriteLine("Exception in StartFullDuplex: " + ex.Message);
}
}
private void StopFullDuplex()
{
if (soundOut != null) soundOut.Dispose();
if (waveIn != null) waveIn.Dispose();
}
This answer was posted as an edit to the question Directing microphone input to speakers and writing a custom DSP function with CSCore library by the OP Ahmet Uzun under CC BY-SA 3.0.
I am trying to save my kinect raw depth-data and i dont want to use the Kinect Studio, because i need the raw-data for further calculations. I am using the kinectv2 and kinect sdk!
My problem is that i just get low FPS for the saved data. Its about 15-17FPS.
Here my Framereader ( in further steps i want to save colorstream also):
frameReader = kinectSensor.OpenMultiSourceFrameReader(FrameSourceTypes.Depth);
frameReader.MultiSourceFrameArrived += Reader_MultiSourceFrameArrived;
Here the Event:
void Reader_MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
{
var reference = e.FrameReference.AcquireFrame();
saveFrameTest(reference);
frame_num++;
}
Here the saving-function:
private unsafe void saveFrameTest(Object reference)
{
MultiSourceFrame mSF = (MultiSourceFrame)reference;
using (var frame = mSF.DepthFrameReference.AcquireFrame())
{
if (frame != null)
{
using (Microsoft.Kinect.KinectBuffer depthBuffer = frame.LockImageBuffer())
{
if ((frame.FrameDescription.Width * frame.FrameDescription.Height) == (depthBuffer.Size / frame.FrameDescription.BytesPerPixel))
{
ushort* frameData = (ushort*)depthBuffer.UnderlyingBuffer;
byte[] rawDataConverted = new byte[(int)(depthBuffer.Size / 2)];
for (int i = 0; i < (int)(depthBuffer.Size / 2); ++i)
{
ushort depth = frameData[i];
rawDataConverted[i] = (byte)(depth >= frame.DepthMinReliableDistance && depth <= frame.DepthMaxReliableDistance ? (depth) : 0);
}
String date = string.Format("{0:hh-mm-ss}", DateTime.Now);
String filePath = System.IO.Directory.GetCurrentDirectory() + "/test/" +date+".raw";
File.WriteAllBytes(filePath, rawDataConverted);
rawDataConverted = null;
}
}
}
}
}
Further Infomration:
I included my code in a simple Console-Application on a Intel® Xeon® Processor E5-1620 3,7GHz with 16 GB RAM.
i think the for-loop is taking to much time:
for (int i = 0; i < (int)(depthBuffer.Size / 2); ++i)
{
ushort depth = frameData[i];
rawDataConverted[i] = (byte)(depth >= frame.DepthMinReliableDistance && depth <= frame.DepthMaxReliableDistance ? (depth) : 0);
}
I could improved my framerate. Now, i am accessing the kinectbuffer directly and resign the for-loop.
Microsoft.Kinect.KinectBuffer depthBuffer = frame.LockImageBuffer();
Marshal.Copy(depthBuffer.UnderlyingBuffer, rawData_depth, 0, (depthImageSize));
depthBuffer.Dispose();
frame.Dispose();
However i couldnt get the 30FPS-rate. Now it is about 25 FPS.
You could try something like this to get your array.
It's what I normally use.
var frame = frameReference.AcquireFrame();
var frameDescription = frame.FrameDescription;
ushort[] frameData = new ushort[frameDescription.Width * frameDescription.Height];
frame.CopyFrameDataToArray(frameData);
I am looking to create an external application that monitors the 'FPS' of a DirectX application (like FRAPS without the recording). I have read several Microsoft articles on performance measuring tools - but I am looking to get the feedback (and experience) of the community.
My question: what is the best method for obtaining the FPS of a DirectX application?
Windows has some Event Tracing for Windows providers related to DirectX profiling. The most intresting ones are Microsoft-Windows-D3D9 and Microsoft-Windows-DXGI, which allow tracing of the frame presentation events. The simplest way to calculate FPS is to count the number of PresentStart events withing a time interval and divide that by the length of the interval.
To work with ETW in C#, install Microsoft.Diagnostics.Tracing.TraceEvent package.
The following code sample displays FPS of running processes:
using System;
using System.Collections.Generic;
using System.Text;
using System.Diagnostics;
using System.Threading;
using Microsoft.Diagnostics.Tracing.Session;
namespace ConsoleApp1
{
//helper class to store frame timestamps
public class TimestampCollection
{
const int MAXNUM = 1000;
public string Name { get; set; }
List<long> timestamps = new List<long>(MAXNUM + 1);
object sync = new object();
//add value to the collection
public void Add(long timestamp)
{
lock (sync)
{
timestamps.Add(timestamp);
if (timestamps.Count > MAXNUM) timestamps.RemoveAt(0);
}
}
//get the number of timestamps withing interval
public int QueryCount(long from, long to)
{
int c = 0;
lock (sync)
{
foreach (var ts in timestamps)
{
if (ts >= from && ts <= to) c++;
}
}
return c;
}
}
class Program
{
//event codes (https://github.com/GameTechDev/PresentMon/blob/40ee99f437bc1061a27a2fc16a8993ee8ce4ebb5/PresentData/PresentMonTraceConsumer.cpp)
public const int EventID_D3D9PresentStart = 1;
public const int EventID_DxgiPresentStart = 42;
//ETW provider codes
public static readonly Guid DXGI_provider = Guid.Parse("{CA11C036-0102-4A2D-A6AD-F03CFED5D3C9}");
public static readonly Guid D3D9_provider = Guid.Parse("{783ACA0A-790E-4D7F-8451-AA850511C6B9}");
static TraceEventSession m_EtwSession;
static Dictionary<int, TimestampCollection> frames = new Dictionary<int, TimestampCollection>();
static Stopwatch watch = null;
static object sync = new object();
static void EtwThreadProc()
{
//start tracing
m_EtwSession.Source.Process();
}
static void OutputThreadProc()
{
//console output loop
while (true)
{
long t1, t2;
long dt = 2000;
Console.Clear();
Console.WriteLine(DateTime.Now.ToString() + "." + DateTime.Now.Millisecond.ToString());
Console.WriteLine();
lock (sync)
{
t2 = watch.ElapsedMilliseconds;
t1 = t2 - dt;
foreach (var x in frames.Values)
{
Console.Write(x.Name + ": ");
//get the number of frames
int count = x.QueryCount(t1, t2);
//calculate FPS
Console.WriteLine("{0} FPS", (double)count / dt * 1000.0);
}
}
Console.WriteLine();
Console.WriteLine("Press any key to stop tracing...");
Thread.Sleep(1000);
}
}
public static void Main(string[] argv)
{
//create ETW session and register providers
m_EtwSession = new TraceEventSession("mysess");
m_EtwSession.StopOnDispose = true;
m_EtwSession.EnableProvider("Microsoft-Windows-D3D9");
m_EtwSession.EnableProvider("Microsoft-Windows-DXGI");
//handle event
m_EtwSession.Source.AllEvents += data =>
{
//filter out frame presentation events
if (((int)data.ID == EventID_D3D9PresentStart && data.ProviderGuid == D3D9_provider) ||
((int)data.ID == EventID_DxgiPresentStart && data.ProviderGuid == DXGI_provider))
{
int pid = data.ProcessID;
long t;
lock (sync)
{
t = watch.ElapsedMilliseconds;
//if process is not yet in Dictionary, add it
if (!frames.ContainsKey(pid))
{
frames[pid] = new TimestampCollection();
string name = "";
var proc = Process.GetProcessById(pid);
if (proc != null)
{
using (proc)
{
name = proc.ProcessName;
}
}
else name = pid.ToString();
frames[pid].Name = name;
}
//store frame timestamp in collection
frames[pid].Add(t);
}
}
};
watch = new Stopwatch();
watch.Start();
Thread thETW = new Thread(EtwThreadProc);
thETW.IsBackground = true;
thETW.Start();
Thread thOutput = new Thread(OutputThreadProc);
thOutput.IsBackground = true;
thOutput.Start();
Console.ReadKey();
m_EtwSession.Dispose();
}
}
}
Based on the source code of PresentMon project.
Fraps inserts a DLL into every running application and hooks specific DX calls to figure out the framerate and capture video, pretty sure that you'll have to do something similar. After a bit of poking around I found a Github project that does some basic DX hooking for doing captures and overlays, so that might be a good spot to start out with. Though I've not used it personally so I can't totally vouch for the quality.
http://spazzarama.com/2011/03/14/c-screen-capture-and-overlays-for-direct3d-9-10-and-11-using-api-hooks/
Building on https://stackoverflow.com/a/54625953/12047161:
I had more success not using the stopwatch as the event triggers seems to be asynchronous with the actual frames. I kept getting batches of 20-50 frames all at once, making the estimated FPS fluctuate between 50 and 250% of the actual value.
Instead i used TimeStampRelativeMSec
//handle event
m_EtwSession.Source.AllEvents += data =>
{
//filter out frame presentation events
if((int) data.ID == EventID_DxgiPresentStart && data.ProviderGuid == DXGI_provider)
{
int pid = data.ProcessID;
long t;
t = watch.ElapsedMilliseconds;
//if process is not yet in Dictionary, add it
if (!frames.ContainsKey(pid))
{
frames[pid] = new TimestampCollection();
string name = "";
var proc = Process.GetProcessById(pid);
if (proc != null)
{
using (proc)
{
name = proc.ProcessName;
}
}
else name = pid.ToString();
frames[pid].Name = name;
}
frames[pid].Add((long)data.TimeStampRelativeMSec);
}
};
property from the TraceEvent class, and calculate FPS by rounding the average time between an arbitrary number of past entries:
public double GetFrameTime(int count)
{
double returnValue = 0;
int listCount = timestamps.Count;
if(listCount > count)
{
for(int i = 1; i <= count; i++)
{
returnValue += timestamps[listCount - i] - timestamps[listCount - (i + 1)];
}
returnValue /= count;
}
return returnValue;
}
This method gave me far more accurate (Compared to, as available, in-game counters) of several different games i've tried.