I am pretty new in c# and could need some help on a audio project.
My audio input buffers call a method when they are filled up. In the method I marshall this buffers to a local float[], pass this to a function, where some audio processing is done. After processing the function throws back the manipulated float[], which I pass by Marshall.copy to the audio aout buffer. it works but it is is pretty hard to get the audioprocessing done fast enough, to pass the result back to the method without ending in ugly glitches. If I enlarge the audio buffers it gets better but I get untolarable high latency in the signal chain.
One problem is the GC. My DSP Routine is doing some FFT and the methods frequently need allocate local variables. I think this is slowing down my process a lot.
So I need a way to allogate once (and re-access) a few pieces of umanaged memory, keep this mem fpr the entire runtime and just reference to it from the methods.
I found e.g:
IntPtr hglobal = Marshal.AllocHGlobal(8192);
Marshal.FreeHGlobal(hglobal);
SO what I tried is to define a global static class "Globasl" with static member and asigned that IntPtr to that.
Globals.mem1 = hglobal;
From within any nother method I can access this now by
e.g.
int[] f = new int[2];
f[0] = 111;
f[1] = 222;
Marshal.Copy(f, 0, Globals.mem1, 2);
Now comes my problem:
If I want to access this int[] from the example above in another method, how could I do this?
thank you for your fast help.
I was little unprecise seems, sorry
my audiodevice driver throws a buffer filled event which I catch, (in pseudocode since I dont have access to my home desktop right now). looks like:
void buffer (....)
{
byte[] buf = new byte[asiobuffersize];
marshall.copy(asioinbuffers, 0, buf, asiobufferlenth);
buf= manipulate(buf);
marshall.copy(buf, 0, asiooutbuffer, asiobufferlenth);
}
the manipulate function is doing some conversions from byte to float then some math (FFT) and backtransform to byte and looks like e.g.
private byte[] manipulate(byte[] buf, Complex[] filter)
{
float bu = convertTofloat(buf); //conversion from byte to audio float here
Complex[] inbuf = new Complex[bu.Length];
Complex[] midbuf = new Complex[bu.Length];
Complex[] mid2buf = new Complex[bu.Length];
Complex[] outbuf = new Complex[bu.Length];
for ( n....)
{
inbuf[n]= bu[n]; //Copy to Complex
}
midbuf=FFT(inbuf); //Doing FFT transform
for (n....)
{
mid2buf[n]=midbuf[n]*filter[n]; // Multiply with filter
}
outbuf=iFFT(mid2buf) //inverse iFFT transform
byte bu = convertTobyte(float); //conversion from float back to audio byte
return bu;
}
here I expect my speed issue to be. So I thought the problem could be solved if the manipulating funktion could just "get" a fixed piece of unmanaged memory where (once pre-created) sits fixed all those those variables (Complex e.g.) and pre allocated menory, so that I must not create new ones each time the function is called. I first expected the reason for my glitches in wrong FFT or math but it happens in kind of "sharp" time distances of few seconds, so it is not connected to audio signal issues like clipping. I think the isseue happens, when the GC is doing some serious work and eats me exactly this few miliseconds missing to get the outbut buffer filled in time.
acceptable.
I really doubt the issues you are experiencing are induced by your managed buffer creation/copying. Instead I think your problem is that you have your data capture logic coupled with your DSP logic. Usually, captured data resides in a circular buffer, where the data is rewritten after some period, so you should be fetching this data as soon as posible.
The problem is that you don't fetch the next available data block until after your DSP is done; you already know FFT operations are really CPU intensive! If you have a processing peak, you may not be able to retrieve data before it is rewritten by the capture driver.
One possibility to address your issue is try to increase, if posible, the size and/or available amount of capture buffers. This buys you more time before the captured data is rewritten. The other possibility, and is the one that I favor is decoupling your processing stage from your capture stage, in this way, if new data is available while you are busy performing your DSP computations, you can still grab it and buffer it almost as soon as it becomes available. You become much more resilient to garbage collection induced pauses or computing peaks inside your manipulate method.
This would involve creating two threads: the capture thread and the processing thread. You would also need an "Event" that signals the processing thread that new data is available, and a queue, that will serve as a dynamic, expandable buffer.
The capture thread would look something like this:
// the following are class members
AutoResetEvent _bufQueueEvent = new AutoResetEvent(false);
Queue<byte[]> _bufQueue = new Queue<byte[]>;
bool _keepCaptureThreadAlive;
Thread _captureThread;
void CaptureLoop() {
while( _keepCaptureThreadAlive ) {
byte[] asioinbuffers = WaitForBuffer();
byte[] buf = new byte[asiobuffers.Length]
marshall.copy(asioinbuffers, 0, buf, asiobufferlenth);
lock( _bufQueue ) {
_bufQueue.Enqueue(buf);
}
_bufQueueEvent.Set(); // notify processing thread new data is available
}
}
void StartCaptureThread() {
_keepCaptureThreadAlive = true;
_captureThread = new Thread(new(ThreadStart(CaptureLoop));
_captureThread.Name = "CaptureThread";
_captureThread.IsBackground = true;
_captureThread.Start();
}
void StopCaptureThread() {
_keepCaptureThreadAlive = false;
_captureThread.Join(); // wait until the thread exits.
}
The processing thread would look something like this
// the following are class members
bool _keepProcessingThreadAlive;
Thread _processingThread;
void ProcessingLoop() {
while( _keepProcessingThreadAlive ) {
_bufQueueEvent.WaitOne(); // thread will sleep until fresh data is available
if( !_keepProcessingThreadAlive ) {
break; // check if thread is being waken for termination.
}
int queueCount;
lock( _bufQueue ) {
queueCount = _bufQueue.Count;
}
for( int i = 0; i < queueCount; i++ ) {
byte[] buffIn;
lock( _bufQueue ) {
// only lock during dequeue operation, this way the capture thread Will
// be able to enqueue fresh data even if we are still doing DSP processing
buffIn = _bufQueue.Dequeue();
}
byte[] buffOut = manipulate(buffIn); // you are safe if this stage takes more time than normal, you will still get the incoming data
// additional logic using manipulate() return value
...
}
}
}
void StartProcessingThread() {
_keepProcessingThreadAlive = true;
_processingThread = new Thread(new(ThreadStart(ProcessingLoop));
_processingThread.Name = "ProcessingThread";
_processingThread.IsBackground = true;
_processingThread.Start();
}
void StopProcessingThread() {
_keepProcessingThreadAlive = false;
_bufQueueEvent.Set(); // wake up thread in case it is waiting for data
_processingThread.Join();
}
At my job we also perform a lot of DSP and this pattern has really helped us with the kind of issues you are experiencing.
Related
I'm using native library via PInvoke calls which returns Byte* and want to make sure that in a producer/consumer scenario that the consumer thread gets the latest data.
I have a very contrived example here, to try to illustrate what I'm asking. I'm aware of the 'hot loop' in the Consumer (it's just an example, the real code is much larger and would not be feasible to paste here).
public unsafe class ThreadExample {
class PInvokeResult {
public Byte* Data;
public Int32 Length;
}
// Shared Objects Used For Synchronization
Object SyncRoot = new Object();
Queue<PInvokeResult> WorkQueue = new Queue<PInvokeResult>();
// Producer Thread
void Producer() {
while (true) {
PInvokeResult workItem;
workItem = new PInvokeResult();
workItem.Data = PInvokeNativeLongRunningCall(out workItem.Length);
lock (SyncRoot) {
WorkQueue.Enqueue(workItem);
}
}
}
// Consume Thread
void Consumer() {
while (true) {
bool workAvailable = false;
PInvokeResult workItem = null;
lock (SyncRoot) {
if (WorkQueue.Count > 0) {
workItem = WorkQueue.Dequeue();
workAvailable = true;
}
}
if (workAvailable) {
ProcessWorkItem(workItem);
PInvokeReturnPointerBuffer(workItem.Data);
}
}
}
}
Is the lock here enough to make sure that the data that is read from the Byte* pointer on PInvokeResult never points to 'stale' data for the consumer?
What I mean by stale data here is this case:
One specific Byte* buffer gets returned from the PInvokeNativeLongRunningCall invocation.
The buffer is passed from the producer to the consumer, this uses locking on the SyncRoot to make sure access to the Queue is safe.
The consumer does some work on the item and then returns it to the native code via PInvokeReturnPointerBuffer
The buffer is then recycled and re-used on the native side and the data in the buffer is first set to all zeroes and then written to again.
The cycle then starts over from 1.
When the buffer for the second times comes to the Consumer, how can I be sure that the data that it sees it the latest new data that was written to by the PInvoke call?
This question is purely from the C# perspective, I know that the native code is fine and solid, it's a well used library.
Is this even something that a language has to account for, or is this completely handled by the CPU itself?
I've notice in my test code that using the ConcurrentQueue<> somehow does not release resources after Dequeueing and eventually I run out of memory. Or the Garbage collection is not happening frequently enough.
Here is a snippet of the code. I know the ConcurrentQueue<> store references and yes, I do want create a new object each time so if the enqueueing is faster than dequeueing, memory will continue to rise. Also a screenshot of the memory usage. For testing, I sent through 5000 byte arrays with 500000 elements each.
There is a similar question asked:
ConcurrentQueue holds object's reference or value? "out of memory" exception
and everything mentioned in that post is what I experienced ... except that the memory won't release after dequeueing, even when the Queue is emptied.
I would appreciate any thoughts/insights to this.
ConcurrentQueue<byte[]> TestQueue = new ConcurrentQueue<byte[]>();
Task EnqTask = Task.Factory.StartNew(() =>
{
for (int i = 0; i < ObjCount; i++)
{
byte[] InData = new byte[ObjSize];
InData[0] = (byte)i; //used to show different array object
TestQueue.Enqueue(InData);
System.Threading.Thread.Sleep(20);
}
});
Task DeqTask = Task.Factory.StartNew(() =>
{
int Count = 0;
while (Count < ObjCount)
{
byte[] OutData;
if (TestQueue.TryDequeue(out OutData))
{
OutData[1] = 0xFF; //just do something with the data
Count++;
}
System.Threading.Thread.Sleep(40);
}
Picture of memory
I am writing a windows service application that is capable of collecting data from sensors like temperature, pressure volume etc...
The frequency at which the data is read is pretty high, there could be a hundred sensors and the data being received could be at a frequency could be one per second per sensor..
I need to store this data to an oracle database, for obvious reasons i dont want to hit the database at such a high rate.
Hence i want to create a Buffer.
My plan is to create a Buffer using the standard .NET Queue, a few threads keep Enqueue data into the queue and another timer driven thread can keep writing into the database at regular intervals.
What i want to know is..?? Is This thread safe
If this is not, what is the best way of creating a in memory buffer
To answer your question, as long as you lock accesses, you can have multiple threads access a regular queue.
For me though, I didn't use that and wanted to use queues with locks to keep them thread safe. I have been doing this in c# for one of my programs. I just use a regular queue, and then put a locker on accesses to it (enqueue, dequeue, count). It is completely thread safe if you just lock the accesses.
My setup comes from the tutorial/example here: http://www.albahari.com/threading/part2.aspx#_ProducerConsumerQWaitHandle
My situation is a little different than yours, but pretty similar. For me, my data can come in very fast, and if I don't queue it I lose the data if multiple come in at the same time. Then I have a thread running that slowly takes items off the queue and processes them. This hand-off uses an AutoResetEvent to hold my working-thread until data is ready to be processed. In your case you would use a timer or something that happens regularly.
I copy/pasted my code and tried to change the names. Hopefully I didn't completely break it by missing some name changes, but you should be able to get the gist.
public class MyClass : IDisposable
{
private Thread sensorProcessingThread = null;
private Queue<SensorData> sensorQueue = new Queue<SensorData>();
private readonly object _sensorQueueLocker = new object();
private EventWaitHandle _whSensorEvent = new AutoResetEvent(false);
public MyClass () {
sensorProcessingThread = new Thread(sensorProcessingThread_DoWork);
sensorProcessingThread.Start();
}
public void Dispose()
{
// Signal the end by sending 'null'
EnqueueSensorEvent(null);
sensorProcessingThread.Join();
_whSensorEvent.Close();
}
// The fast sensor data comes in, locks queue, and then
// enqueues the data, and releases the EventWaitHandle
private void EnqueueSensorEvent( SensorData wd )
{
lock ( _sensorQueueLocker )
{
sensorQueue.Enqueue(wd);
_whSensorEvent.Set();
}
}
// When asynchronous events come in, I just throw them into queue
private void OnSensorEvent( object sender, MySensorArgs e )
{
EnqueueSensorEvent(new SensorData(sender, e));
}
// I have several types of events that can come in,
// they just get packaged up into the same "SensorData"
// struct, and I worry about the contents later
private void FileSystem_Changed( object sender, System.IO.FileSystemEventArgs e )
{
EnqueueSensorEvent(new SensorData(sender, e));
}
// This is the slower process that waits for new SensorData,
// and processes it. Note, if it sees 'null' as data,
// then it knows it should quit the while(true) loop.
private void sensorProcessingThread_DoWork( object obj )
{
while ( true )
{
SensorData wd = null;
lock ( _sensorQueueLocker )
{
if ( sensorQueue.Count > 0 )
{
wd = sensorQueue.Dequeue();
if ( wd == null )
{
// Quit the loop, thread finishes
return;
}
}
}
if ( wd != null )
{
try
{
// Call specific handlers for the type of SensorData that was received
if ( wd.isSensorDataType1 )
{
SensorDataType1_handler(wd.sender, wd.SensorDataType1Content);
}
else
{
FileSystemChanged_handler(wd.sender, wd.FileSystemChangedContent);
}
}
catch ( Exception exc )
{
// My sensor processing also has a chance of failing to process completely, so I have a retry
// methodology that gives up after 5 attempts
if ( wd.NumFailedUpdateAttempts < 5 )
{
wd.NumFailedUpdateAttempts++;
lock ( _sensorQueueLocker )
{
sensorQueue.Enqueue(wd);
}
}
else
{
log.Fatal("Can no longer try processing data", exc);
}
}
}
else
_whWatchEvent.WaitOne(); // No more tasks, wait for a signal
}
}
Something you could maybe look at is Reactive (Rx) for .net from Microsoft. Check out: https://msdn.microsoft.com/en-us/data/gg577611.aspx and at the bottom of page is a pdf tutorial "Curing the asynchronous blues": http://go.microsoft.com/fwlink/?LinkId=208528 This is something very different but maybe you will see something you like.
I'm trying to create a stream application based on Spotify's libspotify SDK.
To achieve this in C# I'm using the ohLibspotify bindings and wrapper. This is only a thin abstraction layer so most of it will be a 1:1 mapping to the libspotify SDK. To play the incoming PCM data I'm using the NAudio library.
Most of the times I can play the first track. Then when I load the second one I get a AccessViolationException whilst trying to call sp_session_player_load(). Also this sometimes happens the first time I try to play a track and sometimes it happens the third time.
This is the function I use to play a track.
public void playTrack(string track, string juke)
{
new Thread(new ThreadStart(() =>
{
var playable = Link.CreateFromString(string.Format("spotify:track:{0}", track)).AsTrack();
if (playing)
{
player.Pause();
App.Logic.spotify.sp_session.PlayerUnload();
}
buffer = new BufferedWaveProvider(new WaveFormat())
{
BufferDuration = TimeSpan.FromSeconds(120),
DiscardOnBufferOverflow = false
};
var waitEvent = new AutoResetEvent(false);
while (!playable.IsLoaded())
{
waitEvent.WaitOne(30);
}
App.Logic.spotify.sp_session.PlayerLoad(playable);
App.Logic.spotify.sp_session.PlayerPlay(true);
player = new WaveOut();
player.Init(buffer);
player.Play();
playing = true;
})).Start();
}
The AccessViolationException occurs on line 6 of the following piece of code within the wrapper library.
[DllImport("libspotify")]
internal static extern SpotifyError sp_session_player_load(IntPtr #session, IntPtr #track);
public void PlayerLoad(Track #track)
{
SpotifyError errorValue;
errorValue = NativeMethods.sp_session_player_load(this._handle, track._handle);
SpotifyMarshalling.CheckError(errorValue);
}
The streaming callbacks:
public override void GetAudioBufferStats(SpotifySession session, out AudioBufferStats stats)
{
stats = new AudioBufferStats()
{
samples = App.Logic.spotify.player.buffer.BufferedBytes / 2,
stutter = 0
};
}
public override int MusicDelivery(SpotifySession session, AudioFormat format, IntPtr frames, int num_frames) {
int incoming_size = num_frames * format.channels * 2;
try
{
if (incoming_size > sample_buffer.Length)
{
short rendered_frames = Convert.ToInt16(Math.Floor((sample_buffer.Length / format.channels / 2d)));
short rendered_size = Convert.ToInt16(rendered_frames * format.channels * 2);
Marshal.Copy(frames, sample_buffer, 0, rendered_size);
App.Logic.spotify.player.buffer.AddSamples(sample_buffer, 0, rendered_size);
return rendered_frames;
}
else
{
Marshal.Copy(frames, sample_buffer, 0, incoming_size);
App.Logic.spotify.player.buffer.AddSamples(sample_buffer, 0, incoming_size);
return num_frames;
}
}
catch (Exception e)
{
return 0;
}
}
As #Weeble said in his comment to my question. It is very hard to diagnose the possible source of an AccessViolationException. In my case it was threading, there were multiple threads accessing the Spotify session object at once.
If this could be your problem as well you might want to look at this article. It talks about thread synchronization in C# and VB.Net. It's very easy really.
lock(sessionLock)
{
App.Logic.spotify.sp_session.PlayerLoad(playable);
App.Logic.spotify.sp_session.PlayerPlay(true);
}
The sessionLock object can be a simple instantiation of the Object type. Though it's not recommended to use any "actual" objects for this. In my case I did the following:
public class Player
{
private object sessionLock = new object();
// Rest of code with locks
}
This way you can lock the sessionLock object every time you want to access the, in this case, SpotifySession object. So that when thread 1 is loading a song and thread 2 wants to process the events at the same time, thread 2 will be blocked until the lock on sessionLock has been lifted.
As #Weeble suggested there are possibly some other things you might want to look into if threading is not the issue.
You might not be handling spotify's ref-counting correctly, and accidentally decreasing a ref-count too early or forgetting to increment one somewhere it is necessary.
You might be corrupting memory in the music callbacks that deal with native pointers.
There might be a bug in ohLibspotify or libspotify. If you think this is the case then go to the ohLibspotify repo or the openhome forum to report your issue.
I've been trying to create an observable which streams a state-of-the-world (snapshot) from a repository cache, followed by live updates from a separate feed. The catch is that the snapshot call is blocking, so the updates have to be buffered during that time.
This is what I've come up with, a little simplified. The GetStream() method is the one I'm concerned with. I'm wondering whether there is a more elegant solution. Assume GetDataFeed() pulses updates to the cache all day long.
private static readonly IConnectableObservable<long> _updateStream;
public static Constructor()
{
_updateStream = GetDataFeed().Publish();
_updateStream.Connect();
}
static void Main(string[] args)
{
_updateStream.Subscribe(Console.WriteLine);
Console.ReadLine();
GetStream().Subscribe(l => Console.WriteLine("Stream: " + l));
Console.ReadLine();
}
public static IObservable<long> GetStream()
{
return Observable.Create<long>(observer =>
{
var bufferedStream = new ReplaySubject<long>();
_updateStream.Subscribe(bufferedStream);
var data = GetSnapshot();
// This returns the ticks from GetSnapshot
// followed by the buffered ticks from _updateStream
// followed by any subsequent ticks from _updateStream
data.ToObservable().Concat(bufferedStream).Subscribe(observer);
return Disposable.Empty;
});
}
private static IObservable<long> GetDataFeed()
{
var feed = Observable.Interval(TimeSpan.FromSeconds(1));
return Observable.Create<long>(observer =>
{
feed.Subscribe(observer);
return Disposable.Empty;
});
}
Popular opinion opposes Subjects as they are not 'functional', but I can't find a way of doing this without a ReplaySubject. The Replay filter on a hot observable wouldn't work because it would replay everything (potentially a whole day's worth of stale updates).
I'm also concerned about race conditions. Is there a way to guarantee sequencing of some sort, should an earlier update be buffered before the snapshot? Can the whole thing be done more safely and elegantly with other RX operators?
Thanks.
-Will
Whether you use a ReplaySubject or the Replay function really makes no difference. Replay uses a ReplaySubject under the hood. I'll also note that you are leaking subscriptions like mad, which can cause a resource leak. Also, you put no limit on the size of the replay buffer. If you watch the observable all day long, then that replay buffer will keep growing and growing. You should put a limit on it to prevent that.
Here is an updated version of GetStream. In this version I take the simplistic approach of just limitting the Replay to the most recent 1 minute of data. This assumes that GetData will always complete and the observer will observe the results within that 1 minute. Your mileage may vary and you can probably improve upon this scheme. But at least this way when you have watched the observable all day long, that buffer will not have grown unbounded and will still only contain a minute's worth of updates.
public static IObservable<long> GetStream()
{
return Observable.Create<long>(observer =>
{
var updateStreamSubscription = new SingleAssignmentDisposable();
var sequenceDisposable = new SingleAssignmentDisposable();
var subscriptions = new CompositeDisposable(updateStreamDisposable, sequenceDisposable);
// start buffering the updates
var bufferedStream = _updateStream.Replay(TimeSpan.FromMinutes(1));
updateStreamSubscription.Disposable = bufferedStream.Connect();
// now retrieve the initial snapshot data
var data = GetSnapshot();
// subscribe to the snapshot followed by the buffered data
sequenceDisposable.Disposable = data.ToObservable().Concat(bufferedStream).subscribe(observer);
// return the composite disposable which will unsubscribe when the observer wishes
return subscriptions;
});
}
As for your questions about race conditions and filtering out "old" updates...if your snapshot data includes some sort of version information, and your update stream also providers version information, then you can effectively measure the latest version returned by your snapshot query and then filter the buffered stream to ignore values for older versions. Here is a rough example:
public static IObservable<long> GetStream()
{
return Observable.Create<long>(observer =>
{
var updateStreamSubscription = new SingleAssignmentDisposable();
var sequenceDisposable = new SingleAssignmentDisposable();
var subscriptions = new CompositeDisposable(updateStreamDisposable, sequenceDisposable);
// start buffering the updates
var bufferedStream = _updateStream.Replay(TimeSpan.FromMinutes(1));
updateStreamSubscription.Disposable = bufferedStream.Connect();
// now retrieve the initial snapshot data
var data = GetSnapshot();
var snapshotVersion = data.Length > 0 ? data[data.Length - 1].Version : 0;
var filteredUpdates = bufferedStream.Where(update => update.Version > snapshotVersion);
// subscribe to the snapshot followed by the buffered data
sequenceDisposable.Disposable = data.ToObservable().Concat(filteredUpdates).subscribe(observer);
// return the composite disposable which will unsubscribe when the observer wishes
return subscriptions;
});
}
I have successfully used this pattern when merging live updates with a stored snapshot. I haven't yet found an elegant Rx operator that already does this without any race conditions. But the above method could probably be turned into such. :)
Edit: Note I have left out error handling in the examples above. In theory the call to GetSnapshot could fail and you'd leak the subscription to the update stream. I suggest wrapping everything after the CompositeDisposable declaration in a try/catch block, and in the catch handler, ensure call subscriptions.Dispose() before re-throwing the exception.