Ffmpeg possible sws_scale memory leak - c#

I'm decoding whatever the camera codec is and then always encode it to H264 and more specifically qsv if it is supported. Currently having 2 cameras to test. One is H264 encoding and one is rawvideo. The problem comes with rawvideo. The pixel format is BGR24 and scaling to NV12
I will simplify the code because it is as any other example.
avcodec_send_packet()
// while
avcodec_receive_frame()
// if frame is not EAGAIN convert BGR24 to NV12
if (_pConvertContext == null)
{
_pConvertContext = CreateContext(sourcePixFmt, targePixFmt);
}
if (_convertedFrameBufferPtr == IntPtr.Zero)
{
int buffSize = ffmpeg.av_image_get_buffer_size(targePixFmt, sourceFrame->width, sourceFrame->height, 1);
_convertedFrameBufferPtr = Marshal.AllocHGlobal(buffSize);
ffmpeg.av_image_fill_arrays(ref _dstData, ref _dstLinesize, (byte*)_convertedFrameBufferPtr, targePixFmt, sourceFrame->width, sourceFrame->height, 1);
}
return ScaleImage(_pConvertContext, sourceFrame, targePixFmt, _dstData, _dstLinesize);
And ScaleImage method
ffmpeg.sws_scale(ctx, sourceFrame->data, sourceFrame->linesize, 0, sourceFrame->height, dstData, dstLinesize);
AVFrame* f = ffmpeg.av_frame_alloc();
var data = new byte_ptrArray8();
data.UpdateFrom(dstData);
var linesize = new int_array8();
linesize.UpdateFrom(dstLinesize);
f->data = data;
f->linesize = linesize;
f->width = sourceFrame->width;
f->height = sourceFrame->height;
f->format = (int)targePixelFormat;
return f;
After that sending the scaled frame to the encoder and receiving it and writing the file. After that I call av_frame_free(&frame) on the frame returned from the method. But when I set breakpoint I can see the address of the frame is the same even after calling av_frame_alloc() and cleaning everytime. And I think this is the reason for the memory leak. If I do deep clone of the f before returning it everything is fine. Why is that happening while the same logic works fine with the other camera?

Well, after 3 lost days of trying to figure out why it happens with certain cameras only, I've made a memory dump using DebugDiag 2 with a memory leak tracker. It turned out there is tons of memory allocations by _aligned_offset_malloc as you can see in the picture below. I was using ffmpeg 4.2.2, downgrading to 4.0.2 fixed the problem for me. No memory leak now. I'm using 32bit version.

Related

Fo-Dicom Window Change is slow

I am using the following render code. This is a slightly modified code because, in the actual code, I am getting all the files from the FTP server. CurrentWindowCenter and CurrentWindowWidth hold the current value of WL and WW. This value can be changed from the UI. I am showing the rendered WritableBitmap directly to the user using an image component inside a canvas.
But the rendering of the image is very slow. Especially for images with large file sizes such as X-Ray. So, the WW and WL change is also very slow since it also uses the render function.
I am not very knowledgeable about this. But is there a way to make the rendering or WW/WL change faster? Is there a way to skip the rendering of the image every time a WW/WL change happens?
Any advice in the right direction is appreciated.
Thanks in advance.
// assume filePath holds an actual file location.
var filePath = "";
var dicomFile = new DicomFile.Open(filePath);
var dicomImage = new DicomImage(dicomFile.DataSet);
if (CurrentWindowCenter.HasValue && CurrentWindowWidth.HasValue)
{
dicomImage.WindowCenter = Convert.ToDouble(CurrentWindowCenter.Value);
dicomImage.WindowWidth = Convert.ToDouble(CurrentWindowWidth.Value);
}
dicomImage.RenderImage().AsWriteableBitmap();
Environment
Fo-Dicom (4.0.8)
.NET Framework 4.8
WPF
My guess is that fo-dicom is not really intended for high performance and more for compatibility. For best performance you should probably use the GPU via DirectX, OpenCL or similar. Second best should be some tightly optimized SIMD code, probably using c++.
But there might be some improvements to be had using just c#. From the looks of it fo-dicom creates a new image, copies pixels to this image, and then creates a writeableBitmap and does another copy. These step will take some extra time.
My code for copying pixels and applying a lut/transfer function look like this:
public static unsafe void Mono16ToMono8(
byte* source,
byte* target,
int sourceStride,
int targetStride,
int width,
int height,
byte[] lut)
{
Parallel.For(0, height, y =>
{
var ySource = source + y * sourceStride;
var yTarget = target + y * targetStride;
var srcUshort = (ushort*)ySource;
for (int x = 0; x < width; x++)
{
var sample = srcUshort[x];
yTarget[x] = lut[sample];
}
});
}
And the code to do the actual update of the writeable bitmap:
public static unsafe void Update(
this WriteableBitmap self,
IImage source,
byte[] lut)
{
self.Lock();
try
{
var targetPtr = (byte*)self.BackBuffer;
fixed (byte* sourcePtr = source.Data)
{
if (source.PixelFormat == PixelType.Mono16)
{
Mono16ToMono8(
sourcePtr,
targetPtr,
source.Stride,
self.BackBufferStride,
source.Width,
source.Height,
lut);
}
}
self.AddDirtyRect(new Int32Rect(0, 0, (int)self.Width, (int)self.Height));
}
finally
{
self.Unlock();
}
}
This uses an internal IImage format, where source.Data is a ReadOnlySpan<byte>, but could just as well be a byte[]. I hope most of the other properties are self-explanatory. I would expect this code to be a bit faster since it avoids both allocations and some copying steps.
All of this assumes the image is in 16-bit unsigned format, that is common for dicom, but not the only format. It also assumes you can actually get a hold of a pointer to the actual pixel-buffer, and an array of the lut that maps each possible pixelvalue to a byte. It also assumes a writeablebitmap of the correct size and color space.
And as previously mentioned, if you want both high performance, and handle all possible image formats, you might need to invest time to build your own image rendering pipeline.

NAudio proper way to stream MediaFoundationReader

I'm trying to use NAudio to play shoutcast streams. This is doable (apparently) with a few lines of code:
var url = "http://dance.radiomonster.fm/320.mp3";
using(var radioStream = new MediaFoundationReader(url))
using(var wo = new WaveOutEvent())
{
wo.Init(radioStream);
wo.Play();
while (wo.PlaybackState == PlaybackState.Playing)
{
Thread.Sleep(1000);
}
}
Works fine for playback. However I need to take that as float samples converted to 48 KHz mono to be sent off as a bytestream elsewhere. So I do this:
//Convert the wave to 48 khz, mono
str1 = new WaveFormatConversionProvider(new WaveFormat(48000,1), radioStream);
//Convert the converted wave to floats
str = new Wave16ToFloatProvider(str1);
Under previous iterations of this I would do something like this to retrieve the data:
while ((readResultCount = str.Read(result, 0, result.Length)) > 0)
{
float[] output = new float[result.Length / sizeof(float)];
Buffer.BlockCopy(result, 0, output, 0, sizeof(float) * output.Length);
au.EnqueueEncodeVoiceData(output);
}
However, this results in audio output that skips worse than a CD that's been ran over.
I've tried to see if there was any way to determine if there was data enough in the buffer and avoid reading the stream if not, but there doesn't seem to be any facilities to do this.
What am I doing wrong?
Ah, here's a matter of hidden knowledge.
The loop itself was being called in a timer event tick, which meant that multiple instances of the loop could (and would) get called, since the Read operation is apparently blocking until either the requested data is received or the stream ends for whatever reason.
Since the timer ticked indiscriminately of whether the loop had exited or not, it was running over its own toes.
I resolved this by removing the timer tick and shoving the routine into a Task.

Fast way to compare 2 byte arrays

I am uploading jpeg images as fast as i can to a web service (it is the requirement I have been given).
I am using async call to the web service and I calling it within a timer.
I am trying to optimise as much as possible and tend to use an old laptop for testing. On a normal/reasonable build PC all is OK. On the laptop I get high RAM usage.
I know I will get a higher RAM usage using that old laptop but I want to know the lowest spec PC the app will work on.
As you can see in the code below I am converting the jpeg image into a byte array and then I upload the byte array.
If I can reduce/compress/zip the bye array then I am hoping this will be 1 of the ways of improving memory usage.
I know jpegs are already compressed but if I compare the current byte array with the previous byre array then uploading the difference between this byte arrays I could perhaps compress it even more on the basis that some of the byte values will be zero.
If I used a video encoder (which would do the trick) I would not be real time as much I would like.
Is there an optimum way of comparing 2 byte arrays and outputting to a 3rd byte array? I have looked around but could not find an answer that I liked.
This is my code on the client:
bool _uploaded = true;
private void tmrLiveFeed_Tick(object sender, EventArgs e)
{
try
{
if (_uploaded)
{
_uploaded = false;
_live.StreamerAsync(Shared.Alias, imageToByteArray((Bitmap)_frame.Clone()), Guid.NewGuid().ToString()); //web service being called here
}
}
catch (Exception _ex)
{
//do some thing but probably time out error here
}
}
//web service has finished the client invoke
void _live_StreamerCompleted(object sender, AsyncCompletedEventArgs e)
{
_uploaded = true; //we are now saying we start to upload the next byte array
}
private wsLive.Live _live = new wsLive.Live(); //web service
private byte[] imageToByteArray(Image imageIn)
{
MemoryStream ms = new MemoryStream();
imageIn.Save(ms,System.Drawing.Imaging.ImageFormat.Jpeg); //convert image to best image compression
imageIn.Dispose();
return ms.ToArray();
}
thanks...
As C.Evenhuis said - JPEG files are compressed, and changing even few pixels results in complettly differrent file. So - comparing resulting JPEG files is useless.
BUT you can compare your Image objects - quick search results in finding this:
unsafe Bitmap PixelDiff(Bitmap a, Bitmap b)
{
Bitmap output = new Bitmap(a.Width, a.Height, PixelFormat.Format32bppArgb);
Rectangle rect = new Rectangle(Point.Empty, a.Size);
using (var aData = a.LockBitsDisposable(rect, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb))
using (var bData = b.LockBitsDisposable(rect, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb))
using (var outputData = output.LockBitsDisposable(rect, ImageLockMode.ReadWrite, PixelFormat.Format32bppArgb))
{
byte* aPtr = (byte*)aData.Scan0;
byte* bPtr = (byte*)bData.Scan0;
byte* outputPtr = (byte*)outputData.Scan0;
int len = aData.Stride * aData.Height;
for (int i = 0; i < len; i++)
{
// For alpha use the average of both images (otherwise pixels with the same alpha won't be visible)
if ((i + 1) % 4 == 0)
*outputPtr = (byte)((*aPtr + *bPtr) / 2);
else
*outputPtr = (byte)~(*aPtr ^ *bPtr);
outputPtr++;
aPtr++;
bPtr++;
}
}
return output;
}
If your goal is to find out whether two byte arrays contain exactly the same data, you can create an MD5 hash and compare these as others have suggested. However in your question you mention you want to upload the difference which means the result of the comparison must be more than a simple yes/no.
As JPEGs are already compressed, the smallest change to the image could lead to a large difference in the binary data. I don't think any two JPEGs contain binary data similar enough to easily compare.
For BMP files you may find that changing a single pixel affects only one or a few bytes, and more importantly, the data for the pixel at a certain offset in the image is located at the same position in both binary files (given that both images are of equal size and color depth). So for BMPs the difference in binary data directly relates to the difference in the images.
In short, I don't think obtaining the binary difference between JPEG files will improve the size of the data to be sent.

Playing sounds simultaneously on windows phone

At first, I want to say that I had read all topics related to my problem here, on stackoverflow (and of course googled), but those research provided no solution to my problem.
I'm writing app for Windows Phone and I need to play two sounds simultaneously, but this code doesn't work, because there is slight, but noticeable dealy between two sounds, and there must be NO perceptible delay in my project.
Stream s1 = TitleContainer.OpenStream("C.wav");
Stream s2 = TitleContainer.OpenStream("C1.wav");
SoundEffectInstance sci = sc.CreateInstance();
SoundEffectInstance sci1 = sc1.CreateInstance();
sci.Play();
sci1.Play();
I also tried to perform a simple mix of two wav files, but it doesn't work for a reason that I don't know. (ArgumentException - Ensure that the specified stream contains valid PCM mono or stereo wave data. - is thrown, when calling SoundEffect.FromStream(WAVEFile.Mix(s1, s2));
public static Stream Mix(Stream in1,Stream in2)
{
BinaryWriter bw;
bw = new BinaryWriter(new MemoryStream());
byte[] header = new byte[44];
in1.Read(header, 0, 44);
bw.Write(header);
in2.Seek(44, SeekOrigin.Begin);
BinaryReader r1 = new BinaryReader(in1);
BinaryReader r2 = new BinaryReader(in2);
while (in1.Position != in1.Length)
{
bw.Write((short)(r1.ReadInt16() + r2.ReadInt16()));
}
r1.Dispose();
r2.Dispose();
bw.BaseStream.Seek(0, SeekOrigin.Begin);
return bw.BaseStream;
}
Stream s1 = TitleContainer.OpenStream("C.wav");
Stream s2 = TitleContainer.OpenStream("C1.wav");
s3 = SoundEffect.FromStream(WAVEFile.Mix(s1, s2));
So, does anyone know how to play two sounds at the time?
So your first solution SHOULD work. I have another solution that is very similar with a twist that I KNOW works.
static Stream stream1 = TitleContainer.OpenStream("soundeffect.wav");
static SoundEffect sfx = SoundEffect.FromStream(stream1);
static SoundEffectInstance soundEffect = sfx.CreateInstance();
public void playSound(){
FrameworkDispatcher.Update();
soundEffect.Play();
}
The reason your second solution didnt work is because there are very specific file formats that the windows phone can play.
List of supported formats
http://msdn.microsoft.com/en-us/library/windowsphone/develop/ff462087(v=vs.105).aspx
Reference for this code is my blog
http://www.anthonyrussell.info/postpage.php?name=60
Edit
You can see the above solution working here
http://www.windowsphone.com/en-us/store/app/xylophone/fe4e0fed-1130-e011-854c-00237de2db9e
Edit#2
In response to the comment below that this code doesn't work I have also posted a working, published app on my blog that implements this very code. It's called Xylophone, it's free and you can download the code here at the bottom of the page.
http://anthonyrussell.info/postpage.php?name=60

.NET - Bitmap.Save ignores Bitmap.SetResolution on Windows 7

I'm writing a .NET 4 application that imports and saves images for printing. It's important that the saved images resolution (DPI not pixel dimensions) be set to the value we specify so they print correctly.
Some of the images we import come without the resolution value (bad EXIF when they were generated), so we have to correct that before writing them. We use Bitmap.SetResolution for that. It works fine on XP and Windows 8, but when we write (Bitmap.Save) the images on Windows 7, they are always written with the original resolution meta info, ignoring SetResolution.
Here's a test we made, works on XP and 8, not on 7.
string originalFile = #"D:\temp\img\original_img.jpg";
string newFile = #"D:\temp\img\new_img.jpg";
Bitmap bitmap = (Bitmap)Image.FromFile(originalFile);
bitmap.SetResolution(200, 200);
bitmap.Save(newFile, ImageFormat.Jpeg);
Image image = Image.FromFile(newFile);
int dpiX = (int)Math.Round(image.HorizontalResolution, MidpointRounding.ToEven);
int dpiY = (int)Math.Round(image.VerticalResolution, MidpointRounding.ToEven);
Console.WriteLine("DPI is {0} x {1}", dpiX, dpiY);
Before saving, debug always shows the correct resolution assigned by SetResolution, the saved image is where the problem is.
This is probably what was reported here:
http://social.msdn.microsoft.com/Forums/vstudio/en-US/62368caa-05f4-4798-9c59-5d82f881a97c/systemdrawingbitmapsetresolution-is-completely-broken-on-windows-7?forum=netfxbcl
But the issue there seems to remain unsolved. Is there really no way to just make it work? Do I have to use extra libraries for this?
I've found a workaround that will do the job. It's not elegant but...
Instead of applying the resolution to the original image, make a copy of it and work on the copy:
Bitmap bitmap = (Bitmap)Image.FromFile(originalFile);
Bitmap newBitmap = new Bitmap(bitmap)
newBitmap.SetResolution(200, 200);
newBitmap.Save(newFile, ImageFormat.Jpeg);
Now it works on Windows 7. Go figure.
I like Hans Passant's idea, though, it's cleaner. I don't know if what I did messes up with the image, if there is recompression or not.
Hmya, this is a bug in a Windows component. The Windows group is always very reluctant to get bugs like this fixed, breaking changes are postponed to a next Windows version. It did get fixed in Windows 8. Do consider how unusual it is what you are doing, the DPI of an image should always be set by the device that recorded the image. Like the camera or scanner, they never get this wrong. There just isn't any device around that has a 200 dots-per-inch resolution.
If you are desperate enough to find a workaround then you could consider patching the file itself. Not hard to do for a JPEG file, the fields in the file header are pretty easy to get to:
using System.IO;
...
public static void SetJpegResolution(string path, int dpi) {
using (var jpg = new FileStream(path, FileMode.Open, FileAccess.ReadWrite, FileShare.None))
using (var br = new BinaryReader(jpg)) {
bool ok = br.ReadUInt16() == 0xd8ff; // Check header
ok = ok && br.ReadUInt16() == 0xe0ff;
br.ReadInt16(); // Skip length
ok = ok && br.ReadUInt32() == 0x4649464a; // Should be JFIF
ok = ok && br.ReadByte() == 0;
ok = ok && br.ReadByte() == 0x01; // Major version should be 1
br.ReadByte(); // Skip minor version
byte density = br.ReadByte();
ok = ok && (density == 1 || density == 2);
if (!ok) throw new Exception("Not a valid JPEG file");
if (density == 2) dpi = (int)Math.Round(dpi / 2.56);
var bigendian = BitConverter.GetBytes((short)dpi);
Array.Reverse(bigendian);
jpg.Write(bigendian, 0, 2);
jpg.Write(bigendian, 0, 2);
}
}

Categories

Resources