Develop Reference

C#, DataGridView not garbage collecting properly?

I've been looking at this problem the last two days and I'm all out of ideas, so I'm hoping someone might have some insight into what exactly is going on.
In brief, the program loads in a CSV file (with 240K lines, so not a ton) into a DataGridView, does a bunch of different operations on the data, and then dumps it into an Excel file for easy viewing and graphic and what not. However, as I iterate over the data in the DataGridView, ram usage constantly increases. I trimmed down the code to the simplest form and it's still happened and I don't quite understand why.
try
{
string conStr = #"Driver={Microsoft Text Driver (*.txt; *.csv)};Dbq=C:\Downloads;Extensions=csv,txt";
OdbcConnection conn = new OdbcConnection(conStr);
//OdbcDataAdapter da = new OdbcDataAdapter("Select * from bragg2.csv", conn);
OdbcDataAdapter da = new OdbcDataAdapter("Select * from bragg4.csv", conn);
DataTable dt = new DataTable("DudeYa");
da.Fill(dt);
dgvCsvFile.DataSource = dt;
da.Dispose();
conn.Close();
conn.Dispose();
}
catch (Exception e) { }
string sDeviceCon;
for (int i = 0; i < dgvCsvFile.Rows.Count; i++)
{
if (dgvCsvFile.Rows[i].Cells[48].Value != DBNull.Value)
sDeviceCon = (string)dgvCsvFile.Rows[i].Cells[48].Value;
else
sDeviceCon = "";
if ((sDeviceCon == null) || (sDeviceCon == ""))
continue;
}
When I enter that for loop, the program is using 230 megs. When I finish it, I'm using 3 gigs. At this point I'm not even doing anything with the data, I'm literally just looking at it and then moving on.
Futhermore, if I add these lines after it:
dgvCsvFile.DataSource = null;
dgvCsvFile.Rows.Clear();
dgvCsvFile.Columns.Clear();
dgvCsvFile.Dispose();
GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
The RAM usage stays the same. I can't figure out how to actually get the memory back without closing the program.
I've downloaded various memory profiling tools to try and figure out exactly whats going on:
Using VMMap from Sys Internals to take periodic snapshots, there were instances of the Garbage Collector running that were filling up. Each later snapshot I took usually had more Garbage Collectors running with all the previous ones using ~260MB of ram.
Using .NET Memory Profiler, I was able to get a little bit more detailed information. According to that, I have approximately 4 million instances of PropertyStore, DataGrieViewTextBoxCell and PropertyStore.IntegerEntry[]. All of them appear to have 1 reference and are existing uncollected in the Garbage Collector.
I ran the program in x86 mode and it caused an out of memory exception (not terribly unexpected). It throws the error inside of DataGridViewTextBoxCell.Clone(), which is in turn inside a DataGridViewRow.CloneCells() and DataGridViewRow.Clone(). So obviously it's doing a whole bunch of cloning to access the cell data... but why is it never being cleaned up? What could possibly be holding onto the reference for them?
I also tried getting the data into the DataSource in a different way using the CachedCsvReader from LumenWorks.Framwork.IO with no change in the functionality.
Is there something that I'm missing? Is there a setting I should be changing somewhere? Why aren't any of these things being disposed of?
Any help would be greatly appreciated, thanks!

FileFormatException loading tons of BitmapImages

I am programming in C# and WPF and trying to load tons of images, but after about 5 seconds I'm getting Exceptions, which are not thrown directly (Visual Studio doesn't stop the program) and are logged as catched in Visual Studio.
My code looks like this:
public Dictionary<string, System.Windows.Media.Imaging.BitmapImage> images { get; private set; }
public MainWindow()
{
images = new Dictionary<string, System.Windows.Media.Imaging.BitmapImage>();
foreach (string file in Directory.GetFiles(#"C:\path\to\thumbnails"))
{
string id = Path.GetFileName(file);
id = id.Substring(0, id.Length - 4);
var image = new System.Windows.Media.Imaging.BitmapImage(new Uri(file));
image.Freeze();
images.Add(id, image);
}
InitializeComponent();
}
There are more than 8k jpg pictures in the folder, which are 44 x 64 pixels big. When running the program I see the Memory usage in Visual Studio and if there was an Exception. As mentioned before, the first 5 seconds or so are fine, but then I am getting an Exception with catch for every following picture I think. It's hard to tell because they are thrown too fast and Visual Studio displays them too slow, so they pop up in packs.
When the Exceptions start to show up, the Memory usage is rising very slowly compared to before, but if I let the program finish the loading all pictures can be displayed without problems. It also looks like Visual Studio is overwhelmed by the amount of Exceptions causing the Logger to be slower than real time. 1 second in Visual Studio seems to become 6 real time seconds and it's taking more than a minute of these "fake" seconds.
The Exception is the following (I am from Germany):
Ausnahme ausgelöst: 'System.IO.FileFormatException' in PresentationCore.dll ("Der Bitmap-Farbkontext ist ungültig.")
It looks like the picture is corrupted or has some errors, but I wasn't able to retrieve a filename and when I was, I tested the file seperately and it was okay.
I am just looking for a way to load all the images in normal speed.
Thanks in advance!
Update 1:
Somehow I disabled the "Just My Code" option in Visual Studio, causing to show exceptions outside my code and slowing Visual Studio down. Now it is running as fast as in the modification mentioned in the comments and takes about a minute to load all images.
I'm still wondering, if there is a way to load the images faster. All I am doing with them is displaying them.

C# Possible Memory Leak?

So, I have an app, written in C# (vs2010), performing OCR using the tesseract 3.02 dll and Charles Weld's terreract .net wrapper.
I think I have a memory leak and it seems to be in the area of code where the Pix object is allocated. I am taking a PDF, converting that to a grayscale PNG, then loading that into a Pix object for OCR. When it works, it works really well. Image is large in size (5100 or so pixels in each dim) but not so large in size (only 500k or so).
My code:
Init engine at app startup:
private TesseractEngine engine = new TesseractEngine(#"./tessdata/", "eng+fra", EngineMode.Default);
Method to convert PDF to PNG, then calls:
// Load the image file created earlier into a Pix object.
Pix pixImage = Pix.LoadFromFile(Path.Combine(textBoxSourceFolder.Text, sourceFile));
And then calls the following:
// Perform OCR on the image referenced in the Pix object.
private String PerformImageOCR(Pix pixImage)
{
int safety = 0;
do
{
try
{
// Deskew the image.
pixImage = pixImage.Deskew();
//pixImage.Save(#"c:\temp\img_deskewed.png", ImageFormat.Png); // Debugging - verify image deskewed properly to allow good OCR.
string text = "";
// Use the tesseract OCR engine to process the image
using (var page = engine.Process(pixImage))
{
// and then extract the text.
text = page.GetText();
}
return text;
}
catch (Exception e)
{
MessageBox.Show(string.Format("There was an error performing OCR on image, Retrying.\n\nError:\n{0}", e.Message), "Error", MessageBoxButtons.OK);
}
} while (++safety < 3);
return string.Empty;
}
I have observed that memory usage jumps by about 31MB when the Pix object is created, then jumps again while OCR is being performed, then finally settles about 33MB higher than before it started. ie: if app, after loading, was consuming 50MB, loading the Pix object causes the memory usage to jump to about 81MB. Performing OCR will see it spike to 114+MB, then, once the process is complete and the results saved, the memory usage settles to about 84MB. Repeating this over many files in a folder will eventually cause the app to barf at 1.5GB or so consumed.
I think my code is okay, but there's something somewhere that's holding onto resources.
The tesseract and leptonica dlls are written in C and I have recompiled them with VS2010 along with the latest or recommended image lib versions, as appropriate. What I'm unsure of, is how to diagnose a memory leak in a C dll from a C# app using visual studio. If I were using Linux, I'd use a tool such as valgrind to help me spot the leak, but my leak sniffing skills on the windows side are sadly lacking. Looking for advice on how to proceed.

Reading your code here I do not see you disposing your Pix pixImage anywhere? That's what is taking up all the resources when you are processing x images.
Before you return your string result you should call the dispose method on your pixImage. That should reduce the amount of resources used by your program.

I'm not familliar with Tesseract or the wrapper, but for memory profiling issues, if you have Visual Studio 2012/2013, you can use the Performance Wizard. I know it's available in Ultimate, but not sure on other versions.
http://blogs.msdn.com/b/dotnet/archive/2013/04/04/net-memory-allocation-profiling-with-visual-studio-2012.aspx
It's either something in your code or something in the wrapper is not disposing an unmanaged object properly. My guess would be it's in the wrapper. Running the Performance Wizard or another C# memory profiler (like JetBrains DotTrace) may help you track it down.

Trying to load 150+ grayscale 4096 x 4096 bitmaps. Need help getting around the 2GB limit, I think

Solved: I assumed, incorrectly, that Visual Studio 2012 defaulted to building 64-bit apps when making new projects. Under Solution Properties, Build tab, there is a checkbox marked, "Prefer 32-bit". When I unchecked this checkbox and rebuilt my solution, I was able to load and process over 200 bitmaps at a time. Thanks to Mike Trusov for so politely pointing out the solution to me.
Original question: I've run into a small problem. I have more than 150 grayscale bitmaps that I want to load into RAM on an 8 GB system, but I can't seem to get past 50 or so without throwing an exception. I've tried loading the bitmaps into an array, which is a lost cause, of course, because .NET has a 2GB limit. But it failed long before 2GB worth of bitmaps were loaded. So I tried loading them into a List<List<Bitmap>>, but even that fails at about the same place. At the moment of the exception, Task Manager says I have 3939 MB of Available RAM waiting to be filled. Very strange.
I don't need these bitmaps to be contiguous in RAM. They can be a scattered bunch of 16 MB allocations for all I care. I just want to be able to fill available RAM with a bunch of bitmaps. That's all.
The exception has at various times been an OutOfMemory exception or an ArgumentException, depending on how much available RAM I had when I started the program. In either case, the stack trace dies inside System.Drawing.Bitmap..ctor(String filename). There is nothing wrong with the specific file being loaded at the time of the exception. In fact, when I have it load a different (or even overlapping) set of bitmaps, the error occurs at the same iteration.
Does anyone have a clue they can lend me on how to do this? Am I running into the .NET 2GB limit in some strange way?
To respond to a few questions and comments: I'm using Visual Studio 2012, .NET 4.5, 64-bit Windows 7, on a computer with 8 GB of RAM. Yes, I need all those bitmaps in RAM at the same time for a variety of reasons (performance, image processing reasons, etc). I have pondered using gcAllowVeryLargeObjects, but I don't need or want all my bitmaps in a long chunk of contiguous memory. I would much rather each Bitmap used its own separate memory allocation. Besides, if I had a machine with 64 GB of RAM, it would be absurd to be limited to even 150 Bitmaps of that size. Why won't these bitmaps load without throwing an OutOfMemoryException?
To me, it seems that .NET is trying to keep all Bitmaps in a single 2 GB region. If there was a way to get each Bitmap to (saying more than I know here) have its own separate address space, that might solve the problem. To invoke the language of the long ago days of MS-DOS, I want to allocate and access far memory using a long pointer, not have all my data stuck in a single near segment.
Here is the array code:
List<String> imageFiles; // List of .bmp filenames.
Bitmap[] bmps = new Bitmap[100]; // Stores/accesses the Bitmaps.
private void goButton_Click(object sender, EventArgs e)
{
int i;
// Load the bitmaps
if (bmps[0] == null)
{
// Load the list of bitmap files.
imageFiles = Directory.EnumerateFiles(#"C:\Temp", "*.bmp", SearchOption.TopDirectoryOnly).ToList();
// Read bitmap files
for (i = 0; i < bmps.Length; ++i)
{
bmps[i] = new Bitmap(imageFiles[i]); // <-- Exception occurs here when i == 52 or so.
}
}
}
Here is the List> code:
List<String> imageFiles; // List of .bmp filenames.
List<List<Bitmap>> bmps = new List<List<Bitmap>>(100); // Stores/accesses the Bitmaps.
private void goButton_Click(object sender, EventArgs e)
{
int i;
// Load the bitmaps
if (bmps.Count == 0)
{
// Load the list of bitmap files.
imageFiles = Directory.EnumerateFiles(#"C:\Temp", "*.bmp", SearchOption.TopDirectoryOnly).ToList();
// Read bitmap files
for (i = 0; i < 100; ++i)
{
// Load the bitmap into temporary Bitmap b.
Bitmap b = new Bitmap(imageFiles[i]); // <-- Exception occurs here when i == 52 or so.
// Create and add a List<Bitmap> that will receive the clone of Bitmap b.
bmps.Add(new List<Bitmap>(1));
// Clone Bitmap b and add that cloned Bitmap to the Bitmap of List<Bitmap>.
bmps[i].Add((Bitmap)b.Clone());
// Dispose Bitmap b.
b.Dispose();
}
}
}

There should be no issues loading more than 2gb of bitmaps into memory in a 64bit app (check project settings - might have to create a new configuration for Any CPU based on x86) running on 64bit OS (I'm guessing you are). Also a simple list should work:
var imageFiles = Directory.EnumerateFiles(#"C:\Temp", "*.bmp", SearchOption.TopDirectoryOnly).ToList();
var lst = new List<Bitmap>();
foreach (var imageFile in imageFiles)
{
lst.Add(new Bitmap(imageFile));
}

Do they ALL have to be loaded at the same time? Could you load say, 20 of them, then while you are processing or displaying those, you have a background thread prep the next 20.

OutOfMemoryException handling and Windows Media Player SDK

How it must work:
WindowsMediaPlayer windowsMediaPlayer = new WindowsMediaPlayer();
IWMPMediaCollection collection = windowsMediaPlayer.mediaCollection;
IWMPMedia newMedia = collection.add(path); //causes OutOfMemoryException after some thousands method's iterations
I've tried to avoid it this way:
try
{
newMedia = collection.add(path);
return newMedia;
}
catch (Exception)
{
collection = null;
GC.Collect();
GC.WaitForPendingFinalizers();
WindowsMediaPlayer windowsMediaPlayer = new WindowsMediaPlayer();
IWMPMediaCollection collectionNew = windowsMediaPlayer.mediaCollection;
return CreateNewMedia(collectionNew, path);
}
But this does not work – I still get infinite exception loop inside catch.

You can not handle OutOfMemoryException like ordinary one. The reason you may have handling that, is just, in some way, to save the state of application, in order to provide to the consumer of your application a way to recover lost data.
What I mean is that there is no meaning calling GC.Collect or whatever, cause application is going to dead anyway, but CLR kindly give you notification about that before.
So to resolve this issue, check the memory consumption of your application, that on 32bit machine has to be something about 1.2GB of RAM, or control the quantity of the elements in collections you have, that can not exceed, for ordinary list, 2GB of memory on 32bit and on 64bit too.