Good afternoon,
I have some text files containing a list of (2-gram, count) pairs collected by analysing a corpus of newspaper articles which I need to load into memory when I start a given application I am developing. To store those pairs, I am using a structure like the following one:
private static Dictionary<String, Int64>[] ListaDigramas = new Dictionary<String, Int64>[27];
The ideia of having an array of dictionaries is due to efficiency questions, since I read somewhere that a long dictionary has a negative impact on performance. That said, every 2-gram goes into the dictionary that corresponds to it's first character's ASCII code minus 97 (or 26 if the first character is not a character in the range from 'a' to 'z').
When I load the (2-gram, count) pairs into memory, the application takes an overall 800Mb of RAM, and stays like this until I use a program called Memory Cleaner to free up memory. After this, the memory taken by the program goes down to the range 7Mb-100Mb, without losing functionality (I think).
Is there any way I can free up memory this way but without using an external application? I tried to use GC.Collect() but it doesn't work in this case.
Thank you very much.
You are using a static field so chances are once it is loaded it never gets garbage collected, so unless you call the .Clear() method on this dictionary it probably won't be subject to garbage collection.
It is fairly mysterious to me how utilities like that ever make it onto somebody's machine. All they do is call EmptyWorkingSet(). Maybe it looks good in Taskmgr.exe, but it is otherwise just a way to keep the hard drive busy unnecessarily. You'll get the exact same thing by minimizing the main window of your app.
I don't know the details of how memory cleaner works, but given that it's unlikely to know the inner workings of a programs memory allocations, the best it can probably do is just cause pages to be swapped out to disk reducing the apparent memory usage of the program.
Garbage collection won't help unless you actually have objects you aren't using any more. If you are using your dictionaries, which the GC considers that you are since it is a static field, then all the objects in them are considered in use and must belong to the active memory of the program. There's no way around this.
What you are seeing is the total usage of the application. This is 800MB and will stay that way. As the comments say, memory cleaner makes it look like the application uses less memory. What you can try to do is access all values in the dictionary after you've run the memory cleaner. You'll see that the memory usage goes up again (it's read from swap).
What you probably want is to not load all this data into memory. Is there a way you can get the same results using an algorithm?
Alternatively, and this would probably be the best option if you are actually storing information here, you could use a database. If it's cumbersome to use a normal database like SQLExpress, you could always go for SQLite.
About the only other idea I could come up with, if you really want to keep your memory usage down, would be store the dictionary in a stream and compress it. Factors to consider would be how often you're accessing/inflating this data, and how compressible the data is. Text from newspaper articles would compress extremely well, and the performance hit might be less than you'd think.
Using an open-source library like SharpZipLib ( http://www.icsharpcode.net/opensource/sharpziplib/ ), your code would look something like:
MemoryStream stream = new MemoryStream();
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(stream, ListaDigramas);
byte[] dictBytes = stream.ToArray();
Stream zipStream = new DeflaterOutputStream(new MemoryStream());
zipStream.Write(dictBytes, 0, dictBytes.Length);
Inflating requires an InflaterInputStream and a loop to inflate the stream in chunks, but is fairly straightforward.
You'd have to play with the app to see if performance was acceptable. Keeping in mind, of course, that you'll still need enough memory to hold the dictionary when you inflate it for use (unless someone has a clever idea to work with the object in its compressed state).
Honestly, though, keeping it as-is in memory and letting Windows swap it to the page file is probably your best/fastest option.
Edit
I've never tried it, but you might be able to serialize directly to the compression stream, meaning the compression overhead is minimal (you'd still have the serialization overhead):
MemoryStream stream = new MemoryStream();
BinaryFormatter formatter = new BinaryFormatter();
Stream zipStream = new DeflaterOutputStream(new MemoryStream());
formatter.Serialize(zipStream, ListaDigramas);
Thank you very much for all the answers. The data actually needs to be loaded during the whole running time of the application, so based on your answers I think there is nothing better to do... I could perhaps try an external database, but since I already need to deal with two other databases at the same time, I think it is not a good idea.
Do you think it is possible to be dealing with three databases at the same time and do not lose on performance?
If you are disposing of your applications resources correctly then the actual used memory may not be what you are seeing (if verifying through Task Manager).
The Garbage Collector will free up the unused memory at the best possible time. It usually isn't really a good idea to force collection either...see this post
"data actually needs to be loaded during the whole running time of the application" - why?
Related
Like GuyFawkes, I would like to use MemoryStream to store a large amount of data, but keep encountering the 'out of memory' exceptions.
TomTom's answer is what I would like to do - use an implementation that does not require a contiguous block - but I'm wondering if there is already a free implementation available, to save me writing one myself?
Does anyone know of a good, free re-implementation of MemoryStream that can work with large streams?
EDIT:
The MemoryMappedFile solution is very interesting and I will be remembering it for other projects, however as Henk says, it strays too far from the abstraction that MemoryStream is aiming for. Specifically, the requirement of a known capacity.
The data that the replacement shall handle will in some cases be very large, but in others relatively small (and no we don't know which it will be until its too late ;)); further, many instances of the class will be in existence at the same time.
Ultimately the work required to use MemoryMappedFiles (to determine an appropriate size for each one) would be equivalent to that of implementing TomTom's solution.
Here is my implementation in case anyone needs it; I will leave this question open for a bit in case someone still responds with anything better.
http://www.codeproject.com/Articles/348590/A-replacement-for-MemoryStream
You yould create a MemoryMappedFile without a file, i.e. one that lives in system memory.
The DelayAllocatePages option delays allocations until the memory is actually needed. You need to specify a maximum capacity in advance though. Use the CreateViewStream Method to create a stream.
Not exactly a re-implementation of MemoryStream, but consider whether you can use a Memory Mapped File for your requirement.
Memory Mapped Files can solve many of the classes of problems that large memory buffers can solve, are very efficient, and are supported directly by .NET.
I have started memory profiling our app because we have recently received several reports about performance and out of memory exceptions. The app is developed in C# .Net Winforms (.Net Framework 2.0)
When the application started, the ANT profiler shows 17.7 MB objects live in Gen 2.
When the app starts, it reads the 77000+ zipcodes from a xml serialized file on the disk and saves in a Hashtable. Please see the sample code below
public Class ZipCodeItem
{
private string zipCode;
private string city;
private string state;
private string county;
private int tdhCode;
private string fipsCounty;
private string fipsCity;
Public ZipCodeItem()
{
// Constructor.. nothing interesting here
}
// Bunch of public getter/setter properties
}
Here is the static class that reads the serialized zip data from a file on disk and loads the zipcodes.
internal sealed class ZipTypes
{
private static readonly Hashtable zipCodes = new Hashtable();
public static ArrayList LookupZipCodes(string zipCode)
{
if (zipCodes.Count == 0)
LoadZipCodes();
ArrayList arZips = new ArrayList();
// Search for given zip code and return the matched ZipCodeitem collection
if (zipCodes.ContainsKey(zipCode))
{
// Populate the array with the matched items
}
// Omitted the details to keep it simple
return arZips;
}
private static bool LoadZipCodes()
{
using (FileStream stream = new FileStream(filename, FileMode.Open, FileAccess.Read))
{
// unzip it.. Omitted the details to keep it simple
// Read the zipcodes from the flat xml file on disk and load the local zipCodes HashTable
}
}
}
This Class and the corr. ZipCodes are accessed all over the app.
About 14 meg out of 17.7 meg of Gen 2 objects are either zipCodeItems or Its child String classes.
I would like to change my code to some how NOT to keep these 77000+ zipcode item objects in memory (in a hashTable), but give the mapped zipCode items when the app needs it.
Any suggestions How to resolve this issue? Thanks in advance.
I'm going to avoid answering the question directly in hopes of providing a more useful answer because I don't believe that the ~14MB associated with that hash is actually causing a problem.
You say that you are using the ANTS memory profiler. That's a great tool and I have used it before, so perhaps I can help you track down the real problem here.
We can safely assume that your hash is not causing the OutOfMemoryException as it is nowhere near big enough to do so. Keep it as it is now, except for two small changes:
Use a strongly typed Dictionary<K,V> instead of Hashtable. Hashtable was essentially deprecated once .NET 2.0 introduce generics. You can also go ahead and replace that ArrayList with a List<T>.
Instead of performing a Contains and then looking up the value in the hash, use TryGetValue instead. This cuts the number of hash table lookups in half. Now that may not be a performance bottleneck in your app, but I don't think it amounts to premature optimization either.
Now, onto the crux of the issue...
You have your profiler results. Go back and look at where your memory is being allocated. In order, check these things:
Is .NET holding most of the memory or is it native code (possibly creating a lot of objects that implement IDisposable and not calling Dispose() on them in a timely manner.) If it's the latter you probably know where to look.
How does the Large Object Heap (LOH) look? Is most of the memory allocated there? Many large allocations can fragment the LOH and it may not be compacted for quite some time. ANTS will tell you this at the top right of the results overview page.
Event Handlers. When an object subscribes to an event a reference to the subscriber (method) is stored by the subscribee (by the MultiCastDelegate, i.e., the event object). This may cause object lifetimes to never end and, over some amount of time, this may be adding up memory wise. You need to make sure that, if there are objects being created and then going out of scope, it also unsubscribes from any events it had previously subscribed to. Static events can be a killer here.
Use ANTS to track object lifetimes. Similar to the above, make sure that there are no objects being kept alive inadvertently due to stale references. This can occur more easily than you may think. Again, look in areas where a relatively large number of objects are created and go out of scope, but also instances where other objects maintain references to them. ANTS can show you this in the object graph.
That should at least give you a good picture of what memory is being allocated where. You will likely need to run your program for some time under the profiler and simply watch memory usage. If it steadily goes up then you can perform the steps I listed above in an attempt to isolate which objects are piling up.
You are going to need some sort of storage along with an easy access mechanism.
I would suggest using some form of SQL based database.
We've had good success using SQL Compact Edition as it can be deployed without pre-requisites (aka a "private deployment"). Subsequently the integration story for querying is really tight - for example, using Linq to SQL is dead easy.
You could also look at SQL Lite or other providers. I would steer you away from a SQL Express dependency though just because it will be too heavy for these needs.
Also, seeing as though you would now be caching the zip codes in a database, you might also consider looking at some sort of "sync process" (say daily) for downloading and parsing your input XML file that you mentioned above (assuming the XML file is retrieved from some web service) or else you could just deploy the database already populated with the data.
If your users are complaining about out of memory exceptions and you're addressing something that's taking 15MB of memory, you're looking in the wrong place.
The rest of my answer assumes that the 15MB really does matter.
Having said that, I would like to offer an alternative to the SQL solutions already proposed (which are good solutions, depending on your situation, if you determine that you really don't want to load the 15MB into memory).
We load 3GB of IP data into the process space of our web servers. However, most of the IP addresses are not accessed most of the time (especially since we have a strong geographic bias in the user base, but still need to have the data available for visitors from less frequently seen parts of the world).
To keep the memory footprint small and the access very fast, we made use of memory mapped files. While there's support for them built-in to .NET 4, you can still use them via interop calls in any previous .NET version.
Memory mapped files allow you to very rapidly map data from a file into memory in your process space. SQL adds overhead for things you probably don't need in this situation (transactional support, key constraints, table relationships, etc... all great features in general but unnecessary to solve this problem, and cost something in terms of performance and memory footprint).
When the application started, the ANT profiler shows 17.7 MB objects
live in Gen 2
We had a similar issue and we find it is ok to keep these details in memoray rather than loading it several times. BUT it should not load in the start up. Change the logic to load that at the first usage on demand. Because there can be use cases where it is not using at all. No point of keeping that chunk in Gen2 for such cases.
I am sure there may be sever memory leak issues than this one if you
have got memory issues in your application. ANT profiler is a good for
such cases :-)
I was considering using UnmanagedMemoryStream rather than MemoryStream for dealing with incoming (and perhaps outgoing?) packets in a network server. What I hope to achieve is less copying of values, and if possible, avoid copying to the heap (too much).
For example, for an incoming packet, one could do:
fixed (byte* p = &data) // where data comes from a socket receive
{
using (var stream = new UnmanagedMemoryStream(p, data.Length))
{
// do handling here...
}
}
Still, I'm not quite sure if there is any realistic benefit in doing this. Could anyone come with some feedback as to whether or not there would be any value in doing this, rather than using the good old managed MemoryStream?
Thanks in advance.
This sounds like premature optimization to me. Have you used MemoryStream and done some profiling to show that it caused you tangible, measurable performance problems?
I would stick with the conventional MemoryStream and save myself the headache until it became obvious (through profiling) that it was necessary to change.
Nope, you didn't improve what is already there. A byte[] is a reference type. You can simply pass it to the MemoryStream(byte[]) constructor and no data is getting copied. MS simply stores a reference to the same array.
In fact, you made it worse because you pinned the array. Getting a garbage collection to run inside the body of your snippet isn't unlikely, you are reading stuff from the array and are probably creating objects from the data, strings and what-not. The garbage collector needs to work around the pinned array, making its life considerably more difficult. This can actually affect the perf of your program for a while, compacting the heap is important to make the CPU cache efficient.
I have a very large set of binary files where several thousand raw frames of video are being sequentially read and processed, and I’m now looking to optimize it as it appears to be more CPU-bound than I/O-bound.
The frames are currently being read in this manner, and I suspect this is the biggest culprit:
private byte[] frameBuf;
BinaryReader binRead = new BinaryReader(FS);
// Initialize a new buffer of sizeof(frame)
frameBuf = new byte[VARIABLE_BUFFER_SIZE];
//Read sizeof(frame) bytes from the file
frameBuf = binRead.ReadBytes(VARIABLE_BUFFER_SIZE);
Would it make much of a difference in .NET to re-organize the I/O to avoid creating all these new byte arrays with each frame?
My understanding of .NET’s memory allocation mechanism is weak as I am coming from a pure C/C++ background. My idea is to re-write this to share a static buffer class that contains a very large shared buffer with an integer keeping track of the frame’s actual size, but I love the simplicity and readability of the current implementation and would rather keep it if the CLR already handles this in some way I am not aware of.
Any input would be much appreciated.
You don't need to init frameBuf if you use binRead.ReadBytes -- you'll get back a new byte array which will overwrite the one you just created. This does create a new array for each read, though.
If you want to avoid creating a bunch of byte arrays, you could use binRead.Read, which will put the bytes into an array you supply to it. If other threads are using the array, though, they'll see the contents of it change right in front of them. Be sure you can guarantee you're done with the buffer before reusing it.
You need to be careful here. It is very easy to get completely bogus test results on code like this, results that never repro in real use. The problem is the file system cache, it will cache the data you read from a file. The trouble starts when you run your test over and over again, tweaking the code and looking for improvements.
The second, and subsequent times you run the test, the data no longer comes off the disk. It is still present in the cache, it only takes a memory-to-memory copy to get it into your program. That's very fast, a microsecond or so of overhead plus the time needed to copy. Which runs at bus-speeds, at least 5 gigabytes per second on modern machines.
Your test will now reveal that you spend a lot of time on allocating the buffer and processing the data, relative from the amount of time spent reading the data.
This will rarely repro in real use. The data won't be in the cache yet, now the sluggy disk drive needs to seek the data (many milliseconds) and it needs to be read off the disk platter (a couple of dozen megabytes per second, at best). Reading the data now takes a good three of four magnitudes of time longer. If you managed to make the processing step twice as fast, your program will actually only run 0.05% faster. Give or take.
My application does a good deal of binary serialization and compression of large objects. Uncompressed the serialized dataset is about 14 MB. Compressed it is arround 1.5 MB. I find that whenever I call the serialize method on my dataset my large object heap performance counter jumps up from under 1 MB to about 90 MB. I also know that under a relatively heavy loaded system, usually after a while of running (days) in which this serialization process happens a few time, the application has been known to throw out of memory excpetions when this serialization method is called even though there seems to be plenty of memory. I'm guessing that fragmentation is the issue (though i can't say i'm 100% sure, i'm pretty close)
The simplest short term fix (i guess i'm looking for both a short term and a long term answer) i can think of is to call GC.Collect right after i'm done the serialization process. This, in my opinion, will garbage collect the object from the LOH and will do so likely BEFORE other objects can be added to it. This will allow other objects to fit tightly tightly against the remaining objects in the heap without causing much fragmentation.
Other than this ridiculous 90MB allocation i don't think i have anything else that uses a lost of the LOH. This 90 MB allocation is also relatively rare (arround every 4 hours). We of course will still have the 1.5 MB array in there and maybe some other smaller serialized objects.
Any ideas?
Update as a result of good responses
Here is my code which does the work. I've actually tried changing this to compress WHILE serializing so that serialization serializes to a stream at the same time and i don't get much better result. I've also tried preallocating the memory stream to 100 MB and trying to use the same stream twice in a row, the LOH goes up to 180 MB anyways. I'm using Process Explorer to monitor it. It's insane. I think i'm going to try the UnmanagedMemoryStream idea next.
I would encourage you guys to try it out if you wont. It doesn't have to be this exact code. Just serialize a large dataset and you will get surprising results (mine has lots of tables, arround 15 and lots of strings and columns)
byte[] bytes;
System.Runtime.Serialization.Formatters.Binary.BinaryFormatter serializer =
new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();
System.IO.MemoryStream memStream = new System.IO.MemoryStream();
serializer.Serialize(memStream, obj);
bytes = CompressionHelper.CompressBytes(memStream.ToArray());
memStream.Dispose();
return bytes;
Update after trying binary serialization with UnmanagedMemoryStream
Even if I serialize to an UnmanagedMemoryStream the LOH jumps up to the same size. It seems that no matter what i do, called the BinaryFormatter to serialize this large object will use the LOH. As for pre-allocating, it doesn't seem to help much. Say i pre-allocate say i preallocate 100MB, then i serialize, it will use 170 MB. Here is the code for that. Even simpler than the above code
BinaryFormatter serializer = new BinaryFormatter();
MemoryStream memoryStream = new MemoryStream(1024*1024*100);
GC.Collect();
serializer.Serialize(memoryStream, assetDS);
The GC.Collect() in the middle there is just to update the LOH performance counter. You will see that it will allocate the correct 100 MB. But then when you call the serialize, you will notice that it seems to add that on top of the 100 that you have already allocated.
Beware of the way collection classes and streams like MemoryStream work in .NET. They have an underlying buffer, a simple array. Whenever the collection or stream buffer grows beyond the allocated size of the array, the array gets re-allocated, now at double the previous size.
This can cause many copies of the array in the LOH. Your 14MB dataset will start using the LOH at 128KB, then take another 256KB, then another 512KB, etcetera. The last one, the one actually used, will be around 16MB. The LOH contains the sum of these, around 30MB, only one of which is in actual use.
Do this three times without a gen2 collection and your LOH has grown to 90MB.
Avoid this by pre-allocating the buffer to the expected size. MemoryStream has a constructor that takes an initial capacity. So do all collection classes. Calling GC.Collect() after you've nulled all references can help unclog the LOH and purge those intermediate buffers, at the cost of clogging the gen1 and gen2 heaps too soon.
Unfortunately, the only way I could fix this was to break up the data in chunks so as not to allocate large chunks on the LOH. All the proposed answers here were good and were expected to work but they did not. It seems that the binary serialization in .NET (using .NET 2.0 SP2) does its own little magic under the hood which prevents users from having control over memory allocation.
Answer then to the question would be "this is not likely to work". When it comes to using .NET serialization, your best bet is to serialize the large objects in smaller chunks. For all other scenarios, the answers mentioned above are great.
90MB of RAM is not much.
Avoid calling GC.Collect unless you have a problem. If you have a problem, and no better fix, try calling GC.Collect and seeing if your problem is solved.
Don't worry about LOH size jumping up. Worry about allocating/deallocating LOH. .Net very dumb about LOH -- rather than allocating LOH objects far away from regular heap, it allocates at next available VM page. I have a 3D app that does much allocate/deallocate of both LOH and regular objects -- the result (as seen in DebugDiag dump report) is that pages of small heap and large heap end up alternating throughout RAM, until there are no large chunks of the applications 2 GB VM space left. The solution when possible is to allocate once what you need, and then don't release it -- re-use it next time.
Use DebugDiag to analyze your process. See how the VM addresses gradually creep up towards 2 GB address mark. Then make a change that keeps that from happening.
I agree with some of the other posters here that you might want to try and use tricks to work with the .NET Framework instead of trying to force it to work with you via GC.Collect.
You may find this Channel 9 video helpful which discusses ways to ease pressure on the Garbage collector.
If you really need to use the LOH for something like a service or something that needs to be running for a long time, you need to use buffer pools that are never deallocated and that you can ideally allocate on start-up. This means you'll have to do your 'memory management' yourself for this, of course.
Depending on what you're doing with this memory, you might also have to p/Invoke over to native code for selected parts to avoid having to call some .NET API that forces you to put the data on newly allocated space in the LOH.
This is a good starting point article about the issues: https://devblogs.microsoft.com/dotnet/using-gc-efficiently-part-3/
I'd consider you very lucky if you GC trick would work, and it would really only work if there isn't much going on at the same time in the system. If you have work going on in parallel, this will just slightly delay the unevitable.
Also read up on the documentation about GC.Collect.IIRC, GC.Collect(n) only says that it collects no further than the generation n -- not that it actually ever GETS to generation n.