I'm trying to test the error handling of my code under limited memory situations.
I'm also keen to see how the performance of my code is affected in low memory situations where perhaps the GC has to run more often.
Is there a way of running a .Net applicataion (or NUnit test-suite) with limited memory? I know with Java you can limit the amount of memory the JVM has access to - is there something similar in .Net?
This is not an option in the CLR. Memory is managed very differently, there are at least 10 distinct heaps in a .NET process. A .NET program can use the entire virtual memory space available in a Windows process without restrictions.
The simplest approach is to just allocate the memory when your program starts. You have to be a bit careful, you cannot swallow too much in one gulp, the address space is fragmented due to it containing a mix of code and data at different addresses. Memory is allocated from the holes in between. To put a serious dent into the available address space, you have to allocate at least a gigabyte and that's not possible with a single allocation.
So just use a loop to allocate smaller chunks, say one megabyte at a time:
private static List<byte[]> Gobble = new List<byte[]>();
static void Main(string[] args) {
for (int megabyte = 0; megabyte < 1024; megabyte++)
Gobble.Add(new byte[1024 * 1024]);
// etc..
}
Note that this is very fast, the allocated address space is just reserved and doesn't occupy any RAM.
You can enlist your process into a Windows Job Object. You can set memory (and other) limits for the job. This is the cleanest and the only sane way to limit the amount of memory that your process can use.
Related
is there any option to lock or allocate memory in C#?
Scenario 1:
In my virtual machine, there is 16GB RAM, for a test I need to use 8GB RAM so 8GB will remain 'free' for the operating system and rest application
Scenario 2:
The same virtual machine with 16GB RAM, and now I need to use 14GB RAM.
For now, I create a memory leak function but this is not good cause it takes all memory from my virtual machine.
List<byte[]> memoryUsage = new List<byte[]>();
while (true)
{
try
{
memoryUsage.Add(new byte[1024]);
}
catch (OutOfMemoryException)
{
throw;
}
}
Allocate or Lock RAM that User (me) will input to file e.g
I want to allocate/lock 8GB RAM and program allocate/lock 8GB RAM and 8GB RAM will remain as 'free'
If you're running this on a Windows box, you need to keep in mind the concept of virtual memory. You can allocate - or leak memory - to your heart's content with C#, however if the underlying operating system deems safe to page the memory being used by your process to the paging file (assuming at least one such file is defined), it will do so.
Let's take your first scenario - the one where you want 8 GB of RAM allocated. Your code can do precisely that. However your OS can kick in and move some of the pages representing your allocated data in RAM to disk. There are several reasons why the memory manager would do this - take a look at some here (under the "Pages can be removed from a process working set..." paragraph). You'll thus be left with less used RAM that you originally intended.
To my understanding you're after a constant working set occupied by your process, and I'm not sure C# - even in an unsafe context - allows you to do that. You could try invoking Win32 functions that work at a low level, possibly using P/Invoke as stated here, but like I've said, I'm not sure it will work.
You'll also have to keep an eye out on the variable(s) that reference your allocated data. If the GC (Garbage Collector) decides the data you've allocated is no longer required from a point on (say because your remaining code no longer references it) it will happily reclaim it, again leaving you with less allocated memory that you've originally wanted.
You haven't really said anything about the target platform you've used to build this. An out-of-memory error will be thrown much earlier - after allocating only around 3 GB - by a process that was built for x86 (or AnyCPU + Prefer 32-bit), since that's how things work in Wow64.
If you don't really have to write the code that's doing the allocation yourself in C#, maybe you can merely invoke something like Testlimit (and play around with setting the minimum working set).
I noticed that my application runs out of memory quicker than it should. It creates many byte arrays of several megabytes each. However when I looked at memory usage with vmmap, it appears .NET allocates much more than needed for each buffer. To be precise, when allocating a buffer of 9 megabytes, .NET creates a heap of 16 megabytes. The remaining 7 megabytes cannot be used to create another buffer of 9 megabytes, so .NET creates another 16 megabytes. So each 9MB buffer wastes 7MB of address space!
Here's a sample program that throws an OutOfMemoryException after allocating 106 buffers in 32-bit .NET 4:
using System.Collections.Generic;
namespace CSharpMemoryAllocationTest
{
class Program
{
static void Main(string[] args)
{
var buffers = new List<byte[]>();
for (int i = 0; i < 130; ++i)
{
buffers.Add(new byte[9 * 1000 * 1024]);
}
}
}
}
Note that you can increase the size of the array to 16 * 1000 * 1024 and still allocate the same amount of buffers before running out of memory.
VMMap shows this:
Also note how there's an almost 100% difference between the total Size of the Managed Heap and the total Commited size. (1737MB vs 946MB).
Is there a reliable way around this problem on .NET, i.e. can I coerce the runtime into allocating no more than what I actually need, or maybe much larger Managed Heaps that can be used for several contiguous buffers?
Internally the CLR allocates memory in segments. From your description it sounds like the 16 MB allocations are segments and your arrays are allocated within these. The remaining space is reserved and not really wasted under normal circumstances, as it will be used for other allocations. If you don't have any allocation that will fit within the remaining chunks these are essentially overhead.
As your arrays are allocated using contiguous memory you can only fit a single of those within a segment and hence the overhead in this case.
The default segment size is 16 MB, but if you allocation is larger than that the CLR will allocate segments that are larger. I'm not aware of the details, but e.g. if I allocate 20 MB Wmmap shows me 24 MB segments.
One way to reduce the overhead is to make allocations that fit with the segment sizes if possible. But keep in mind that these are implementation details and could change with any update of the CLR.
The CLR reserving a 16MB chunk in one go from the OS, but only actively occupying 9MB.
I believe you are expecting the 9MB and 9MB to be in one heap. The difficulty is that the variable is now split over 2 heaps.
Heap 1 = 9MB + 7MB
Heap 2 = 2MB
The problem we have now, is if the original 9MB is deleted, we now have 2 heaps we can't tidy up, as the contents are shared across heaps.
To improve performance, the approach is to put them in single heaps.
If you are worried about memory usage, don't be. Memory usage is not a bad thing with .NET, as it if no-one is using it, what's the problem? The GC will at some point kick in, and memory will be tidied up. GC will only kick in either
When the CLR deems it necessary
When the OS tells the CLR to give back memory
When forced to by the code
But memory usage, especially in this example shouldn't be a concern. The memory usage stops CPU cycles occurring. Otherwise if it tidied up memory constantly, your CPU would be high, and your process (and all others on your machine) would run much slower.
Age old symptom of the buddy system heap management algorithm, where powers of 2 are used to split each block recursively, in a binary tree, so for 9M the next size is 16M. If you dropped your array size down to 8mb, you will see your usage drop by half. Not a new problem, native programmers deal with it too.
The small object pool (less than 85,000 bytes) is managed differently, but at 9MB your arrays are in the large object pool. As of .NET 4.5, the large object heap doesn't participate in compaction, large objects are immediately promoted to generation 2.
You can't coerce the algorithm, but you can certainly coerce your user code by figuring out what sizes to use that will most efficiently fill the binary segments.
If you need to fill your process space with 9 MB arrays, either:
Figure out how to save 1MB to reduce the arrays to 8MB segments
Write or use a segmented array class that abstracts an array of 1 or 2MB array segments, using an indexer property. Same way you build an unlimited bitfield or a growable ArrayList. Actually, I thought one of the containers already did this.
Move to 64-bit
Reclaiming fragmented portion of a buddy system heap is an optimization with logarithmic returns (ie. you are approximately running out of memory anyway). At some point you'll have to move to 64-bit whether its convenient or not, unless your data size is fixed.
The question is regarding the allocation of arrays in .net. i have a sample program below in which maximum array i can get is to length. I increase length to +1 it gives outofMemory exception. But If I keep the length and remove the comments I am able to allocation 2 different big arrays. Both arrays are less the .net permissible object size of 2 gb and total memory also less than virtual Memory. Can someone put any thoughts?
class Program
{
static int length = 203423225;
static double[] d = new double[length];
//static int[] i = new int[15000000];
static void Main(string[] args)
{
Console.WriteLine((sizeof(double)*(double)length)/(1024*1024));
Console.WriteLine(d.Length);
//Console.WriteLine(i.Length);
Console.WriteLine(Process.GetCurrentProcess().VirtualMemorySize64.ToString());
}
}
A 32-bit process must allocate virtual memory for the array from the address space it has available. by default 2 gigabytes. Which contains a mix of both code and data. Allocations are made from the holes between the existing allocations.
Such allocations always fail not because there is no more virtual memory left, they fail because the available holes are not big enough. And you asking for a big hole, getting 1.6 jiggabytes is very rare and will only work on very simple programs that don't load any additional DLLs. A poorly based DLL is a good way to cut a large hole in two, drastically reducing the odds of such an allocation succeeding. The more typical works-on-the-first-try allocation is around 650 megabytes. The second allocation didn't fail because there was another hole available. The odds go down considerably after a program has been running for a while and the address space got fragmented. A 90 MB allocation can fail.
You can get insight in how the virtual memory address space is carved up for a program with the SysInternals' VMMap utility.
A simple workaround is to set the EXE project's Platform target setting to AnyCPU and run the program on a 64-bit operating system. It will have gobs of addressable virtual memory space available, you'll only be limited by the maximum allowed size of the paging file and the .NET 2 gigabyte object size limit. A limitation that's addressed in .NET 4.5 with the new <gcAllowVeryLargeObjects> config element. Even a 32-bit program can take advantage of the available 4 gigabyte 32-bit address space on a 64-bit operating system with the /LARGEADDRESSAWARE option of editbin.exe, you'll have to run it in a post-build event.
This will be because when allocating memory for arrays the memory must be contiguous (i.e. the array must be allocated as one large block of memory). Even if there is enough space in total to allocate the array, if the free address space is split up then the memory allocation will still fail unless the largest of those free spaces is large enough for the entire array.
Is it possible to put an artificial limit on the amount of memory a .NET process can use? I'm looking to simulate low-memory situations for testing purposes.
Right now we use a virtual machine for this type of thing. It works, but I'm curious to know if we can figure out a more convenient approach. One that could easily be automated would be ideal.
Edit: As Hans Passant points out, just limiting the amount of virtual memory available to the process won't replace the VM-based tests. The tests are for performance, so I would need to get swapping instead of OutOfMemoryException.
One that could easily be automated would be ideal.
You can use Windows Job Objects to manage this from code. Processes can be associated with a job, and JOBOBJECT_BASIC_LIMIT_INFORMATION allows you to restrict the working set size.
Alteratively, you can call SetProcessWorkingSetSize on the process directly, which will restrict the maximum allowable memory usage for that process.
The windows Application Verifier tool can simulate low resource levels, like low memory: http://msdn.microsoft.com/en-us/library/aa480483.aspx
A more programmatic solution using job objects (as #ReedCopsey also suggests) is here: http://www.codeproject.com/Articles/17831/How-to-Set-the-Maximum-Memory-a-Process-can-Use-at
Right now we use a virtual machine for this type of thing
Which probably means you limited the amount of RAM. That's not much of a test, a Windows process can never run out of RAM. You'll just slow down the process, the operating system will be swapping pages more frequently.
The real limitation is available virtual memory address space. Which is a fixed number for a 32-bit process, 2 gigabytes. Give or take a few non-standard options to increase that number. An OutOfMemoryException tells you that the process doesn't have a hole left in VM big enough to fit the allocation.
You can limit the amount of VM space by simply allocating a bunch of 50 megabyte byte[] arrays and storing them in a static variable of type List<byte[]> at the beginning of Main().
Monitoring my program's Virtual Bytes usage while it is running showed that by doing some kind operations, the virtual bytes usage goes up by about 1GB in about 5 minutes.
The program deals with tcp sockets and high data transfer throughput between them (~800Mbps).
Loading a dump file of the program in windbg showed that the reason for the very high and fast memory usage is about 1GB of "free" objects.
Indeed, when I call the garbage collector (gen 0, 1 & 2) from the console screen of the program (after getting it to this state) it frees up about 1GB of memory usage.
I'm trying to understand what exactly are these free objects and why aren't they garbage collected by the garbage collector automatically.
Edit: One suggestion was that I may be creating the objects in the Large Object Heap and it becomes fragmanted but this is not the case as I've seen that all of the "free" objects sits in Gen 2 Heap.
Other suggestion was that maybe Gen 2 Heap gets fragmented because of pinned objects but if this was the case, GC.Collect wouldn't fix the problem but it actually does so I believe this is not the case as well.
What I suspect from the discussion with Paul is that the memory does gets freed but is from some reason returned to the OS rarely or only when I manually call GC.Collect.
They are not free 'objects', they are free space. .NET does not release memory it has used back to the operating system immediately. Any free blocks can be used for subsequent object allocations, providing they fit inside the free block (otherwise the heap has to be extended by asking the operating system to allocate more memory).
The garbage collector makes efforts to combine free space into large usable blocks by compacting generation 2. This is not always possible: for example an application may pin objects which will potentially prevent the garbage collector from combining free space by moving the live objects to the front of the heap. If this happens a lot, the application's memory will be broken up into useless small blocks and this affect is known as 'heap fragmentation'.
In addition, there is a Large Object Heap (LOH) in which larger objects are allocated. The rationale is that there is a cost associated with heap compaction as data must be copied around and so the LOH is not compacted, avoiding these costs. However, the flipside is that the LOH can become easily fragmented, with small, less useful blocks of free memory interspersed between live objects.
I would suggest running a dumpheap -stat. This command will report at the end of the list any fragmented blocks. You can then dump those blocks to get an idea of what is happening.
By the way, it looks like you have a well-known problem (at least among socket gurus) that most socket servers get in .Net. Paul has already touched on what it means. To elaborate more, what is going wrong is that during a Read/Write on the socket the buffer gets pinned - which means the GC isn't allowed to move it around (as such your heap fragments). Coversant (who pioneered the solution) were seeing an OutOfMemoryException when their actual memory usage was only about 500MB (due to such heavy fragmentation). Fixing it is another story entirely.
What you want to do is at application start up allocate a very large amount of buffers (I am currently doing 50MB). You will find the new ArraySegment<T> (v2.0) and ConcurrentQueue<T> (v4.0) classes particularly useful when writing this. There are a few lock-free queues floating around on the tubes if you are not using v4.0 yet.
// Pseudo-code.
ArraySegment<byte> CheckOut()
{
ArraySegment<byte> result;
while(!_queue.TryDequeue(out result))
GrowBufferQueue(); //Enqueue a bunch more buffers.
return result;
}
void CheckOut(ArraySegment<byte> buffer)
{
_queue.Enqueue(buffer);
}
void GrowBufferQueue()
{
// Verify this, I did throw it together in 30s.
// Allocates nearly 2MB. You might want to tweak that.
for(var j = 0; j < 5; j++)
{
var buffer = new byte[409600]; // 4096 = page size on Windows.
for(var i = 0; i < 409600; i += 4096)
_queue.Enqueue(new ArraySegment<byte>(buffer, i, 4096));
}
}
Following this you will need to subclass NetworkStream and swap out the incoming buffer with one from your buffer pool. Buffer::BlockCopy will help performance (don't use Array::Copy). This is complicated and hairy; especially if you make it async-capable.
If you are not layering streams (e.g. SSLStream <--> DeflateStream <--> XmlWriter etc.) you should use the new socket async pattern in .Net 4.0; which has more efficiency around the IAsyncResults that get passed around. Because you are not layering streams you have full control over the buffers that get used - so you don't need to go the NetworkStream subclass route.