This question already has answers here:
change array size
(15 answers)
Closed 4 months ago.
This post was edited and submitted for review 4 months ago and failed to reopen the post:
Original close reason(s) were not resolved
If I create and array without initialization the program uses only a few MB of memory to save the reference. For example like this:
int[] arr = new int[1000000000];
Later if I initialize the elements of array the memory usage goes up by the full array amount.
The memory usage is lower if I initialize only a part of array.
for (int i = 0; i < arr.Length; i++) // Full memory usage, length * sizeof(int)
{
arr[i] = i;
}
----------------------------------------------------------
for (int i = 0; i < arr.Length/2; i++) // Only ~half memory usage, (length / 2) * sizeof(int)
{
arr[i] = i;
}
Now I need to use my full initialized array to create an data structure and then reduce its size by keeping every N element. Because the array is very big reducing its size can save several GB. The missing elements can be calculated from the remaining ones and created data structure. Its a trading calculation time for memory usage.
Now to the question:
Is it possible in C# to release some memory from array after the array is fully initialized?
I have tried to set every Nth element to zero but the memory usage is still the same.
Although Array.Resize() sounds like an option, in reality wouldn't satisfy your question. Your question is: How can I release RAM? Am I correct? If so, using Array.Resize() will only make it worse, because in reality it creates a new array, leaving the source untouched, so in reality this consumes even more RAM.
In the safe space of .Net, I don't think you can immediately drop the RAM. I think you will just have to create a final list of numbers and drop all references to the original array and wait for the garbage collector to do its job, or see if you can collect faster by calling GC.Collect().
In the unsafe space, however, you can control your RAM usage.
I will stop here for now because I just read an incoming comment from you stating that you need to delete the space but keep the size. You are using an array of a primitive data type. This array, once initialized, cannot be deleted, regardless of being in a safe or unsafe context.
In your case, I would then use a dictionary to save the remaining values and drop the entire original array. Line in your comment example, where you have 8 elements and you are conserving every 3rd element. Your dictionary will have 3 elements, then you drop the entire original array. The dictionary's key will tell you the array's original index, while its value will give you, well, the value contained in that position.
What if you initialize a second array
int[] arrCopy = new int[1000000000];
Copy the first array into the second
Array.Copy(arr, arrCopy, 1000000000);
Re-initialize the first array
arr = new int[1000000000];
Iterate over the second array and store every Nth value into the first
for (int i = 0 ; i < 1000000000; i += N){
arr[i] = i;
}
Re-initialize the second array
arrCopy = new int[1000000000];
After doing some profiling, we've discovered that the current way in which our app concatenates strings causes an enormous amount of memory churn and CPU time.
We're building a List<string> of strings to concatenate that is on the order of 500 thousand elements long, referencing several hundred megabytes worth of strings. We're trying to optimize this one small part of our app since it seems to account for a disproportionate amount of CPU and memory usage.
We do a lot of text processing :)
Theoretically, we should be able to perform the concatenation in a single allocation and N copies - we can know how many total characters are available in our string, so it should just be as simple as summing up the lengths of the component strings and allocating enough underlying memory to hold the result.
Assuming we're starting with a pre-filled List<string>, is it possible to concatenate all strings in that list using a single allocation?
Currently, we're using the StringBuilder class, but this stores its own intermediate buffer of all of the characters - so we have an ever growing chunk array, with each chunk storing a copy of the characters we're giving it. Far from ideal. The allocations for the array of chunks aren't horrible, but the worst part is that it allocates intermediate character arrays, which means N allocations and copies.
The best we can do right now is to call List<string>.ToArray() - which performs one copy of a 500k element array - and pass the resulting string[] to string.Concat(params string[]). string.Concat() then performs two allocations, one to copy the input array into an internal array, and the one to allocate the destination string's memory.
From referencesource.microsoft.com:
public static String Concat(params String[] values) {
if (values == null)
throw new ArgumentNullException("values");
Contract.Ensures(Contract.Result<String>() != null);
// Spec#: Consider a postcondition saying the length of this string == the sum of each string in array
Contract.EndContractBlock();
int totalLength=0;
// -----------> Allocation #1 <---------
String[] internalValues = new String[values.Length];
for (int i=0; i<values.Length; i++) {
string value = values[i];
internalValues[i] = ((value==null)?(String.Empty):(value));
totalLength += internalValues[i].Length;
// check for overflow
if (totalLength < 0) {
throw new OutOfMemoryException();
}
}
return ConcatArray(internalValues, totalLength);
}
private static String ConcatArray(String[] values, int totalLength) {
// -----------------> Allocation #2 <---------------------
String result = FastAllocateString(totalLength);
int currPos=0;
for (int i=0; i<values.Length; i++) {
Contract.Assert((currPos <= totalLength - values[i].Length),
"[String.ConcatArray](currPos <= totalLength - values[i].Length)");
FillStringChecked(result, currPos, values[i]);
currPos+=values[i].Length;
}
return result;
}
Thus, in the best case, we have three allocations, two for arrays referencing the component strings, and one for the destination concatenated string.
Can we improve on this? Is it possible to concatenate a List<string> using a single allocation and a single loop of character copies?
Edit 1
I'd like to summarize the various approaches discussed so far, and why they are still sub-optimal. I'd also like to set the parameters of the situation in concrete a little more, since I've received a lot of questions that try to side step the central question.
...
First, the structure of the code that I am working within. There are three layers:
Layer one is a set of methods that produce my content. These methods return small-ish string objects, which I will call my 'component' strings'. These string objects will eventually be concatenated into a single string. I do not have the ability to modify these methods; I have to face the reality that they return string objects and move forward.
Layer two is my code that calls these content producers and assembles the output, and is the subject of this question. I must call the content producer methods, collect the strings they return, and eventually concatenate the returned strings into a single string (reality is a little more complex; the returned strings are partitioned depending on how they're routed for output, and so I have several sets of large collections of strings).
Layer three is a set of methods that accept a single large string for further processing. Changing the interface of that code is beyond my control.
Talking about some numbers: a typical batch run will collect ~500000 strings from the content producers, representing about 200-500 MB of memory. I need the most efficient way to concatenate these 500k strings into a single string.
...
Now I'd like to examine the approaches discussed so far. For the sake of numbers, assume we're running 64-bit, assume that we are collecting 500000 string objects, and assume that the aggregate size of the string objects totals 200 megabytes worth of character data. Also, assume that the original string object's memory is not counted toward any approach's total in the below analysis. I make this assumption because it is necessarily common to any and all approaches, because it is an assumption that we cannot change the interface of the content producers - they return 500k relatively small fully formed strings objects that I must then accept and somehow concatenate. As stated above, I cannot change this interface.
Approach #1
Content producers ----> StringBuilder ----> string
Conceptually, this would be invoking the content producers, and directly writing the strings they return to a StringBuilder, and then later calling StringBuilder.ToString() to obtain the concatenated string.
By analyzing StringBuilder's implementation, we can see that the cost of this boils down to 400 MB of allocations and copies:
During the stage where we collect the output from the content producers, we're writing 200 MB of data to the StringBuilder. We would be performing one 200 MB allocation to pre-allocate the StringBuilder, and then 200 MB worth of copies as we copy and discard the strings returned from the content producers
After we've collected all output from the content producers and have a fully formed StringBuilder, we then need to call StringBuilder.ToString(). This performs exactly one allocation (string.FastAllocateString()), and then copies the string data from its internal buffers to the string object's internal memory.
Total cost: approximately 400 MB of allocations and copies
Approach #2
Content producers ---> pre-allocated char[] ---> string
This strategy is fairly simple. Assuming we know roughly how much character data we're going to be collecting from the producers, we can pre-allocate a char[] that is 200 MB large. Then, as we call the content producers, we copy the strings they return into our char[]. This accounts for 200 MB of allocations and copies. The final step to turn this into a string object is to pass it to the new string(char[]) constructor. However, since strings are immutable and arrays are not, the constructor will make a copy of that entire array, causing it to allocate and copy another 200 MB of character data.
Total cost: approximately 400 MB of allocations and copies
Approach #3:
Content producers ---> List<string> ----> string[] ----> string.Concat(string[])
Pre-allocate a List<string> to be about 500k elements - approximately 4 MB of allocations for List's underlying array (500k * 8 bytes per pointer == 4 MB of memory).
Call all of the content producers to collect their strings. Approximately 4 MB of copies, as we copy the pointer to the returned string into List's underlying array.
Call List<string>.ToArray() to obtain a string[]. Approximately 4 MB of allocations and copies (again, we're really just copying pointers).
Call string.Concat(string[]):
Concat will make a copy of the array provided to it before it does any real work. Approximately 4 MB of allocations and copies, again.
Concat will then allocate a single 'destination' string object using the internal string.FastAllocateString() special method. Approximately 200 MB of allocations.
Concat will then copy strings from its internal copy of the provided array directly into the destination. Approximately 200 MB of copies.
Total cost: approximately 212 MB of allocations and copies
None of these approaches are ideal, however approach #3 is very close. We're assuming that the absolute minimum of memory that needs to be allocated and copied is 200 MB (for the destination string), and here we get pretty close - 212 MB.
If there were a string.Concat overload that 1) Accepted an IList<string> and 2) did not make a copy of that IList before using it, then the problem would be solved. No such method is provided by .Net, hence the subject of this question.
Edit 2
Progress on a solution.
I've done some testing with some hacked IL, and found that directly invoking string.FastAllocateString(n) (which is not usually invokable...) is about as fast as invoking new string('\0', n), and both seem to allocate exactly as much memory as is expected.
From there, it seems its possible to acquire a pointer to the freshly allocated string using the unsafe and fixed statements.
And so, a rough solution begins to appear:
private static string Concat( List<string> list )
{
int concatLength = 0;
for( int i = 0; i < list.Count; i++ )
{
concatLength += list[i].Length;
}
string newString = new string( '\0', concatLength );
unsafe
{
fixed( char* ptr = newString )
{
...
}
}
return newString;
}
The next biggest hurdle is implementing or finding an efficient block copy method, ala Buffer.BlockCopy, except one that will accept char* types.
If you can determine the length of the concatenation before trying to perform the operation, a char array can beat string builder in some use cases. Manipulating the characters within the array prevents the multiple allocations.
See: http://blogs.msdn.com/b/cisg/archive/2008/09/09/performance-analysis-reveals-char-array-is-better-than-stringbuilder.aspx
UPDATE
Please check out this internal implementation of the String.Join from .NET - it uses unsafe code with pointers to avoid multiple allocations. Unless I'm missing something, it would seem you can re-write this using your List to accomplish what you want:
[System.Security.SecuritySafeCritical] // auto-generated
public unsafe static String Join(String separator, String[] value, int startIndex, int count) {
//Range check the array
if (value == null)
throw new ArgumentNullException("value");
if (startIndex < 0)
throw new ArgumentOutOfRangeException("startIndex", Environment.GetResourceString("ArgumentOutOfRange_StartIndex"));
if (count < 0)
throw new ArgumentOutOfRangeException("count", Environment.GetResourceString("ArgumentOutOfRange_NegativeCount"));
if (startIndex > value.Length - count)
throw new ArgumentOutOfRangeException("startIndex", Environment.GetResourceString("ArgumentOutOfRange_IndexCountBuffer"));
Contract.EndContractBlock();
//Treat null as empty string.
if (separator == null) {
separator = String.Empty;
}
//If count is 0, that skews a whole bunch of the calculations below, so just special case that.
if (count == 0) {
return String.Empty;
}
int jointLength = 0;
//Figure out the total length of the strings in value
int endIndex = startIndex + count - 1;
for (int stringToJoinIndex = startIndex; stringToJoinIndex <= endIndex; stringToJoinIndex++) {
if (value[stringToJoinIndex] != null) {
jointLength += value[stringToJoinIndex].Length;
}
}
//Add enough room for the separator.
jointLength += (count - 1) * separator.Length;
// Note that we may not catch all overflows with this check (since we could have wrapped around the 4gb range any number of times
// and landed back in the positive range.) The input array might be modifed from other threads,
// so we have to do an overflow check before each append below anyway. Those overflows will get caught down there.
if ((jointLength < 0) || ((jointLength + 1) < 0) ) {
throw new OutOfMemoryException();
}
//If this is an empty string, just return.
if (jointLength == 0) {
return String.Empty;
}
string jointString = FastAllocateString( jointLength );
fixed (char * pointerToJointString = &jointString.m_firstChar) {
UnSafeCharBuffer charBuffer = new UnSafeCharBuffer( pointerToJointString, jointLength);
// Append the first string first and then append each following string prefixed by the separator.
charBuffer.AppendString( value[startIndex] );
for (int stringToJoinIndex = startIndex + 1; stringToJoinIndex <= endIndex; stringToJoinIndex++) {
charBuffer.AppendString( separator );
charBuffer.AppendString( value[stringToJoinIndex] );
}
Contract.Assert(*(pointerToJointString + charBuffer.Length) == '\0', "String must be null-terminated!");
}
return jointString;
}
Source: http://www.dotnetframework.org/default.aspx/4#0/4#0/DEVDIV_TFS/Dev10/Releases/RTMRel/ndp/clr/src/BCL/System/String#cs/1305376/String#cs
UPDATE 2
Good point on the fast allocate. According to an old SO post, you can wrap FastAllocate using reflection (assuming of course you'd cache the fastAllocate method reference so you just called Invoke each time. Perhaps the tradeoff of the call is better than what you're doing now.
var fastAllocate = typeof (string).GetMethods(BindingFlags.NonPublic | BindingFlags.Static)
.First(x => x.Name == "FastAllocateString");
var newString = (string)fastAllocate.Invoke(null, new object[] {20});
Console.WriteLine(newString.Length); // 20
Perhaps another approach is to use unsafe code to copy your allocation into a char* array, then pass this to the string constructor. The string constructor with char* is an extern passed to the underlying C++ implementation. I haven't found a reliable source for that code to confirm, but perhaps this can be faster for you. The non-prod ready code (no checks for potential overflow, add fixed to lock strings from garbage collection, etc) would start with:
public unsafe string MyConcat(List<string> values)
{
int index = 0;
int totalLength = values.Sum(m => m.Length);
char* concat = stackalloc char[totalLength + 1]; // Add additional char for null term
foreach (var value in values)
{
foreach (var c in value)
{
concat[index] = c;
index++;
}
}
concat[index] = '\0';
return new string(concat);
}
Now I'm all out of ideas for this :) Perhaps somebody can figure out a method here with marshalling to avoid unsafe code. Since introducing unsafe code requires adding the unsafe flag to compilation, consider adding this piece as a separate dll to minimize your app's security risk if you go down that route.
Unless the average length of the strings is very small, the most efficient approach, given a List<String>, will be to use ToArray() to copy it to a new String[], and pass that to a concatenation or joining method. Doing that may cause a wasted allocation for an array of references if the concatenation or joining method wants to make a copy of its array before it starts, but that would only allocate one reference per string, there will only be one allocation to hold character data, and it will be correctly sized to hold the entire string.
If you're building the data structure yourself, you might gain a little bit of efficiency by initializing a String[] to the estimated required size, populating it yourself, and expanding it as needed. That would save one allocation of a String[] worth of data.
Another approach would be to allocate a String[8192][] and then allocate a String[8192] for each array of strings as you go along. Once you're all done, you'll know exactly what size String[] you need to pass to the Concat method so you can create an array of that exact size. This approach would require a greater quantity of allocations, but only the final String[] and the String itself would need to go on the Large Object Heap.
It's a shame the constraints you're putting on yourself. It's very blockily structured, and it's hard to get any flow going. For example, if you didn't expect a IList but only expected IEnumerable you might be able to make it easier for the producer of your content. Not only that, you could make your processing benefit from being able to consume the strings only as you need them - and only as they're produced.
This gets you on down the road to some nice asynchrony.
One the other end, they're making you send to whole thing at once. That's tough.
But having said that, and since you're going to run it over and over, etc... I'm wondering if you couldn't create your string buffer or byte buffer or StringBuilder or whatever - and reuse it between executions - allocate the max monster (or progressively bump-reallocate it as needed) one time - and don't let the gc have it. The string constructor will copy it over and over again - but that's a single allocation per cycle. If you're running this so much you're making the machine hot, then it might be worth the hit. I've made precisely that tradeoff in the near past (but I didn't have 5gb to choke on). It felt dirty at first - but ooohh - the throughput spoke loudly!
Also, it may be possible, that while your native API expects a string, but you can lie to it - let it think you're giving it a string. You can very probably pass the buffer with a null char at the end - or with the length - depending on the API's particulars. I think one or two commenters spoke to this. In such a case, you may probably need your buffer pinned for the duration of the calls to the native consumer of your big ol' string.
If this is the case, you're down to a one-time allocation of a buffer, repeated copies into it, and that's it. It could go way under your proposed best case.
I have implemented a method to concatenate a List into a single string that performs exactly one allocation.
The following code compiles under .Net 4.6 - Block.MemoryCopy wasn't added to .Net until 4.6.
The "unsafe" implementation:
public static unsafe class FastConcat
{
public static string Concat( IList<string> list )
{
string destinationString;
int destLengthChars = 0;
for( int i = 0; i < list.Count; i++ )
{
destLengthChars += list[i].Length;
}
destinationString = new string( '\0', destLengthChars );
unsafe
{
fixed( char* origDestPtr = destinationString )
{
char* destPtr = origDestPtr; // a pointer we can modify.
string source;
for( int i = 0; i < list.Count; i++ )
{
source = list[i];
fixed( char* sourcePtr = source )
{
Buffer.MemoryCopy(
sourcePtr,
destPtr,
long.MaxValue,
source.Length * sizeof( char )
);
}
destPtr += source.Length;
}
}
}
return destinationString;
}
}
The competing implementation is the following "safe" implementation:
public static string Concat( IList<string> list )
{
return string.Concat( list.ToArray() )
}
Memory consumption
The "unsafe" implementation performs exactly one allocation and zero temporary allocations. The List<string> is directly concatenated into a single, freshly allocated string object.
The "safe" implementation requires two copies of the list - one, when I call ToArray() to pass it to string.Concat, and another when string.Concat performs its own internal copy of the array.
When concatenating a 500k element list, the "safe" string.Concat method allocates exactly 8 MB of extra memory in a 64-bit process, which I've confirmed by running the test driver in a memory monitor. This is what we would expect with the array copies performed by the safe implementation.
CPU performance
For small worksets, the unsafe implementation seems to win by about 25%.
The test driver was tested by compiling for 64-bit, installing the program into the native image cache via NGEN, and running from outside the debugger on an unloaded workstation.
From my test driver with a small workset (500k strings each 2-10 chars long):
Unsafe Time: 17.266 ms
Unsafe Time: 18.419 ms
Unsafe Time: 16.876 ms
Safe Time: 21.265 ms
Safe Time: 21.890 ms
Safe Time: 24.492 ms
Unsafe average: 17.520 ms. Safe average: 22.549 ms. Safe takes about 25% longer than unsafe. This is likely due to the extra work the safe implementation has to do, allocating temporary arrays.
...
From my test driver with a large workset (500k strings, each 500-800 chars long):
Unsafe Time: 498.122 ms
Unsafe Time: 513.725 ms
Unsafe Time: 515.016 ms
Safe Time: 487.456 ms
Safe Time: 499.508 ms
Safe Time: 512.390 ms
As you can see, the performance difference with large strings is roughly zero, likely because the time is dominated by the raw copy.
Conclusion
If you don't care about the array copies, the safe implementation is dead simple to implement, and is roughly as fast as the unsafe implementation. If you want to be absolutely perfect with memory usage, use the unsafe implementation.
I've attached the code I used for the test harness:
class PerfTestHarness
{
private List<string> corpus;
public PerfTestHarness( List<string> corpus )
{
this.corpus = corpus;
// Warm up the JIT
// Note that `result` is discarded. We reference it via 'result[0]' as an
// unused paramater to my prints to be absolutely sure it doesn't get
// optimized out. Cheap hack, but it works.
string result;
result = FastConcat.Concat( this.corpus );
Console.WriteLine( "Fast warmup done", result[0] );
result = string.Concat( this.corpus.ToArray() );
Console.WriteLine( "Safe warmup done", result[0] );
GC.Collect();
GC.WaitForPendingFinalizers();
}
public void PerfTestSafe()
{
Stopwatch watch = new Stopwatch();
string result;
GC.Collect();
GC.WaitForPendingFinalizers();
watch.Start();
result = string.Concat( this.corpus.ToArray() );
watch.Stop();
Console.WriteLine( "Safe Time: {0:0.000} ms", watch.Elapsed.TotalMilliseconds, result[0] );
Console.WriteLine( "Memory usage: {0:0.000} MB", Environment.WorkingSet / 1000000.0 );
Console.WriteLine();
}
public void PerfTestUnsafe()
{
Stopwatch watch = new Stopwatch();
string result;
GC.Collect();
GC.WaitForPendingFinalizers();
watch.Start();
result = FastConcat.Concat( this.corpus );
watch.Stop();
Console.WriteLine( "Unsafe Time: {0:0.000} ms", watch.Elapsed.TotalMilliseconds, result[0] );
Console.WriteLine( "Memory usage: {0:0.000} MB", Environment.WorkingSet / 1000000.0 );
Console.WriteLine();
}
}
StringBuilder was designed to concatenate strings efficiently. It has no other purpose.Use the constructor which sets the initial capacity:
int totalLength = CalcTotalLength();
// sufficient capacity
StringBuilder sb = new StringBuilder(totalLength);
But then you say that even StringBuilder allocates intermediate memory, and you want to do better...
These are unusual requirements, so you need to write a function which suits your situation (creating a char[] of appropriate size, then filling it in). I'm sure you are more than capable.
The first two of my answers have now been already incorporated in the question. Here is my highly situation dependent, but useful -
Third Answer
If in all these MBs of string you are getting a lot of strings that are same, then a smarter way would be use two dictionaries, one would be Dictionary<int, int> to store position and "Id" of the string at that position while another would be a Dictionary<int, int> to store the "Id" and the index of actual string in the original string[].
Coincidentally for me, what I am trying to do is already implemented in C#. Goes kinda like this...
If indeed there are a lot of same strings, is it a rare case where String Interning is useful? You are guaranteed to save considerable amount of your 200 MB target if a lot of matching strings are coming from the content producers.
What is String.Intern?
When you use strings in C#, the CLR does something clever called
string interning. It's a way of storing one copy of any string. If you
end up having a hundred—or, worse, a million—strings with the same
value, it's a waste to take up all of that memory storing the same
string over and over again. String interning is a way around that.
The CLR maintains a table called the intern pool that contains a
single, unique reference to every literal string that's either
declared or created programmatically while your program's running. And
the .NET Framework gives you two useful methods for interacting with
the intern pool: String.Intern() and String.IsInterned().
The way String.Intern() works is pretty straightforward. You pass it a
single string as an argument. If that string is already in the intern
pool, it returns a reference to that string. If it's not already in
the intern pool, it adds it and returns the same reference you passed
into it.
The way to use String Interning is explained in the link. For the sake of completeness of this answer I can add the code here but only if you feel that these solutions are useful.
I have huge transient arrays created rapidly. Some are kept, some are GC-d. This defragments the heap and the app consumes approx. 2.5x more memory than it would truly need resulting OutOfMemoryException.
As a solution, I would prefer to have one gigantic array (PointF[]) and do the allocation and management of segments by myself. But I wonder how I could make two (or more) arrays share the same memory space.
PointF[] giganticList = new PointF[100];
PointF[] segment = ???;
// I want the segment length to be 20 and starting e.g at position 50
// within the gigantic list
I am thinking of a trick like the winner answer of this SO question.
Would that be possible? The problem is that the length and the number of the segment arrays are known only in runtime.
Assuming you are sure that your OutOfMemoryException could be avoided, and your approach of having it all in memory isn't the actual issue (the GC is pretty good at stopping this happening if memory is available) ...
Here is your first problem. I'm not sure the CLR supports any single object larger than 2 GB.
Crucial Edit - gcAllowVeryLargeObjects changes this on 64-bit systems - try this before rolling your own solution.
Secondly you are talking about "some are kept some are GC'd". i.e. you want to be able to reallocate elements of your array once you are done with a "child array".
Thirdly, I'm assuming that the PointF[] giganticList = new PointF[100]; in your question is meant to be more like PointF[] giganticList = new PointF[1000000];?
Also consider using MemoryFailPoint as this allows you to "demand" memory and check for exceptions instead of crashing with OutOfMemoryException.
EDIT Perhaps most importantly you are now entering a land of trade-offs. If you do this you could start losing the advantages of things like the jitter optimising for loops by doing array bound checks at the start of the loop (for (int i= 0; i < myArray.Length; i++)
gets optimised, int length = 5; for (int i= 0; i < length; i++) doesn't). If you have high computation resource code, then this could hurt you. You are also going to have to work far harder to process different child arrays in parallel with each other as well. Creating copies of the child arrays, or sections of them, or even items inside them, is still going to allocate more memory which will be GC'd.
This is possible by wrapping the array, and tracking which sections are used for which child arrays. You are essentially talking about allocating a huge chunk of memory, and then reusing parts of it without putting the onus on the GC. You can take advantage of ArraySegment<T>, but that comes with its own potential issues like exposing the original array to all callers.
This is not going to be simple, but it is possible. Likely as not each time you remove a child array you will want to defragment your master array by shifting other child arrays to close the gaps (or do that when you have run out of contiguous segments).
A simple example would look something like the (untested, don't blame me if your computer leaves home and your cat blows up) pseudocode below. There are two other approaches, I mention those at the end.
public class ArrayCollection {
List<int> startIndexes = new List<int>();
List<int> lengths = new List<int>();
const int 1beeellion = 100;
PointF[] giganticList = new PointF[1beeellion];
public ArraySegment<PointF> this[int childIndex] {
get {
// Care with this method, ArraySegment exposes the original array, which callers could then
// do bad things to
return new ArraySegment<String>(giganticList, startIndexes[childIndex], length[childIndex]);
}}
// returns the index of the child array
public int AddChild(int length) {
// TODO: needs to take account of lists with no entries yet
int startIndex = startIndexes.Last() + lengths.Last();
// TODO: check that startIndex + length is not more than giganticIndex
// If it is then
// find the smallest unused block which is larger than the length requested
// or defrag our unused array sections
// otherwise throw out of memory
startIndexes.Add(startIndex); // will need inserts for defrag operations
lengths.Add(length); // will need inserts for defrag operations
return startIndexes.Count - 1; // inserts will need to return inserted index
}
public ArraySegment<PointF> GetChildAsSegment(int childIndex) {
// Care with this method, ArraySegment exposes the original array, which callers could then
// do bad things to
return new ArraySegment<String>(giganticList, startIndexes[childIndex], length[childIndex]);
}
public void SetChildValue(int childIndex, int elementIndex, PointF value) {
// TODO: needs to take account of lists with no entries yet, or invalid childIndex
// TODO: check and PREVENT buffer overflow (see warning) here and in other methods
// e.g.
if (elementIndex >= lengths[childIndex]) throw new YouAreAnEvilCallerException();
int falseZeroIndex = startIndexes[childIndex];
giganticList[falseZeroIndex + elementIndex];
}
public PointF GetChildValue(int childIndex, int elementIndex) {
// TODO: needs to take account of lists with no entries yet, bad child index, element index
int falseZeroIndex = startIndexes[childIndex];
return giganticList[falseZeroIndex + elementIndex];
}
public void RemoveChildArray(int childIndex) {
startIndexes.RemoveAt(childIndex);
lengths.RemoveAt(childIndex);
// TODO: possibly record the unused segment in another pair of start, length lists
// to allow for defraging in AddChildArray
}
}
Warning The above code effectively introduces buffer overflow vulnerabilities if, for instance, you don't check the requested childIndex against length for the child array in methods like SetChildValue. You must understand this and prevent it before trying to do this in production, especially if combining these approaches with use of unsafe.
Now, this could be extended to return psuedo index public PointF this[int index] methods for child arrays, enumerators for the child arrays etc., but as I say, this is getting complex and you need to decide if it really will solve your problem. Most of your time will be spent on the reuse (first) defrag (second) expand (third) throw OutOfMemory (last) logic.
This approach also has the advantage that you could allocate many 2GB subarrays and use them as a single array, if my comment about the 2GB object limit is correct.
This assumes you don't want to go down the unsafe route and use pointers, but the effect is the same, you would just create a wrapper class to manage child arrays in a fixed block of memory.
Another approach is to use the hashset/dictionary approach. Allocate your entire (massive 2GB array) and break it into chunks (say 100 array elements). A child array will then have multiple chunks allocated to it, and some wasted space in its final chunk. This will have the impact of some wasted space overall (depending on your average "child length vs. chunk length" predictions), but the advantage that you could increase and decrease the size of child arrays, and remove and insert child arrays with less impact on your fragmentation.
Noteworthy References:
Large arrays in 64 bit .NET 4: gcAllowVeryLargeObjects
MemoryFailPoint - allows you to "demand" memory and check for exceptions instead of crashing with OutOfMemoryException after the fact
Large Arrays, and LOH Fragmentation. What is the accepted convention?
3GB process limit on 32 bit, see: 3_GB_barrier, Server Fault /3GB considerations and AWE/PAE
buffer overflow vulnerability, and why you can get this in C#
Other examples of accessing arrays as a different kind of array or structure. The implementations of these might help you develop your own solution
BitArray Class
BitVector32 Structure
NGenerics - clever uses of array like concepts in some of the members of this library, particularly the general structures such as ObjectMatrix and Bag
C# array of objects, very large, looking for a better way
Array optimisation
Eric Gu - Efficiency of iteration over arrays? - note the age of this, but the approach of how to look for JIT optimisation is still relevant even with .NET 4.0 (see Array Bounds Check Elimination in the CLR? for example)
Dave Detlefs - Array Bounds Check Elimination in the CLR
Warning - pdf: Implicit Array Bounds Checking on 64-bit Architectures
LinkedList - allows you to reference multiple disparate array buckets in a sequence (tieing together the chunks in the chunked bucket approach)
Parallel arrays and use of unsafe
Parallel Matrix Multiplication With the Task Parallel Library (TPL), particularly UnsafeSingle - a square or jagged array represented by a single array is the same class of problem you are trying to solve.
buffer overflow vulnerability, and why you can get this in C# (yes I have mentioned this three times now, it is important)
Your best bet here is probably to use multiple ArraySegment<PointF> all on the same PointF[] instance, but at different offsets, and have your calling code take note of the relative .Offset and .Count. Note that you would have to write your own code to allocate the next block, and look for gaps, etc - essentially your own mini-allocator.
You can't treat the segments just as a PointF[] directly.
So:
PointF[] giganticList = new PointF[100];
// I want the segment length to be 20 and starting e.g at position 50
// within the gigantic list
var segment = new ArraySegment<PointF>(giganticList, 50, 20);
As a side note: another approach might be to use a pointer to the data - either from an unmanaged allocation, or from a managed array that has been pinned (note: you should try to avoid pinning), but : while a PointF* can convey its own offset information, it cannot convey length - so you'd need to always pass both a PointF* and Length. By the time you've done that, you might as well have just used ArraySegment<T>, which has the side-benefit of not needing any unsafe code. Of course, depending on the scenario, treating the huge array as unmanaged memory may (in some scenarios) still be tempting.
It's easier to write
intArray1.CopyTo( intArray2, 0 )
than the for-loop equivalent, but System.Array does not provide any generic Copy/CopyTo methods.
Is it better to write the for-loop? Or is using Copy/CopyTo compiled or JIT'd efficiently enough?
Array.Copy/CopyTo will perform faster than a manual loop in most cases as it can do direct memory copying.
If you don't have huge arrays or speed is not an issue, use whatever would look best in your code where you need to copy the items.
If you are copying an array of primitive types as your sample would imply, you can us the memory copy technique yourself using the Buffer classes BlockCopy method.
int[] CopyArray(int[] A, int index)
{
const int INT_SIZE = 4;
int length = A.Length - index;
int[] B = new int[A.Length - index];
Buffer.BlockCopy(A, index * INT_SIZE, B,
0 * INT_SIZE, length * INT_SIZE);
return B;
}
This method is the most efficient manner in which to copy an array of primitives. (It only works with primitives)
I say if you know that you want to copy the entirety of the first array to the second array without changing the values or doing any specific processing on the copy, then use Array.CopyTo.
There are some limitations to this. The array must only have a single dimension as I remember it. Also if the arrays are quite large you might have some speed related issues with the copyto, but I would imagine that would only come into play with very large arrays. So, I would try it and test it, but your mileage may vary.
Is there an efficient way to take a subset of a C# array and pass it to another peice of code (without modifying the original array)? I use CUDA.net which has a function which copies an array to the GPU. I would like to e.g. pass the function a 10th of the array and thus copy each 10th of the array to the GPU seperately (for pipelining purposes).
Copying the array in this way should be as efficient as copying it in one go. It can be done with unsafe code and just referencing the proper memory location but other than that I'm not sure. The CopyTo function copies the entire array to another array so this does not appear useful.
Okay, I'd misunderstood the question before.
What you want is System.Buffer.BlockCopy or System.Array.Copy.
The LINQ ways will be hideously inefficient. If you're able to reuse the buffer you're copying into, that will also help the efficiency, avoiding creating a new array each time - just copy over the top. Unless you can divide your "big" array up equally though, you'll need a new one for the last case.
I'm not sure how efficient this is but...
int[] myInts = new int[100];
//Code to populate original arrray
for (int i = 0; i < myInts.Length; i += 10)
{
int[] newarray = myInts.Skip(i).Take(10).ToArray();
//Do stuff with new array
}
You could try Marshal.Copy if you need to go from an array of bytes to an unmanaged pointer. That avoids creating unsafe code yourself.
Edit: This would clearly only work if you reimplement their API. Sorry - misunderstood. You want an efficient subarray method.
It strikes me that what you really want is an api in the original class of the form
void CopyToGpu(byte[] source, int start, int length);
You could use extension methods and yield return:
public static IEnumerable Part<T>(this T[] array, int startIndex, int endIndex )
{
for ( var currentIndex = startIndex; currentIndex < endIndex; ++currentIndex )
yield return array[currentIndex];
}