Develop Reference

Is it possible to reduce array memory usage after Initialization but keeping its original size? [duplicate]

This question already has answers here:
change array size
(15 answers)
Closed 4 months ago.
This post was edited and submitted for review 4 months ago and failed to reopen the post:
Original close reason(s) were not resolved
If I create and array without initialization the program uses only a few MB of memory to save the reference. For example like this:
int[] arr = new int[1000000000];
Later if I initialize the elements of array the memory usage goes up by the full array amount.
The memory usage is lower if I initialize only a part of array.
for (int i = 0; i < arr.Length; i++) // Full memory usage, length * sizeof(int)
{
arr[i] = i;
}
----------------------------------------------------------
for (int i = 0; i < arr.Length/2; i++) // Only ~half memory usage, (length / 2) * sizeof(int)
{
arr[i] = i;
}
Now I need to use my full initialized array to create an data structure and then reduce its size by keeping every N element. Because the array is very big reducing its size can save several GB. The missing elements can be calculated from the remaining ones and created data structure. Its a trading calculation time for memory usage.
Now to the question:
Is it possible in C# to release some memory from array after the array is fully initialized?
I have tried to set every Nth element to zero but the memory usage is still the same.

Although Array.Resize() sounds like an option, in reality wouldn't satisfy your question. Your question is: How can I release RAM? Am I correct? If so, using Array.Resize() will only make it worse, because in reality it creates a new array, leaving the source untouched, so in reality this consumes even more RAM.
In the safe space of .Net, I don't think you can immediately drop the RAM. I think you will just have to create a final list of numbers and drop all references to the original array and wait for the garbage collector to do its job, or see if you can collect faster by calling GC.Collect().
In the unsafe space, however, you can control your RAM usage.
I will stop here for now because I just read an incoming comment from you stating that you need to delete the space but keep the size. You are using an array of a primitive data type. This array, once initialized, cannot be deleted, regardless of being in a safe or unsafe context.
In your case, I would then use a dictionary to save the remaining values and drop the entire original array. Line in your comment example, where you have 8 elements and you are conserving every 3rd element. Your dictionary will have 3 elements, then you drop the entire original array. The dictionary's key will tell you the array's original index, while its value will give you, well, the value contained in that position.

What if you initialize a second array
int[] arrCopy = new int[1000000000];
Copy the first array into the second
Array.Copy(arr, arrCopy, 1000000000);
Re-initialize the first array
arr = new int[1000000000];
Iterate over the second array and store every Nth value into the first
for (int i = 0 ; i < 1000000000; i += N){
arr[i] = i;
}
Re-initialize the second array
arrCopy = new int[1000000000];

"I i can traverse a list using indexing as i do in an array.Why can't i add an item in a list as i do in an array using indexing."

Why do I need to use the Add() to add elements to a List. Why can't I use indexing and do it. When I traverse the elements through the List I do it using the help of indexes.
int head = -1;
List<char> arr = new List<char>();
public void push(char s)
{
++head;
arr[head] = s;//throws runtime error.
arr.Add(s);
}
It doesn't throw any error during compile time. But throws an error at runtime stating IndexOutOfRangeException.

++head;
arr[head] = s;
This attempts to set element 1 of the list to s, but there is no element 1 yet because you've not added anything, or set the length of the list.
When you create an array, you define a length, so each item has a memory address that can be assigned to.
Lists are useful when you don't know how many items you're going to have, or what their index is going to be.

Arrays are fixed sizes. Once you allocate them, you can not add or remove "slots" from it. So if you need it to be bigger, you need to:
Detect that you need a bigger array.
Allocate a new, bigger array
copy all existing values to teh new, bigger array
start using the bigger array from now on everywhere
All that Lists do is automate that precise process. It will automatically detect that it needs to increase during Add() and then do step 2-4 automagically. It is even responsible to pick the initial size and by how much to grow it (to avoid having to grow to often.
They could in theory jsut react to List[11000] by growing the size to 11000. But chances are very big, that this value is a huge mistake. And preventing the Progarmmer from doing huge mistakes is what half the classes and compiler rules (like strong typisation) are there for. So they force you to use Add() so such a mistake can not happen.

Actually calling myArray[2] does not add the element, but just assigns the object to the specified index within the array. If the array´s size is less you´d get an IndexOutOfBoundsException, as in a list<T> also. So also in case of an array using the indexer assumes you actually have that many elements:
var array = new int[3];
array[5] = 4; // bang
This is because arrays have a fixed size which you can´t change. If you assign an object to an index greater the arrays size you get the exat same exception as for a List<T> also, there´s no difference here.
The only real difference here is that when using new array[3] you have an array of size 3 with indices up to 2 and you can call array[2]. However this would just return the default-value - in case of int this is zero. When using new List<int>(3) in contrast you don´t have actually three elements. In fact the list has no items at all and calling list[2] throws the exception. The parameter to a list is just the capacity, which is a parameter for the runtime to indicate when the underlying array of a list should be resized - an ability your array does not even have.

A list is an array wrapper, where the internal array size is managed by its methods. The constructor that takes a capacity simply creates an array of that size internally, but the count property (which reflects the count elements that has been added) will be zero. So in essence, zero slots in the array has been assigned a value.
The size of an array is managed by you the programmer. That is why you have to call static methods like System.Array.Resize (notice that the array argument is ref), if you want to change an array yourself. That method allocates a new chunk of memory for the new size.
So to sum up, the list essentially manages an array for you, and as such, the tradeoff is that you can only access as many array-like slots as has been added to it.

How to create an array without knowing how many values will be store?

In C# I'm storing values in an array.
So to create this array I'm using this code, 'int[] values = new int[10];'
But, what if I need more than 10 values, or in the case I never know how many values I will have. Could be 1, 10 or 100.
I understand the idea that I need to let the compiler know how big the array should be so it can allocate memory space for it.
Is there a way to work around that?

You could just use a List and let it do all the heavy lifting for you:
List<int> values = new List<int>();

Arrays must have defined length. If you want dynamic size, consider using List class.

Please take a look at and research the concept of "Immutable objects"
An array has a fixed size, If you need an array with a dynamic size it is best to either create extension methods or a handler that does the work for you.
The work to be done is to get the array, create a new array with the new size based on whether you want to add or remove something, and to populate the new array with the data from the previous array. This will create a new object instead of modifying the previous object and will make sure you don't push items to a full array, or have an array with a size larger than the items that fit in it.
Ofcourse the List class would work as well and would probably solve your problem.

What collection could I use when arrays may seem out of the question? Does the data need to be immutable and can I have a position?

Not long ago I posted a question about a design decision of my F1 game. One person replying noted my use of Dictionary<int, Driver> and asked what it was for. I use this collection for having a finish position and a Driver at that finish position. He suggested I should use arrays, because arrays have a index that could be used as position.
Having not worked on the game for a while, I picked up where I left and started to think about replacing my Dictionaries with arrays. I started googling arrays and one thing let to another and I came across posts here on SO, some blogs (http://blogs.msdn.com/b/ericlippert/archive/2008/09/22/arrays-considered-somewhat-harmful.aspx) about not to use arrays (or, actually, in some very specific cases).
My questions are these:
When data comes from the database, and is only used for comparing and looking up data, what is best used? I have a
prediction, which contains drivers and a predicted finish position
result, which also contains drivers and a finish position
points, which contains the points for every correctly guessed Driver at a certain finish position
The data comes from the database, and shouldn't be edited, added to, or removed.
The data is only used by my code only, but does that mean that I can safely use a collection, or should I really be on the safe side anyway and make sure that what comes back from the database is an immutable collection?
And the second question has to do with the positions. What collection can I use so that I can have finish positions, without maybe the overhead of using a Dictionary?
var prediction = new Driver[3];
prediction[0] = new Driver(10, "Michael Schumacher");
prediction[1] = new Driver(7, "Felipe Massa");
prediction[2] = new Driver(8, "Jensen Button");
var results = new Driver[3];
results[0] = new Driver(10, "Michael Schumacher");
results[1] = new Driver(8, "Jensen Button");
results[2] = new Driver(9, "Fernando Alonso");
int[] points = { 25, 18, 15, 12, 10 };
for (int i = 0; i < prediction.Length; i++)
{
if (prediction[i].Equals(results[i]))
{
result += points[i];
}
}

It depends on if you want to check by position (e.g. "Who finished third?") or by driver (e.g. "What position did Wilson finish at?"). It also depends on whether you care that much about performance in a scenario which sounds like it won't really make a difference to your application as a whole.
Generally a dictionary would be good for lookups. However, if the range of the keys is known, small, and continuous ("race position" fits all three criteria) then a dictionary does not offer anything more than an array (and has more overhead). So if you want to query by position, use an array because it's the simplest tool that fits the job description.
If you want to query by driver, then first you have to answer the question "how exactly is the identity of a Driver defined?" (an answer might be "each driver is uniquely identified by their last name, first name and date of birth").
You could then make the Driver class implement IEquatable<Driver> using these criteria and put it in a Dictionary<Driver, Positions> where I am using the class below to avoid multiple dictionaries.
class Positions
{
public int Predicted { get; set; }
public int Actual { get; set; }
}
However, you have to ask yourself if going to all this trouble makes any sense vs. having an array of e.g. Tuple<Driver, Positions> and looping over the array to find the driver you want each time. This may sound inefficient in theory, but if you have just 20 drivers it will actually be much faster than a dictionary.
Finally, I don't see any need in going out of your way to mark these data structures as immutable and/or enforce immutability. Since your application works in read-only mode, and it does not include code that attempts to push data back to the database, I think it's pretty clear that it's meaningless to go around modifying the data in code.

I use this collection for having a finish position and a Driver at
that finish position. He suggested I should use arrays, because arrays
have a index that could be used as position.
And for that case, he's pretty much right. An array is really the best way to do that because you know how many positions there are and the index acts as the position. It's going to use the least memory and get the most performance, and it's simple. I mean you could also use a List, which for this purpose is really just a fancier array.
There might be cases where arrays are "harmful", but there's also a case to be made that you should stick to simple tools when that's all you need to do the job. Arrays are simple, easily understood, and fast. When all you need is a Driver and a finish position, an Array is a perfect tool.
When data comes from the database, and is only used for comparing and
looking up data, what is best used? I have a
prediction, which contains drivers and a predicted finish position result, which also contains drivers and a finish position points,
which contains the points for every correctly guessed Driver at a
certain finish position
That depends. Do you want one collection that stores all of that information? If you do, Jon's suggestion of a Dictionary is probably your best bet. I suspect that's going to be a good way to go because it's got everything you need in a single place that you can work with.
If instead you want a different collection for each one, then for results and predicted results you can just use an Array for each one (as the data is perfectly suited for it). For points it's a bit different because the points aren't likely to be a perfectly set of index values and you could have two results with the same score.

Array of an unknown length in C#

I've just started learning C# and in the introduction to arrays they showed how to establish a variable as an array but is seems that one must specify the length of the array at assignment, so what if I don't know the length of the array?

Arrays must be assigned a length. To allow for any number of elements, use the List class.
For example:
List<int> myInts = new List<int>();
myInts.Add(5);
myInts.Add(10);
myInts.Add(11);
myInts.Count // = 3

Use List<> to build up an 'array' of unknown length.
Use List<>.ToArray() to return a real array, and not a List.
var list = new List<int>();
list.Add(1);
list.Add(2);
list.Add(3);
var array = list.ToArray();

A little background information:
As said, if you want to have a dynamic collection of things, use a List<T>. Internally, a List uses an array for storage too. That array has a fixed size just like any other array. Once an array is declared as having a size, it doesn't change. When you add an item to a List, it's added to the array. Initially, the List starts out with an array that I believe has a length of 16. When you try to add the 17th item to the List, what happens is that a new array is allocated, that's (I think) twice the size of the old one, so 32 items. Then the content of the old array is copied into the new array. So while a List may appear dynamic to the outside observer, internally it has to comply to the rules as well.
And as you might have guessed, the copying and allocation of the arrays isn't free so one should aim to have as few of those as possible and to do that you can specify (in the constructor of List) an initial size of the array, which in a perfect scenario is just big enough to hold everything you want. However, this is micro-optimization and it's unlikely it will ever matter to you, but it's always nice to know what you're actually doing.

You can create an array with the size set to a variable, i.e.
int size = 50;
string[] words = new string[size]; // contains 50 strings
However, that size can't change later on, if you decide you need 100 words. If you need the size to be really dynamic, you'll need to use a different sort of data structure. Try List.

Use an ArrayList if in .NET 1.x, or a List<yourtype> if in .NET 2.0 or 3.x.
Search for them in System.Collections and System.Collections.Generics.

You might also want to look into Dictionarys if your data is unique, This will give you two columns to work with.
User name , Total bill
it gives you a lot of built in tools to search and update just the value.

var yummy = new List<string>();
while(person.FeelsHappy()) {
yummy.Add(person.GetNewFavoriteFood());
}
Console.WriteLine("Sweet! I have a list of size {0}.", list.Count);
Console.WriteLine("I didn't even need to know how big to make it " +
"until I finished making it!");

try a generic list instead of array

In a nutshell, please use Collections and Generics.
It's a must for any C# developer, it's worth spending time to learn :)

As detailed above, the generic List<> is the best way of doing it.
If you're stuck in .NET 1.*, then you will have to use the ArrayList class instead. This does not have compile-time type checking and you also have to add casting - messy.
Successive versions have also implemented various variations - including thread safe variants.

If you really need to use an array instead of a list, then you can create an array whose size is calculated at run time like so...
e.g i want a two dimensional array of size n by n. n will be gotten at run time from the user
int n = 0;
bool isInteger = int.TryParse(Console.ReadLine(), out n);
var x = new int[n,n];