I am getting a matrix (from file.txt) of known size into the same size of 2D array. This code is fine for that.
But now I am really looking for extending this, like getting a matrix of unknown size into a 2D array, i.e. using a dynamically sized 2D array.
Here is the C# code I am modifying.
int[,] matrix = new int[3,3];
int i = 0, j = 0, k = 0;
#region Reading Matrices From Files
// Matrix 1 Manipulation...............
// Read the file as one string.
string text = System.IO.File.ReadAllText("file.txt");
//reads all the text of file of given path in a string
foreach (var row in text.Split('\n'))
{
//outter foreach loop is for setting number of rows equal to number lines by splittin over \n
j = 0;
foreach (var col in row.Trim().Split(' '))
{
//inner foreach loop is for setting number of columns equal to number chracters by splittin over space
matrix[i, j] = int.Parse(col.Trim());
j++;
}
i++;
}
Any help is appreciated....
An array is a fixed-sized data structure. It is well used in many languages. Unfortunately once created you may not modify its contents.
That is, how do Lists resize?
List is implemented with arrays, actually. So initially a list holds a fixed number of objects. When you want to add an object that exceeds this capacity, we copy the current array into an array of bigger size, thus increasing the capacity dynamically.
Using this idea, we can actually create your matrix class that automatically resizes from array, rather than List.
I hope the above set your mind thinking about how to create a matrix from arrays! If not, here are more hints:
keep two integers tracking the capacity of your current arrays. If you want to add an element that exceeds one (say you add an entry to a row but the row is already full), then you want to do a resize operation
resize operation consists of initialising a new array with bigger (usually double) size, then copy all the values
don't forget to check that all arrays are of the same size after resize! You might need a loop there
Related
I've ran into an issue with lists as I need to create a 2-dimensional list where I can read data by giving the columns and rows, so I could read from my list by using my_List[col][row] so is it possible making a 2D list that way?
How much performance impact can this have and anything I should be aware that could have a performance impact on the code? I might need to read a few hundred times per second from my 2D list
Is it possible to have a more grid type 2D list so if i have data in 3, 4 and 5 but i dont have anything in 0, 1, and 2 think of it like coordinates. so can I read from the list using myList[3][5] and get the data from there with 0, 1 and 2 having nothing? or do i need to loop it through and add something there like null?
thanks in advance!
Yes, you can indeed use multidimensional arrays or jagged arrays for storing "2D data".
As for creating a data structure that doesn't use any memory space for unused indexes, an option could be to use a dictionary where the keys are tuples of two numbers, like this (assuming that your data are strings):
var items = new Dictionary<(int, int), string>();
items.Add((0,1), "0-1"); //this throws an error if the key already exists
items[(2,3)] = "2-3"; //this silently replaces the value if the key already exists
Console.WriteLine(items.Keys.Contains((0,1))); //true
Console.WriteLine(items.Keys.Contains((0,2))); //false
Console.WriteLine(items[(2,3)]); //"2-3"
Of course you probably want to encapsulate this functionality in its own class, but you get the idea.
Note however that this dictionary approach will probably be worse than a plain array in terms of performance, but it's up to you to experiment and collect some metrics.
You can create 2D Arrays like this :
string[,] twoDArray = new string[2,2];
Then you can loop through it like :
for (int i = 0; i < twoDArray.Length; i++)
{
foreach (int j in twoDArray[i,0])
{
}
}
You can also create 2D Lists like this:
List<List<string>> grid = new List<List<string>>();
and iterate through them using an Enumerator and for example a for loop:
var enu = grid.GetEnumerator();
while (enu.MoveNext())
{
for(int i = 0; i < enu.Current.Count; i++)
{
enu.Current.RemoveAt(i);
}
}
You are basically iterating over all lists and then through each list as long as its size is. Inside the for loop you can alter the encapsuled lists in whatever way you like.
I'm trying to initialize an array in three dimension to load a voxel world.
The total size of the map should be (2048/1024/2048). I tried to initialize an jagged array of "int" but I throw a memory exception. What is the size limit?
Size of my table: 2048 * 1024 * 2048 = 4'191'893'824
Anyone know there a way around this problem?
// System.OutOfMemoryException here !
int[][][] matrice = CreateJaggedArray<int[][][]>(2048,1024,2048);
// if i try normal Initialization I also throws the exception
int[, ,] matrice = new int[2048,1024,2048];
static T CreateJaggedArray<T>(params int[] lengths)
{
return (T)InitializeJaggedArray(typeof(T).GetElementType(), 0, lengths);
}
static object InitializeJaggedArray(Type type, int index, int[] lengths)
{
Array array = Array.CreateInstance(type, lengths[index]);
Type elementType = type.GetElementType();
if (elementType != null)
{
for (int i = 0; i < lengths[index]; i++)
{
array.SetValue(
InitializeJaggedArray(elementType, index + 1, lengths), i);
}
}
return array;
}
The maximum size of a single object in C# is 2GB. Since you are creating a multi-dimensional array rather than a jagged array (despite the name of your method) it is a single object that needs to contain all of those items, not several. If you actually used a jagged array then you wouldn't have a single item with all of that data (even though the total memory footprint would be a tad larger, not smaller, it's just spread out more).
Thank you so much to all the staff who tried to help me in understanding and solving my problem.
I tried several solution to be able to load a lot of data and stored in a table.
After two days, here are my tests and finally the solution which can store 4'191'893'824 entry into one array
I add my final solution, hoping someone could help
the goal
I recall the goal: Initialize an integer array [2048/1024/2048] for storing 4'191'893'824 data
Test 1: with JaggedArray method (failure)
system out of memory exception thrown
/* ******************** */
/* Jagged Array method */
/* ******************** */
// allocate the first dimension;
bigData = new int[2048][][];
for (int x = 0; x < 2048; x++)
{
// allocate the second dimension;
bigData[x] = new int[1024][];
for (int y = 0; y < 1024; y++)
{
// the last dimension allocation
bigData[x][y] = new int[2048];
}
}
Test 2: with List method (failure)
system out of memory exception thrown (divide the big array into several small array .. Does not work because "List <>" allows a maximum of "2GB" Ram allocution like a simple array unfortunately.)
/* ******************** */
/* List method */
/* ******************** */
List<int[,,]> bigData = new List<int[,,]>(512);
for (int a = 0; a < 512; a++)
{
bigData.Add(new int[256, 128, 256]);
}
Test 3: with MemoryMappedFile (Solution)
I finally finally found the solution!
Use the class "Memory Mapped File" contains the contents of a file in virtual memory.
MemoryMappedFile MSDN
Use with custom class that I found on codeproject here. The initialization is long but it works well!
/* ************************ */
/* MemoryMappedFile method */
/* ************************ */
string path = AppDomain.CurrentDomain.BaseDirectory;
var myList = new GenericMemoryMappedArray<int>(2048L*1024L*2048L, path);
using (myList)
{
myList.AutoGrow = false;
/*
for (int a = 0; a < (2048L * 1024L * 2048L); a++)
{
myList[a] = a;
}
*/
myList[12456] = 8;
myList[1939848234] = 1;
// etc...
}
From the MSDN documentation on Arrays (emphasis added)
By default, the maximum size of an Array is 2 gigabytes (GB). In a
64-bit environment, you can avoid the size restriction by setting the
enabled attribute of the gcAllowVeryLargeObjects configuration element
to true in the run-time environment. However, the array will still be
limited to a total of 4 billion elements, and to a maximum index of
0X7FEFFFFF in any given dimension (0X7FFFFFC7 for byte arrays and
arrays of single-byte structures).
So despite the above answers, even if you set the flag to allow a larger object size, the array is still limited to the 32bit limit of the number of elements.
EDIT: You'll likely have to redesign to eliminate the need for a multidimensional array as you're currently using it (as others have suggested, there are a few ways to do this between using actual jagged arrays, or some other collection of dimensions). Given the scale of the number of elements, it may be best to use a design that dynamically allocates objects/memory as used instead of arrays that have to pre-allocate it. (unless you don't mind using many gigabytes of memory) EDITx2: That is, perhaps you can define data structures that define filled content rather than defining every possible voxel in the world, even the "empty" ones. (I'm assuming the vast majority of voxels are "empty" rather than "filled")
EDIT: Although not trivial, especially if most of the space is considered "empty", then your best bet would be to introduce some sort of spatial tree that will let you efficiently query your world to see what objects are in a particular area. For example: Octrees (as Eric suggested) or RTrees
Creating this object as described, either as a standard array or as a jagged array, is going to destroy the locality of reference that allows your CPU to be performant. I recommend you use a structure like this instead:
class BigArray
{
ArrayCell[,,] arrayCell = new ArrayCell[32,16,32];
public int this[int i, int j, int k]
{
get { return (arrayCell[i/64, j/64, k/64])[i%64, j%64, k%16]; }
}
}
class ArrayCell
{
int[,,] cell = new int[64,64,64];
public int this[int i, int j, int k]
{
get { return cell[i,j,k]; }
}
}
I'd like to resize an array, however I don't want all the appended values to be zero.
Array.Resize(ref range, range.Length + grow);
If you assume that the appended length is always a multiple of 4 bytes (just enough to hold a float), what is the best way to fill this up with either float.MaxValue or float.MinValue?
Just loop through the added items and assign the value to it. The compiler will produce code to only do the range checking of the array indexes once outside the loop, so the loop will be very efficient.
int oldSize = range.Length;
Array.Resize(ref range, range.Length + grow);
for (int i = oldSize; i < range.Length; i++) {
range[i] = float.MinValue;
}
You should be aware that the Array.Resize doesn't actually resize the array, but it creates a new array with the desired size and copies the data from the original array. So, if you want a way to resize a collection efficiently, you shouldn't use an array at all. Depending on your needs something like a List, a List of arrays, or a LinkedList would probably be better.
I need to create 2D jagged array. Think of a matrix. The number of rows is known, the number of columns is not known. For example I need to create array of 10 elements, where type of each element string[]. Why do I need that? The number of columns is not known - this function must simply do the allocation and pass array to some other function.
string[][] CreateMatrix(int numRows)
{
// this function must create string[][] where numRows is the first dimension.
}
UPDATE
I have C++ background. In C++ I would write the following (nevermind the syntax)
double ** CreateArray()
{
double **pArray = new *double[10]() // create 10 rows first
}
UPDATE 2
I was considering using List, but I need to have indexed-access to both rows and columns.
return new string[numRows][];
Cannot be done. However you could do something like this:
List<List<string>> createMatrix(int numRows)
{
return new List<List<string>>(numRows);
}
That allows you to be able to have a flexible amount of objects in the second dimesion.
You can write:
string[][] matrix = new string[numRows][];
and that will produce a 2D array of null elements. If you want to fill the array with non-null items, you need to write:
for (int row = 0; row < numRows; row++)
{
matrix[row] = new string[/* col count for this row */];
}
You need to specify the column count for each row of the matrix. If you don't know it in advance, you can either.
Leave the matrix items unintialised, and populate them as you know their size.
Use a fixed number that will give you room enough for the maximum possible size.
Avoid arrays and use List<> as suggested by others here.
Hope this helps.
I have a two dimensional array that I need to load data into. I know the width of the data (22 values) but I do not know the height (estimated around 4000 records, but variable).
I have it declared as follows:
float[,] _calibrationSet;
....
int calibrationRow = 0;
While (recordsToRead)
{
for (int i = 0; i < SensorCount; i++)
{
_calibrationSet[calibrationRow, i] = calibrationArrayView.ReadFloat();
}
calibrationRow++;
}
This causes a NullReferenceException, so when I try to initialize it like this:
_calibrationSet = new float[,];
I get an "Array creation must have array size or array initializer."
Thank you,
Keith
You can't use an array.
Or rather, you would need to pick a size, and if you ended up needing more then you would have to allocate a new, larger, array, copy the data from the old one into the new one, and continue on as before (until you exceed the size of the new one...)
Generally, you would go with one of the collection classes - ArrayList, List<>, LinkedList<>, etc. - which one depends a lot on what you're looking for; List will give you the closest thing to what i described initially, while LinkedList<> will avoid the problem of frequent re-allocations (at the cost of slower access and greater memory usage).
Example:
List<float[]> _calibrationSet = new List<float[]>();
// ...
while (recordsToRead)
{
float[] record = new float[SensorCount];
for (int i = 0; i < SensorCount; i++)
{
record[i] = calibrationArrayView.ReadFloat();
}
_calibrationSet.Add(record);
}
// access later: _calibrationSet[record][sensor]
Oh, and it's worth noting (as Grauenwolf did), that what i'm doing here doesn't give you the same memory structure as a single, multi-dimensional array would - under the hood, it's an array of references to other arrays that actually hold the data. This speeds up building the array a good deal by making reallocation cheaper, but can have an impact on access speed (and, of course, memory usage). Whether this is an issue for you depends a lot on what you'll be doing with the data after it's loaded... and whether there are two hundred records or two million records.
You can't create an array in .NET (as opposed to declaring a reference to it, which is what you did in your example) without specifying its dimensions, either explicitly, or implicitly by specifying a set of literal values when you initialize it. (e.g. int[,] array4 = { { 1, 2 }, { 3, 4 }, { 5, 6 }, { 7, 8 } };)
You need to use a variable-size data structure first (a generic list of 22-element 1-d arrays would be the simplest) and then allocate your array and copy your data into it after your read is finished and you know how many rows you need.
I would just use a list, then convert that list into an array.
You will notice here that I used a jagged array (float[][]) instead of a square array (float [,]). Besides being the "standard" way of doing things, it should be much faster. When converting the data from a list to an array you only have to copy [calibrationRow] pointers. Using a square array, you would have to copy [calibrationRow] x [SensorCount] floats.
var tempCalibrationSet = new List<float[]>();
const int SensorCount = 22;
int calibrationRow = 0;
while (recordsToRead())
{
tempCalibrationSet[calibrationRow] = new float[SensorCount];
for (int i = 0; i < SensorCount; i++)
{
tempCalibrationSet[calibrationRow][i] = calibrationArrayView.ReadFloat();
} calibrationRow++;
}
float[][] _calibrationSet = tempCalibrationSet.ToArray();
I generally use the nicer collections for this sort of work (List, ArrayList etc.) and then (if really necessary) cast to T[,] when I'm done.
you would either need to preallocate the array to a Maximum size (float[999,22] ) , or use a different data structure.
i guess you could copy/resize on the fly.. (but i don't think you'd want to)
i think the List sounds reasonable.
You could also use a two-dimensional ArrayList (from System.Collections) -- you create an ArrayList, then put another ArrayList inside it. This will give you the dynamic resizing you need, but at the expense of a bit of overhead.