The limitation on the size of .Net array - c#

I heard that there is a hard limit on the size of .Net Array. It is said that the maximum amount of memory that can be allocated to any single instance of an Array object ( regardless of whether it's int[], double[] or your own array) is 2GB. And no, if you have a 64 bit machine, the 2GB limit is still there.
I'm not sure whether my impression is correct or not. Anyone can confirm?

In versions of .NET prior to 4.5, the maximum object size is 2GB. From 4.5 onwards you can allocate larger objects if gcAllowVeryLargeObjects is enabled. Note that the limit for string is not affected, but "arrays" should cover "lists" too, since lists are backed by arrays.

That is correct. No single object can be larger than 2 GB.
As with 32-bit Windows operating
systems, there is a 2GB limit on the
size of an object you can create while
running a 64-bit managed application
on a 64-bit Windows operating system.
This question has additional details and some useful links: Single objects still limited to 2 GB in size in CLR 4.0?

You will run into a practical limit first - it is pretty impossible to get a 2gb array allocated. Practical limits I have encountered are around the 800mb mark AT PROGRAM START - going down drastically after that.
Anything larger than 64mb is a luck gamble on 32 bit - the large object heap is not defragmented, so you need 65mb free in one piece or allocation fails.
Theoretical limits are:
usable memory, especially under 32 bit.
32 bit number space for index (0 upward - no negative numbers for arrays UNLESS YOU PLAY SMART IN CREATION). You can create arrays allowing negative numbers, but not with C# standard syntax - only with reflection.
2gb per object.
But seriously, the practical implications are larger.
For .NET 4.0.... consider using memory mapped files ;)

I would have thought that the limit might be on the index. I thought that index used has to be an integer so anything bigger than integer wouldn't work unless they have some way around that. So that would be 4294967296 elements. Not sure if this is even half true. I would like to know the answer myself.
EDIT:
As tomtom pointed out, integer is usually signed unless they using a non signed integer. So half of 4294967296 or 2147483648 roughly.

Since .NET 6, the maximum number of elements an array can hold is defined by Array.MaxLength. It is currently 0x7FFFFFC7.
While strings are array-like, they have a lower limit. Currently the longest you cam make a string is 0x3FFFFFDF.

Hope this help: http://blogs.msdn.com/joshwil/archive/2005/08/10/450202.aspx
i.e.
It uses int as index, which has max value = 2,147,483,647 (2GB)
Its by design. 2.

Related

What is the point of LongLength Property in Array class in c#? [duplicate]

What is the purpose of the LongLength property for arrays in .Net. Using a standard integer for length, you could accommodate up to 2 billion indices. Are there really people using .Net to maintain a single array with more the 2 billion elements. Even if each element was a single byte, that would still be 2 GB of data. Is it feasible to use such a large array in .Net?
For example, if you had a > 2 GB file and needed to read it all into memory at once, that would call for such an array. Not that that is necessarily a recommended approach most of the time, but there could well be some case (on a powerful enough 64 bit system with a lot of memory anyway) that this might be required (maybe for performance reasons?).
Edit: Of course, it should be noted that as of CLR 2.0, having an array > 2 GB isn't actually supported (all the implementation of LongLength does is cast Length into a long, and attempting to create a bigger array will fail)... but maybe Microsoft is planning to add support later...?
There's a school of thought known as the "0, 1 or N" school which believes that you should have either none of something; one of something; or any number of something, as resources permit.
In other words, don't set arbitrary limits if you don't have to. Arbitrary limits have given us such monstrosities as:
the 640K limit in early PCs.
buffer overflow vulnerabilities.
the hideous CHS disk addressing scheme.
Keep in mind that even two billion 64-bit integers only takes up
17,179,869,184 bytes of the
18,446,744,073,709,551,616 bytes of 64-bit address space available.
That's less than 1 thousand-millionth, or 10-9, or you could have many millions of these massive arrays before running out of address space.
Plus it returns the total number of elements in all the dimensions of the Array so it can be an array with "just" a half billion elements and 4 dimensions to make it needed to be 64-bit int.
It's very possible to have an array with more than 2 Billion entries in a 64 bit scenario. LongLength is indeed meant to support such scenarios.
As to whether or not that is actually used. I can say with certainty that there is some customer, somewhere, that considers this a vital business need. Customers find uses for features that you've never thought possible.

Defining an enumeration with default properties, Its effect on performance

Enumeration's default value is integer. However, when we use Enumeration we don't use so many values. So my questions are:
enum TYPE : byte{HORIZONTAL , DIAGONAL} //uses 1 byte
enum TYPE {HORIZONTAL , DIAGONAL} // int by default. Uses 4 bytes
1)Does 3 byte space save us so much space? How does it effect nowadays computers?
2)If yes, why its default value isn't byte?
3)What should a good programmer do?
P.S I apologize for bad english it is not my native language.
1) Does 3 byte space save us so much space? How does it effect nowadays computers?
Whether it actually saves space depends on a lot of other things. For example, if you use your enum as a field inside another class, things might (or might not) get memory aligned. I am not sure about the specifics in C# (even in C++ this can be a complex topic) - maybe this SO answer clarifies some things.
Even if it were guaranteed that using a byte enum always saves you three bytes, consider how many enum instances you use in a typical application. Ten? One hundred? Maybe a thousand? We are talking in the order of kilobytes here, while modern computers have at least gigabytes of RAM, that is 6 order of magnitude more.
2) If yes, why its default value isn't byte?
This is a decision by the team that designed C#, we can only assume they had their reasons.
Yes, a byte can only hold values up to 255. Now probably most enumerations don't have more than 255 values, but for example for Flags enums that only gives you 8 possible flags and you may need more. Also, since enums are basically just integer numbers, you want to be able to do integer or bitwise operations on them (like + or |) -- again this is especially true if you have enum Flags -- and those work on full 32- or 64-bit integers anyway. In fact they may even be slower for bytes because the value has to be expanded to a full integer and then truncated again.
3) What should a good programmer do?
As usual, unless you are writing performance or memory critical applications, don't worry about it. Especially in C# with its smart compilers, it is easy to apply premature "optimizations" that actually slow down the performance, so unless you have very convincing evidence that you actually need those three extra bytes, just write the code for legibility and not for speed. Even if the theoretical 3 byte difference actually turns out to be there in practice, you probably lose a multitude of that in other "inefficiencies" such as padded classes, inefficient string operations, copies of local variables, etc.
I've never experienced an issue defining Enums with default Integral type, and you seem not to have an issue yourself. If you ever experience a memory issue in your system it is more likely that it is caused by other issues and not by choosing int over byte for your Enums.
Those little micro-Enhancements has rare impact in today's cheap memory computers.
Unless you're working on embedded systems with limited memory, try not to focus on those micro-optimizations.

What is the Max size of Two Dimensional character array?

I have one two dimensional array:
char[,] DataFile;
When I create a object:
DataFile=new char[45000,6000]
It throws an out of memory exception.
What is the Max Size of object in .Net 3.5? What Is the Max Length of char array?
Single objects still limited to 2 GB in size in CLR 4.0? already has quite a nice explanation of the limits in various circumstances.
Well, it depends.
Obviously it'll matter how much physical memory (RAM) you've got installed and/or how large you set up virtual memory (swap).
In any case, in 32bit Windows maximum object size is 2GB. But there's another limit: The process image must have a contiguous block of memory of the required size.
Your array is about 514MB large. You should check for yourself if you have sufficient resources available.
There is no actual limit, it just depends on how much RAM your computer has, and how much contiguous memory the runtime can allocate.

Should I use 'long' instead of 'int' on 64-bits in langs with fixed type size (like Java, C#)

In 10, or even 5 years there will be no [Edit2: server or desktop] 32-bit CPUs.
So, are there any advantages in using int (32bit) over long (64bit) ?
And are there any disadvantages in using int ?
Edit:
By 10 or 5 years I meant on vast majority of places where those langs are used
I meant which type to use by default. This days I won't even bother to think if I should use short as cycle counter, just for(int i.... The same way long counters already win
registers are already 64-bit, there is already no gain in 32 bit types. And I think some loss in 8 bit types (you have to operate on more bits then you're using)
32-bit is still a completely valid data type; just like we have 16-bit and bytes still around. We didn't throw out 16-bit or 8-bit numbers when we moved to 32-bit processors. A 32-bit number is half the size of a 64-bit integer in terms of storage. If I were modeling a database, and I knew the value couldn't go higher than what a 32-bit integer could store; I would use a 32-bit integer for storage purposes. I'd do the same thing with a 16-bit number as well. A 64-bit number takes more space in memory as well; albeit not anything significant given today's personal laptops can ship with 8 GB of memory.
There is no disadvantage of int other than it's a smaller data type. It's like asking, "Where should I store my sugar? In a sugar bowl, or a silo?" Well, that depends on entirely how much sugar you have.
Processor architecture shouldn't have much to do with what size data type you use. Use what fits. When we have 512-bit processors, we'll still have bytes.
EDIT:
To address some comments / edits..
I'm not sure about "There will be no 32-bit desktop CPUs". ARM is currently 32-bit; and has declared little interest in 64-bit; for now. That doesn't fit too well with "Desktop" in your description; but I also think in 5-10 years the landscape of the type of devices we are writing software will drastically change as well. Tablets can't be ignored; people will want C# and Java apps to run on them, considering Microsoft officially ported Windows 8 to ARM.
If you want to start using long; go ahead. There is no reason not to. If we are only looking at the CPU (ignoring storage size), and making assumptions we are on an x86-64 architecture, then it doesn't make much difference.
Assuming that we are sticking with the x86 architecture; that's true as well. You may end up with a slightly larger stack; depending on whatever framework you are using.
If you're on a 64-bit processor, and you've compiled your code for 64-bit, then at least some of the time, long is likely to be more efficient because it matches the register size. But whether that will really impact your program much is debatable. Also, if you're using long all over the place, you're generally going to use more memory - both on the stack and on the heap - which could negatively impact performance. There are too many variables to know for sure how well your program will perform using long by default instead of int. There are reasons why it could be faster and reasons why it could be slower. It could be a total wash.
The typical thing to do is to just use int if you don't care about the size of the integer. If you need a 64-bit integer, then you use long. If you're trying to use less memory and int is far more than you need, then you use byte or short.
x86_64 CPUs are going to be designed to be efficient at processing 32-bit programs and so it's not like using int is going to seriously degrade performance. Some things will be faster due to better alignment when you use 64-bit integers on a 64-bit CPU, but other things will be slower due to the increased memory requirements. And there are probably a variety of other factors involved which could definitely affect performance in either direction.
If you really want to know which is going to do better for your particular application in your particular environment, you're going to need to profile it. This is not a case where there is a clear advantage of one over the other.
Personally, I would advise that you follow the typical route of using int when you don't care about the size of the integer and to use the other types when you do.
Sorry for the C++ answer.
If the size of the type matters use a sized type:
uint8_t
int32_t
int64_t
etc
If the size doesn't matter use an expressive type:
size_t
ptrdiff_t
ssize_t
etc
I know that D has sized types and size_t. I'm not sure about Java or C#.

Memory limitations in a 64-bit .Net application?

On my laptop, running 64 bit Windows 7 and with 2 Gb of free memory (as reported by Task Manager), I'm able to do:
var x = new Dictionary<Guid, decimal>( 30 * 1024 *1024 );
Without having a computer with more RAM at my hands, I'm wondering if this will scale so that on a computer with 4 Gb free memory, I'll be able to allocate 60M items instead of "just" 30M and so on?
Or are there other limitations (of .Net and/or Windows) that I'll bump into before I'm able to consume all available RAM?
Update: OK, so I'm not allowed to allocate a single object larger than 2 Gb. That's important to know! But then I'm of course curious to know if I'll be able to fully utilize all memory by allocating 2 Gb chunks like this:
var x = new List<Dictionary<Guid, decimal>>();
for ( var i = 0 ; i < 10 ; i++ )
x.Add( new Dictionary<Guid, decimal>( 30 * 1024 *1024 ) );
Would this work if the computer have >20Gb free memory?
There's a 2 GiB limitation on all objects in .NET, you are never allowed to create a single object that exceeds 2 GiB. If you need a bigger object you need to make sure that the objects is built from parts smaller than 2 GiB, so you cannot have an array of continuous bits larger than 2 GiB or a single string longer larger than 512 MiB, I'm not entirely sure about the string but I've done some testing on the issue and was getting OutOfMemoryExceptions when I tried to allocate strings bigger than 512 MiB.
These limits though are subject to heap fragmentation and even if the GC does try to compact the heap, large objects (which is somewhat of an arbitrary cross over around 80K) end up on the large object heap which is a heap that isn't compacted. Strictly speaking, and somewhat of a side note, if you can maintain short lived allocations below this threshold it would be better for your overall GC memory management and performance.
Update: The 2Gb single-object memory limit has been lifted on 64 bit with the release of .NET 4.5.
You'll need to set gcAllowVeryLargeObjects in your app.config.
The maximum number of elements in an array is still 2^32-1, though.
See Single objects still limited to 2 GB in size in CLR 4.0? for more details.

Categories

Resources