Develop Reference

What is at the beginning of an array object in C#? [duplicate]

What is the memory layout of a .NET array?
Take for instance this array:
Int32[] x = new Int32[10];
I understand that the bulk of the array is like this:
0000111122223333444455556666777788889999
Where each character is one byte, and the digits corresponds to indices into the array.
Additionally, I know that there is a type reference, and a syncblock-index for all objects, so the above can be adjusted to this:
ttttssss0000111122223333444455556666777788889999
^
+- object reference points here
Additionally, the length of the array needs to be stored, so perhaps this is more correct:
ttttssssllll0000111122223333444455556666777788889999
^
+- object reference points here
Is this complete? Are there more data in an array?
The reason I'm asking is that we're trying to estimate how much memory a couple of different in-memory representations of a rather large data corpus will take and the size of the arrays varies quite a bit, so the overhead might have a large impact in one solution, but perhaps not so much in the other.
So basically, for an array, how much overhead is there, that is basically my question.
And before the arrays are bad squad wakes up, this part of the solution is a static build-once-reference-often type of thing so using growable lists is not necessary here.

One way to examine this is to look at the code in WinDbg. So given the code below, let's see how that appears on the heap.
var numbers = new Int32[] { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
The first thing to do is to locate the instance. As I have made this a local in Main(), it is easy to find the address of the instance.
From the address we can dump the actual instance, which gives us:
0:000> !do 0x0141ffc0
Name: System.Int32[]
MethodTable: 01309584
EEClass: 01309510
Size: 52(0x34) bytes
Array: Rank 1, Number of elements 10, Type Int32
Element Type: System.Int32
Fields:
None
This tells us that it is our Int32 array with 10 elements and a total size of 52 bytes.
Let's dump the memory where the instance is located.
0:000> d 0x0141ffc0
0141ffc0 [84 95 30 01 0a 00 00 00-00 00 00 00 01 00 00 00 ..0.............
0141ffd0 02 00 00 00 03 00 00 00-04 00 00 00 05 00 00 00 ................
0141ffe0 06 00 00 00 07 00 00 00-08 00 00 00 09 00 00 00 ................
0141fff0 00 00 00 00]a0 20 40 03-00 00 00 00 00 00 00 00 ..... #.........
01420000 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
01420010 10 6d 99 00 00 00 00 00-00 00 01 40 50 f7 3d 03 .m.........#P.=.
01420020 03 00 00 00 08 00 00 00-00 01 00 00 00 00 00 00 ................
01420030 1c 24 40 03 00 00 00 00-00 00 00 00 00 00 00 00 .$#.............
I have inserted brackets for the 52 bytes.
The first four bytes are the reference to the method table at 01309584.
Then four bytes for the Length of the array.
Following that are the numbers 0 to 9 (each four bytes).
The last four bytes are null. I'm not entirely sure, but I guess that must be where the reference to the syncblock array is stored if the instance is used for locking.
Edit: Forgot length in first posting.
The listing is slightly incorrect because as romkyns points out the instance actually begins at the address - 4 and the first field is the Syncblock.

Great question. I found this article which contains block diagrams for both value types and reference types. Also see this article in which Ritcher states:
[snip] each array has some additional
overhead information associated with
it. This information contains the rank
of the array (number of dimensions),
the lower bounds for each dimension of
the array (almost always 0), and the
length of each dimension. The overhead
also contains the type of each element
in the array.

Great question! I wanted to see it for myself, and it seemed a good opportunity to try out CorDbg.exe...
It seems that for simple integer arrays, the format is:
ssssllll000011112222....nnnn0000
where s is the sync block, l the length of the array, and then the individual elements. It seems that there is a finally 0 at the end, I'm not sure why that is.
For multidimensional arrays:
ssssttttl1l1l2l2????????
000011112222....nnnn000011112222....nnnn....000011112222....nnnn0000
where s is the sync block, t the total number of elements, l1 the length of the first dimension, l2 the length of the second dimension, then two zeroes?, followed by all the elements sequentially, and finally a zero again.
Object arrays are treated as the integer array, the contents are references this time. Jagged arrays are object arrays where the references point to other arrays.

An array object would have to store how many dimensions it has and the length of each dimension. So there is at least one more data element to add to your model

Binary strings between cASP and .Net - why are they different?

I'm trying to pass a binary string (which represents a file) from classic ASP (vbscript) to a .Net method. This is of course, very simple indeed. Testing with a simple txt file, we find that the data can be reliably passed to .Net and worked with.
However, as soon as we get to files which include chars in the unicode range, we see that they are routinely converted to different characters and therefore the file is corrupted. For example:
3Ü£ÜkÜGÝ
becomes
ýÿýÿýÿýÿ
So I created a couple of methods, one to write out the hex of the string within the .Net world (passed from vb), and the other to write out the hex in the vb world (the source). here are the results (using the above example text)?
What .Net THINKS we sent it:
fd ff fd ff fd ff fd ff 0d 0a 0d 0a 62 65 63 6f 6d 65 73 0d 0a 0d 0a fd ff fd ff fd ff fd ff 00
What we REALLY sent it:
33 DC A3 DC 6B DC 47 DD 0D 0A 0D 0A 62 65 63 6F 6D 65 73 0D 0A 0D 0A FD FF FD FF FD FF FD FF 00
We can see the word "becomes" has been carried across perfectly (ascii chars) as have the final characters. But the initial characters (3Ü£ÜkÜGÝ) were converted to:
fd ff fd ff fd ff fd ff
Or the string:
ýÿýÿýÿýÿ
What is going on here? Both languages support unicode, is there some odd marshalling of the string happening? Why these particular characters? Thanks.
UPDATE 1
If I take 3Ü in a file and write that out as binary string data in vbscript we get í°³. So what's going on there? I can understand that we now have 3 characters as the first is single byte and the second is double byte. But why isn't it 3°³ (or something similar), why is 3 converted to í etc? How can I get my original data from this string? Of course if I say it's encoded to ASCII, I'll get old ? 3f character. I'm obviously missing a fundamental understanding here.

Which encoding to use for reading a string from a file?

I'm parsing a file (which I don't generate) that contains a string. The string is always preceded by 2 bytes which tell me the length of the string that follows.
For example:
05 00 53 70 6F 72 74
would be:
Sport
Using a C# BinaryReader, I read the string using:
string s = new string(binaryReader.ReadChars(size));
Sometimes there's the odd funky character which seems to push the position of the stream on further than it should. For example:
0D 00 63 6F 6F 6B 20 E2 80 94 20 62 6F 6F 6B
Should be:
cook - book
and although it reads fine the stream ends up two bytes further along than it should?! (Which then messes up the rest of the parsing.)
I'm guessing it has something to do with the 0xE2 in the middle, but I'm not really sure why or how to deal with it.
Any suggestions greatly appreciated!

My guess is that the string is encoded in UTF-8. The 3-byte sequence E2 80 94 corresponds to the single Unicode character U+2014 (EM DASH).

In your first example
05 00 53 70 6F 72 74
none of the bytes are over 0x7F and that happens to be the limit for 7 bit ASCII. UTF-8 retains compability with ASCII by using the 8th bit to indicate that there will be more information to come.
0D 00 63 6F 6F 6B 20 E2 80 94 20 62 6F 6F 6B
Just as Ted noticed your "problems" starts with 0xE2 because that is not a 7 bit ASCII character.
The first byte 0x0D tells us there should be 11 characters but there are 13 bytes.
0xE2 tells us that we've found the beginning of a UTF-8 sequence since the most significant bit is set (it's over 127). In this case a sequence that represents — (EM Dash).

As you did correctly state the E2 character is the problem. BinaryReader.ReadChars(n) does not read n-bytes but n UTF-8 encoded Unicode characters. See Wikipedia for Unicode Encodings. The term you are after are Surrogate Characters. In UTF-8 characters in the range of 000080 – 00009F are represented by two bytes. This is the reason for your offset mismatch.
You need to use BinaryReader.ReadBytes to fix the offset issue and the pass it to an Encoding instance.
To make it work you need to read the bytes with BinaryReader and then decode it with the correct encoding. Assuming you are dealing with UTF-8 then you need to pass the byte array to
Encoding.UTF8.GetString(byte [] rawData)
to get your correctly encoded string back.
Yours,
Alois Kraus

Are null terminators part of text encoding?

I'm trying to read a null terminated string from a byte array; the parameter to the function is the encoding.
string ReadString(Encoding encoding)
For example, "foo" in the following encodings are:
UTF-32: 66 00 00 00 6f 00 00 00 6f 00 00 00
UTF-8: 66 6f 6f
UTF-7: 66 6f 6f 2b 41 41 41 2d
If I copied all the bytes into an array (reading up to the null terminator) and passed that array into encoding.GetString(), it wouldn't work because if the string was UTF-32 encoded my algorithm would reach the "null terminator" after the second byte.
So I sort of have a double question: Are null terminators part of the encoding? If not, how could I decode the string character by character and check the following byte for the null terminator?
Thanks in advance
(suggestions are also appreciated)
Edit:
If "foo" was null terminated and utf-32 encoded, which would it be?:
1. 66 00 00 00 6f 00 00 00 6f 00 00 00 00
2. 66 00 00 00 6f 00 00 00 6f 00 00 00 00 00 00 00

The null terminator is not "logically" part of the string; it's not considered payload. It's widely used in C/C++ to indicate where the string ends.
Having said that you can have strings with embedded \0's but then you have to be careful to ensure the string doesn't appear truncated. For example std::string doesn't have a problem with embedded \0's. But if do a c_str() and and not account for the reported length() your string will appear cut off.

Null terminators are not part of the encoding, but the string representation used by some programming language, such as C. In .NET, System.String is prefixed by the string length as a 32-bit integer and is not null-terminated. Internally System.String is always UTF-16, but you can use the encoding to output different representations.
For the second part... Use the classes in System.Text such as UTF8Encoding and UTF32Encoding to read the string. You just have to select the right one based on your parameter...

This seems to work well for me (sample from actual code that reads a unicode, null terminated string from a byte array):
//trim null-termination from end of string
byte[] languageId = ...
string language = Encoding.Unicode.GetString(languageId,
0,
languageId.Length).Trim('\0');

How do I detect frequency of mp3's in .NET?

I want to create a very simple piece of software in C# .NET that I can pass a folder's path to and detect all files with a frequency of below a given threshold. Any pointers on how I would do this?

You have to read mp3 files. To do that you have to find specifications for them.
Generally mp3 file is wrapped into ID3 tag, so that you have to read it, find its length and skip it. Let's take ID3v2.3 for example:
ID3v2/file identifier "ID3"
ID3v2 version $03 00
ID3v2 flags %abc00000
ID3v2 size 4 * %0xxxxxxx
so bytes 6,7,8,9 store header length in big-endian form. Here is sample of some file:
0 1 2 3 4 5 6 7 8 9 A B C D E F
49 44 33 03 00 00 00 00 07 76 54 43 4f 4e 00 00
07 76 - is the size. You need to shift left first byte so that actual size is 3F6. Then add 10 (A) to get the offset = 400. This is address of start of mp3 header.
Then you take description of mp3 header:
bits are: AAAAAAAA AAABBCCD EEEEFFGH IIJJKLMM, we need FF , sampling frequency and convert t to actual frequency:
bits MPEG1 MPEG2 MPEG2.5
00 44100 22050 11025
01 48000 24000 12000
10 32000 16000 8000
11 reserv. reserv. reserv.

You can use UltraID3Lib to get mp3 metadata (bitrate, frequency)

Check value of frequency bits in a file. There is some info about mp3 format.