Develop Reference

Custom 4 bit data type in C#

I want to create a custom data type which is 4 bits (nibble).
One option is this -
byte source = 0xAD;
var hiNybble = (source & 0xF0) >> 4; //Left hand nybble = A
var loNyblle = (source & 0x0F); //Right hand nybble = D
However, I want to store each 4 bits in an array.
So for example, a 2 byte array,
00001111 01010000
would be stored in the custom data type array as 4 nibbles -
0000
1111
0101
0000
Essentially I want to operate on 4 bit types.
Is there any way I can convert the array of bytes into array of nibbles?
Appreciate an example.
Thanks.

You can encapsulate a stream returning 4-bit samples by reading then converting (written from a phone without a compiler to test. Expect typos and off-by-one errors):
public static int ReadNibbles(this Stream s, byte[] data, int offset, int count)
{
if (s == null)
{
throw new ArgumentNullException(nameof(s));
}
if (data == null)
{
throw new ArgumentNullException(nameof(data));
}
if (data.Length < offset + length)
{
throw new ArgumentOutOfRangeException(nameof(length));
}
var readBytes = s.Read(data, offset, length / 2);
for (int n = readBytes * 2 - 1, k = readBytes - 1; k >= 0; k--)
{
data[offset + n--] = data[offset + k] & 0xf;
data[offset + n--] = data[offset + k] >> 4;
}
return readBytes * 2;
}
To do the same for 12-bit integers (assuming MSB nibble ordering):
public static int Read(this Stream stream, ushort[] data, int offset, int length)
{
if (stream == null)
{
throw new ArgumentNullException(nameof(stream));
}
if (data == null)
{
throw new ArgumentNullException(nameof(data));
}
if (data.Length < offset + length)
{
throw new ArgumentOutOfRangeException(nameof(length));
}
if (length < 2)
{
throw new ArgumentOutOfRangeException(nameof(length), "Cannot read fewer than two samples at a time");
}
// we always need a multiple of two
length -= length % 2;
// 3 bytes length samples
// --------- * -------------- = N bytes
// 2 samples 1
int rawCount = (length / 2) * 3;
// This will place GC load. Something like a buffer pool or keeping
// the allocation as a field on the reader would be a good idea.
var rawData = new byte[rawCount];
int readBytes = 0;
// if the underlying stream returns an even number of bytes, we will need to try again
while (readBytes < data.Length)
{
int c = stream.Read(rawData, readBytes, rawCount - readBytes);
if (c <= 0)
{
// End of stream
break;
}
readBytes += c;
}
// unpack
int k = 0;
for (int i = 0; i < readBytes; i += 3)
{
// EOF in second byte is undefined
if (i + 1 >= readBytes)
{
throw new InvalidOperationException("Unexpected EOF");
}
data[(k++) + offset] = (ushort)((rawData[i + 0] << 4) | (rawData[i + 1] >> 4));
// EOF in third byte == only one sample
if (i + 2 < readBytes)
{
data[(k++) + offset] = (ushort)(((rawData[i + 1] & 0xf) << 8) | rawData[i + 2]);
}
}
return k;
}

The best way to do this would be to look at the source for one of the existing integral data types. For example Int16.
If you look a that type, you can see that it implements a handful of interfaces:
[Serializable]
public struct Int16 : IComparable, IFormattable, IConvertible, IComparable<short>, IEquatable<short> { /* ... */ }
The implementation of the type isn't very complicated. It has a MaxValue a MinValue, a couple of CompareTo overloads, a couple of Equals overloads, the System.Object overrides (GetHashCode, GetType, ToString (plus some overloads)), a handful of Parse and ToParse overloads and a range of IConvertible implementations.
In other places, you can find things like arithmetic, comparison and conversion operators.
BUT:
What System.Int16 has that you can't have is this:
internal short m_value;
That's a native type (16-bit integer) member that holds the value. There is no 4-bit native type. The best you are going to be able to do is have a native byte in your implementation that will hold the value. You can write accessors that constrain it to the lower 4 bits, but you can't do much more than that. If someone creates a Nibble array, it will be implemented as an array of those values. As far as I know, there's no way to inject your implementation into that array. Similarly, if someone creates some other collection (e.g., List<Nibble>), then the collection will be of instances of your type, each of which will take up 8 bits.
However
You can create specialized collection classes, NibbleArray, NibbleList, etc. C#'s syntax allows you to provide your own collection initialization implementation for a collection, your own indexing method, etc.
So, if someone does something like this:
var nyblArray = new NibbleArray(32);
nyblArray[4] = 0xd;
Then your code can, under the covers, create a 16-element byte array, set the low nibble of the third byte to 0xd.
Similarly, you can implement code to allow:
var newArray = new NibbleArray { 0x1, 0x3, 0x5, 0x7, 0x9, 0xa};
or
var nyblList = new NibbleList { 0x0, 0x2, 0xe};
A normal array will waste space, but your specialized collection classes will do what you are talking about (with the expense of some bit-twizzling).

The closest you can get to what you want is to use an indexer:
// Indexer declaration
public int this[int index]
{
// get and set accessors
}
Within the body of the indexer you can translate the index to the actual byte that contains your 4 bits.
The next thing you can do is operator overloading. You can redefine +, -, *...

How to read blob data from MySQL in c# [duplicate]

I have a byte[] array that is loaded from a file that I happen to known contains UTF-8.
In some debugging code, I need to convert it to a string. Is there a one-liner that will do this?
Under the covers it should be just an allocation and a memcopy, so even if it is not implemented, it should be possible.

string result = System.Text.Encoding.UTF8.GetString(byteArray);

There're at least four different ways doing this conversion.
Encoding's GetString, but you won't be able to get the original bytes back if those bytes have non-ASCII characters.
BitConverter.ToString The output is a "-" delimited string, but there's no .NET built-in method to convert the string back to byte array.
Convert.ToBase64String You can easily convert the output string back to byte array by using Convert.FromBase64String. Note: The output string could contain '+', '/' and '='. If you want to use the string in a URL, you need to explicitly encode it.
HttpServerUtility.UrlTokenEncodeYou can easily convert the output string back to byte array by using HttpServerUtility.UrlTokenDecode. The output string is already URL friendly! The downside is it needs System.Web assembly if your project is not a web project.
A full example:
byte[] bytes = { 130, 200, 234, 23 }; // A byte array contains non-ASCII (or non-readable) characters
string s1 = Encoding.UTF8.GetString(bytes); // ���
byte[] decBytes1 = Encoding.UTF8.GetBytes(s1); // decBytes1.Length == 10 !!
// decBytes1 not same as bytes
// Using UTF-8 or other Encoding object will get similar results
string s2 = BitConverter.ToString(bytes); // 82-C8-EA-17
String[] tempAry = s2.Split('-');
byte[] decBytes2 = new byte[tempAry.Length];
for (int i = 0; i < tempAry.Length; i++)
decBytes2[i] = Convert.ToByte(tempAry[i], 16);
// decBytes2 same as bytes
string s3 = Convert.ToBase64String(bytes); // gsjqFw==
byte[] decByte3 = Convert.FromBase64String(s3);
// decByte3 same as bytes
string s4 = HttpServerUtility.UrlTokenEncode(bytes); // gsjqFw2
byte[] decBytes4 = HttpServerUtility.UrlTokenDecode(s4);
// decBytes4 same as bytes

A general solution to convert from byte array to string when you don't know the encoding:
static string BytesToStringConverted(byte[] bytes)
{
using (var stream = new MemoryStream(bytes))
{
using (var streamReader = new StreamReader(stream))
{
return streamReader.ReadToEnd();
}
}
}

Definition:
public static string ConvertByteToString(this byte[] source)
{
return source != null ? System.Text.Encoding.UTF8.GetString(source) : null;
}
Using:
string result = input.ConvertByteToString();

Converting a byte[] to a string seems simple, but any kind of encoding is likely to mess up the output string. This little function just works without any unexpected results:
private string ToString(byte[] bytes)
{
string response = string.Empty;
foreach (byte b in bytes)
response += (Char)b;
return response;
}

I saw some answers at this post and it's possible to be considered completed base knowledge, because I have a several approaches in C# Programming to resolve the same problem. The only thing that is necessary to be considered is about a difference between pure UTF-8 and UTF-8 with a BOM.
Last week, at my job, I needed to develop one functionality that outputs CSV files with a BOM and other CSV files with pure UTF-8 (without a BOM). Each CSV file encoding type will be consumed by different non-standardized APIs. One API reads UTF-8 with a BOM and the other API reads without a BOM. I needed to research the references about this concept, reading the "What's the difference between UTF-8 and UTF-8 without BOM?" Stack Overflow question, and the Wikipedia article "Byte order mark" to build my approach.
Finally, my C# Programming for both UTF-8 encoding types (with BOM and pure) needed to be similar to this example below:
// For UTF-8 with BOM, equals shared by Zanoni (at top)
string result = System.Text.Encoding.UTF8.GetString(byteArray);
//for Pure UTF-8 (without B.O.M.)
string result = (new UTF8Encoding(false)).GetString(byteArray);

Using (byte)b.ToString("x2"), Outputs b4b5dfe475e58b67
public static class Ext {
public static string ToHexString(this byte[] hex)
{
if (hex == null) return null;
if (hex.Length == 0) return string.Empty;
var s = new StringBuilder();
foreach (byte b in hex) {
s.Append(b.ToString("x2"));
}
return s.ToString();
}
public static byte[] ToHexBytes(this string hex)
{
if (hex == null) return null;
if (hex.Length == 0) return new byte[0];
int l = hex.Length / 2;
var b = new byte[l];
for (int i = 0; i < l; ++i) {
b[i] = Convert.ToByte(hex.Substring(i * 2, 2), 16);
}
return b;
}
public static bool EqualsTo(this byte[] bytes, byte[] bytesToCompare)
{
if (bytes == null && bytesToCompare == null) return true; // ?
if (bytes == null || bytesToCompare == null) return false;
if (object.ReferenceEquals(bytes, bytesToCompare)) return true;
if (bytes.Length != bytesToCompare.Length) return false;
for (int i = 0; i < bytes.Length; ++i) {
if (bytes[i] != bytesToCompare[i]) return false;
}
return true;
}
}

There is also class UnicodeEncoding, quite simple in usage:
ByteConverter = new UnicodeEncoding();
string stringDataForEncoding = "My Secret Data!";
byte[] dataEncoded = ByteConverter.GetBytes(stringDataForEncoding);
Console.WriteLine("Data after decoding: {0}", ByteConverter.GetString(dataEncoded));

In addition to the selected answer, if you're using .NET 3.5 or .NET 3.5 CE, you have to specify the index of the first byte to decode, and the number of bytes to decode:
string result = System.Text.Encoding.UTF8.GetString(byteArray, 0, byteArray.Length);

Alternatively:
var byteStr = Convert.ToBase64String(bytes);

The BitConverter class can be used to convert a byte[] to string.
var convertedString = BitConverter.ToString(byteAttay);
Documentation of BitConverter class can be fount on MSDN.

To my knowledge none of the given answers guarantee correct behavior with null termination. Until someone shows me differently I wrote my own static class for handling this with the following methods:
// Mimics the functionality of strlen() in c/c++
// Needed because niether StringBuilder or Encoding.*.GetString() handle \0 well
static int StringLength(byte[] buffer, int startIndex = 0)
{
int strlen = 0;
while
(
(startIndex + strlen + 1) < buffer.Length // Make sure incrementing won't break any bounds
&& buffer[startIndex + strlen] != 0 // The typical null terimation check
)
{
++strlen;
}
return strlen;
}
// This is messy, but I haven't found a built-in way in c# that guarentees null termination
public static string ParseBytes(byte[] buffer, out int strlen, int startIndex = 0)
{
strlen = StringLength(buffer, startIndex);
byte[] c_str = new byte[strlen];
Array.Copy(buffer, startIndex, c_str, 0, strlen);
return Encoding.UTF8.GetString(c_str);
}
The reason for the startIndex was in the example I was working on specifically I needed to parse a byte[] as an array of null terminated strings. It can be safely ignored in the simple case

A LINQ one-liner for converting a byte array byteArrFilename read from a file to a pure ASCII C-style zero-terminated string would be this: Handy for reading things like file index tables in old archive formats.
String filename = new String(byteArrFilename.TakeWhile(x => x != 0)
.Select(x => x < 128 ? (Char)x : '?').ToArray());
I use '?' as the default character for anything not pure ASCII here, but that can be changed, of course. If you want to be sure you can detect it, just use '\0' instead, since the TakeWhile at the start ensures that a string built this way cannot possibly contain '\0' values from the input source.

Try this console application:
static void Main(string[] args)
{
//Encoding _UTF8 = Encoding.UTF8;
string[] _mainString = { "Hello, World!" };
Console.WriteLine("Main String: " + _mainString);
// Convert a string to UTF-8 bytes.
byte[] _utf8Bytes = Encoding.UTF8.GetBytes(_mainString[0]);
// Convert UTF-8 bytes to a string.
string _stringuUnicode = Encoding.UTF8.GetString(_utf8Bytes);
Console.WriteLine("String Unicode: " + _stringuUnicode);
}

Here is a result where you didn’t have to bother with encoding. I used it in my network class and send binary objects as string with it.
public static byte[] String2ByteArray(string str)
{
char[] chars = str.ToArray();
byte[] bytes = new byte[chars.Length * 2];
for (int i = 0; i < chars.Length; i++)
Array.Copy(BitConverter.GetBytes(chars[i]), 0, bytes, i * 2, 2);
return bytes;
}
public static string ByteArray2String(byte[] bytes)
{
char[] chars = new char[bytes.Length / 2];
for (int i = 0; i < chars.Length; i++)
chars[i] = BitConverter.ToChar(bytes, i * 2);
return new string(chars);
}

string result = ASCIIEncoding.UTF8.GetString(byteArray);

Writing from a BinaryWriter to a byte[]

quick question. I'm trying to convert an integer to a 7BitEncodedInt through the Binary Writer. Currently it seems the best manner I have to do so is the following:
Stream memoryStream = new MemoryStream();
InternalBinaryWriter ibw = new InternalBinaryWriter(memoryStream);
ibw.Write7BitEncodedInt((int)length);
memoryStream.Position = 0;
memoryStream.Read(object.ByteArray, 0, (int)memoryStream.Length);
Is there a simpler way to do so without requiring the streams? I'm slightly reluctant to reimplement their Write7BitEncodedInt function and just have that output directly to a byte array, but if necessary that's just what I'll do
edit:
What I ended up doing was the following:
the byteArray is resized accordingly, just didn't include that line in the source.
What I ended up doing was the following:
byte[] Write7BitEncodedIntToByteArray(int length) {
List<byte> byteList = new List<byte>();
while (length > 0x80) {
byteList.Add((byte)(length | 0x80));
length >>= 7;
}
byteList.Add((byte)length);
return byteList.ToArray();
}

If you want to write to a byte[] array, using a MemoryStream is the easiest way to do it. In fact, one of the MemoryStream constructors takes a byte array and uses that as the buffer it will write to. So it might be more useful in your case to do something like this:
using( var memoryStream = new MemoryStream( #object.ByteArray ) )
using( var writer = new InternalBinaryWriter( memoryStream ) )
{
writer.Write7BitEncodedInt( (int) length );
}
Keep in mind that this will result in a memory stream that cannot expand its size. Based on the code you supplied I think this will be fine (what you've shown may in fact throw an exception at runtime if #object.ByteArray is smaller than the stream array).
EDIT: Since we're looking at evidently performance-critical code, I tested three different methods:
List
static byte[] MethodA( int length )
{
List<byte> byteList = new List<byte>();
while (length > 0x80) {
byteList.Add((byte)(length | 0x80));
length >>= 7;
}
byteList.Add((byte)length);
return byteList.ToArray();
}
MemoryStream
static byte[] MethodB( int length )
{
using( var stream = new MemoryStream( 5 ) )
{
while( length > 0x80 )
{
stream.WriteByte( (byte) ( length | 0x80 ) );
length >>= 7;
}
stream.WriteByte( (byte) length );
return stream.ToArray();
}
}
Direct Array
static byte[] MethodC( int length )
{
int tmp = length;
int l = 1;
while( ( tmp >>= 7 ) > 0 ) ++l;
var result = new byte[l];
for( int i = 0; i < l; ++i )
{
result[i] = (byte) ( 0x80 | length );
length >>= 7;
}
result[l - 1] &= 0x7f;
return result;
}
By a wide margin, the third method was the fastest (4-5x faster than the previous two). When it comes to performance critical code, creating a List just to turn it into an array is almost always a smell, which is what got my attention here.
It's also possible that another big chunk of time could be getting sucked up elsewhere depending on how this method is used (for example, if this small array is being copied into a larger array every time, you might be able to remove the array allocations altogether and write directly into the larger array).

How windows calculate file dimension?

I Calculate the binary size of a file with this function:
public static int BinarySize(string path)
{
FileStream fs = new FileStream(path, FileMode.Open);
int hexIn;
string ret = "";
for (int i = 0; (hexIn = fs.ReadByte()) != -1; i++)
{
ret += Convert.ToString(hexIn, 2);
}
fs.Close();
return ret.Length;
}
An example of my problem is when I calculate the dimension of this simple black PNG image (10x10 pixels)
With that function I find 640 bits => 80 bytes, but windows say that this file dimension is 136 byte.
Why this difference of 56 bytes? Is the security, permissions or some private information that windows attach to every file?

Convert.ToString(hexIn,2) does not always return 8 characters, it trims leading zeroes, so if hexIn is 4, it returns 100, but not 00000100.
You might want to change it to Convert.ToString(hexIn,2).PadLeft(8, '0');.
Also you'd want to use StringBuilder instead of string for ret variable.
By the way, reading file to determine its size is a bit wasteful. Better use FileInfo class to get file information.

How to convert UTF-8 byte[] to string

I have a byte[] array that is loaded from a file that I happen to known contains UTF-8.
In some debugging code, I need to convert it to a string. Is there a one-liner that will do this?
Under the covers it should be just an allocation and a memcopy, so even if it is not implemented, it should be possible.

string result = System.Text.Encoding.UTF8.GetString(byteArray);

There're at least four different ways doing this conversion.
Encoding's GetString, but you won't be able to get the original bytes back if those bytes have non-ASCII characters.
BitConverter.ToString The output is a "-" delimited string, but there's no .NET built-in method to convert the string back to byte array.
Convert.ToBase64String You can easily convert the output string back to byte array by using Convert.FromBase64String. Note: The output string could contain '+', '/' and '='. If you want to use the string in a URL, you need to explicitly encode it.
HttpServerUtility.UrlTokenEncodeYou can easily convert the output string back to byte array by using HttpServerUtility.UrlTokenDecode. The output string is already URL friendly! The downside is it needs System.Web assembly if your project is not a web project.
A full example:
byte[] bytes = { 130, 200, 234, 23 }; // A byte array contains non-ASCII (or non-readable) characters
string s1 = Encoding.UTF8.GetString(bytes); // ���
byte[] decBytes1 = Encoding.UTF8.GetBytes(s1); // decBytes1.Length == 10 !!
// decBytes1 not same as bytes
// Using UTF-8 or other Encoding object will get similar results
string s2 = BitConverter.ToString(bytes); // 82-C8-EA-17
String[] tempAry = s2.Split('-');
byte[] decBytes2 = new byte[tempAry.Length];
for (int i = 0; i < tempAry.Length; i++)
decBytes2[i] = Convert.ToByte(tempAry[i], 16);
// decBytes2 same as bytes
string s3 = Convert.ToBase64String(bytes); // gsjqFw==
byte[] decByte3 = Convert.FromBase64String(s3);
// decByte3 same as bytes
string s4 = HttpServerUtility.UrlTokenEncode(bytes); // gsjqFw2
byte[] decBytes4 = HttpServerUtility.UrlTokenDecode(s4);
// decBytes4 same as bytes

A general solution to convert from byte array to string when you don't know the encoding:
static string BytesToStringConverted(byte[] bytes)
{
using (var stream = new MemoryStream(bytes))
{
using (var streamReader = new StreamReader(stream))
{
return streamReader.ReadToEnd();
}
}
}

Definition:
public static string ConvertByteToString(this byte[] source)
{
return source != null ? System.Text.Encoding.UTF8.GetString(source) : null;
}
Using:
string result = input.ConvertByteToString();

Converting a byte[] to a string seems simple, but any kind of encoding is likely to mess up the output string. This little function just works without any unexpected results:
private string ToString(byte[] bytes)
{
string response = string.Empty;
foreach (byte b in bytes)
response += (Char)b;
return response;
}

I saw some answers at this post and it's possible to be considered completed base knowledge, because I have a several approaches in C# Programming to resolve the same problem. The only thing that is necessary to be considered is about a difference between pure UTF-8 and UTF-8 with a BOM.
Last week, at my job, I needed to develop one functionality that outputs CSV files with a BOM and other CSV files with pure UTF-8 (without a BOM). Each CSV file encoding type will be consumed by different non-standardized APIs. One API reads UTF-8 with a BOM and the other API reads without a BOM. I needed to research the references about this concept, reading the "What's the difference between UTF-8 and UTF-8 without BOM?" Stack Overflow question, and the Wikipedia article "Byte order mark" to build my approach.
Finally, my C# Programming for both UTF-8 encoding types (with BOM and pure) needed to be similar to this example below:
// For UTF-8 with BOM, equals shared by Zanoni (at top)
string result = System.Text.Encoding.UTF8.GetString(byteArray);
//for Pure UTF-8 (without B.O.M.)
string result = (new UTF8Encoding(false)).GetString(byteArray);

Using (byte)b.ToString("x2"), Outputs b4b5dfe475e58b67
public static class Ext {
public static string ToHexString(this byte[] hex)
{
if (hex == null) return null;
if (hex.Length == 0) return string.Empty;
var s = new StringBuilder();
foreach (byte b in hex) {
s.Append(b.ToString("x2"));
}
return s.ToString();
}
public static byte[] ToHexBytes(this string hex)
{
if (hex == null) return null;
if (hex.Length == 0) return new byte[0];
int l = hex.Length / 2;
var b = new byte[l];
for (int i = 0; i < l; ++i) {
b[i] = Convert.ToByte(hex.Substring(i * 2, 2), 16);
}
return b;
}
public static bool EqualsTo(this byte[] bytes, byte[] bytesToCompare)
{
if (bytes == null && bytesToCompare == null) return true; // ?
if (bytes == null || bytesToCompare == null) return false;
if (object.ReferenceEquals(bytes, bytesToCompare)) return true;
if (bytes.Length != bytesToCompare.Length) return false;
for (int i = 0; i < bytes.Length; ++i) {
if (bytes[i] != bytesToCompare[i]) return false;
}
return true;
}
}

There is also class UnicodeEncoding, quite simple in usage:
ByteConverter = new UnicodeEncoding();
string stringDataForEncoding = "My Secret Data!";
byte[] dataEncoded = ByteConverter.GetBytes(stringDataForEncoding);
Console.WriteLine("Data after decoding: {0}", ByteConverter.GetString(dataEncoded));

In addition to the selected answer, if you're using .NET 3.5 or .NET 3.5 CE, you have to specify the index of the first byte to decode, and the number of bytes to decode:
string result = System.Text.Encoding.UTF8.GetString(byteArray, 0, byteArray.Length);

Alternatively:
var byteStr = Convert.ToBase64String(bytes);

The BitConverter class can be used to convert a byte[] to string.
var convertedString = BitConverter.ToString(byteAttay);
Documentation of BitConverter class can be fount on MSDN.

To my knowledge none of the given answers guarantee correct behavior with null termination. Until someone shows me differently I wrote my own static class for handling this with the following methods:
// Mimics the functionality of strlen() in c/c++
// Needed because niether StringBuilder or Encoding.*.GetString() handle \0 well
static int StringLength(byte[] buffer, int startIndex = 0)
{
int strlen = 0;
while
(
(startIndex + strlen + 1) < buffer.Length // Make sure incrementing won't break any bounds
&& buffer[startIndex + strlen] != 0 // The typical null terimation check
)
{
++strlen;
}
return strlen;
}
// This is messy, but I haven't found a built-in way in c# that guarentees null termination
public static string ParseBytes(byte[] buffer, out int strlen, int startIndex = 0)
{
strlen = StringLength(buffer, startIndex);
byte[] c_str = new byte[strlen];
Array.Copy(buffer, startIndex, c_str, 0, strlen);
return Encoding.UTF8.GetString(c_str);
}
The reason for the startIndex was in the example I was working on specifically I needed to parse a byte[] as an array of null terminated strings. It can be safely ignored in the simple case

A LINQ one-liner for converting a byte array byteArrFilename read from a file to a pure ASCII C-style zero-terminated string would be this: Handy for reading things like file index tables in old archive formats.
String filename = new String(byteArrFilename.TakeWhile(x => x != 0)
.Select(x => x < 128 ? (Char)x : '?').ToArray());
I use '?' as the default character for anything not pure ASCII here, but that can be changed, of course. If you want to be sure you can detect it, just use '\0' instead, since the TakeWhile at the start ensures that a string built this way cannot possibly contain '\0' values from the input source.

Try this console application:
static void Main(string[] args)
{
//Encoding _UTF8 = Encoding.UTF8;
string[] _mainString = { "Hello, World!" };
Console.WriteLine("Main String: " + _mainString);
// Convert a string to UTF-8 bytes.
byte[] _utf8Bytes = Encoding.UTF8.GetBytes(_mainString[0]);
// Convert UTF-8 bytes to a string.
string _stringuUnicode = Encoding.UTF8.GetString(_utf8Bytes);
Console.WriteLine("String Unicode: " + _stringuUnicode);
}

Here is a result where you didn’t have to bother with encoding. I used it in my network class and send binary objects as string with it.
public static byte[] String2ByteArray(string str)
{
char[] chars = str.ToArray();
byte[] bytes = new byte[chars.Length * 2];
for (int i = 0; i < chars.Length; i++)
Array.Copy(BitConverter.GetBytes(chars[i]), 0, bytes, i * 2, 2);
return bytes;
}
public static string ByteArray2String(byte[] bytes)
{
char[] chars = new char[bytes.Length / 2];
for (int i = 0; i < chars.Length; i++)
chars[i] = BitConverter.ToChar(bytes, i * 2);
return new string(chars);
}

string result = ASCIIEncoding.UTF8.GetString(byteArray);