Develop Reference

Converting String to Hex then into Byte Array

So,
I have a string that I want to convert each character to hex values and then put it in a byte array to be sent through a com port.
I can convert the individual characters to the hex that I need to send, but I can't get that array of strings into a byte array correctly.
example:
string beforeConverting = "HELLO";
String[] afterConverting = {"0x48", "0x45", "0x4C", "0x4C", "0x4F"};
should become
byte[] byteData = new byte[]{0x48, 0x45, 0x4C, 0x4C, 0x4F};
I've tried several different things from several different posts but I can't get the right combination of things together. If anyone could point me in the right direction or give me a snippet of example code that would be awesome!

If your final aim is to send byte[], then you can actually skip the middle step and immediately do the conversion from string to byte[] using Encoding.ASCII.GetBytes (provided that you send ASCII char):
string beforeConverting = "HELLO";
byte[] byteData = Encoding.ASCII.GetBytes(beforeConverting);
//will give you {0x48, 0x45, 0x4C, 0x4C, 0x4F};
If you don't send ASCII, you could find the appropriate Encoding type (like Unicode or UTF32), depends on your need.
That being said, if you still want to convert the hex string to byte array, you could do something something like this:
/// <summary>
/// To convert Hex data string to bytes (i.e. 0x01455687) given the data type
/// </summary>
/// <param name="hexString"></param>
/// <param name="dataType"></param>
/// <returns></returns>
public static byte[] HexStringToBytes(string hexString) {
try {
if (hexString.Length >= 3) //must have minimum of length of 3
if (hexString[0] == '0' && (hexString[1] == 'x' || hexString[1] == 'X'))
hexString = hexString.Substring(2);
int dataSize = (hexString.Length - 1) / 2;
int expectedStringLength = 2 * dataSize;
while (hexString.Length < expectedStringLength)
hexString = "0" + hexString; //zero padding in the front
int NumberChars = hexString.Length / 2;
byte[] bytes = new byte[NumberChars];
using (var sr = new StringReader(hexString)) {
for (int i = 0; i < NumberChars; i++)
bytes[i] = Convert.ToByte(new string(new char[2] { (char)sr.Read(), (char)sr.Read() }), 16);
}
return bytes;
} catch {
return null;
}
}
And then use it like this:
byte[] byteData = afterConverting.Select(x => HexStringToBytes(x)[0]).ToArray();
The method I put above is more general which can handle input string like 0x05163782 to give byte[4]. For your use, you only need to take the first byte (as the byte[] will always be byte[1]) and thus you have [0] index in the LINQ Select.
The core method used in the custom method above is Convert.ToByte():
bytes[i] = Convert.ToByte(new string(new char[2] { (char)sr.Read(), (char)sr.Read() }), 16);

To convert just the hexadecimal string to a number, you could use the System.Convert class like so
string hex = "0x3B";
byte b = Convert.ToByte(hex.Substring(2), 16)
// b is now 0x3B
Substring is used to skip the characters 0x

Joining byte arrays using a separator in C#

I want something similar to String.Join(separator_string, list_of_strings) but for byte arrays.
I need it because I am implementing a file format writer, and the specification says
"Each annotation must be encoded as UTF-8 and separated by the ASCII byte 20."
Then I would need something like:
byte separator = 20;
List<byte[]> encoded_annotations;
joined_byte_array = Something.Join(separator, encoded_annotations);

I don't believe there's anything built in, but it's easy enough to write. Here's a generic version, but you could make it do just byte arrays easily enough.
public static T[] Join<T>(T separator, IEnumerable<T[]> arrays)
{
// Make sure we only iterate over arrays once
List<T[]> list = arrays.ToList();
if (list.Count == 0)
{
return new T[0];
}
int size = list.Sum(x => x.Length) + list.Count - 1;
T[] ret = new T[size];
int index = 0;
bool first = true;
foreach (T[] array in list)
{
if (!first)
{
ret[index++] = separator;
}
Array.Copy(array, 0, ret, index, array.Length);
index += array.Length;
first = false;
}
return ret;
}

I ended up using this, tuned to my specific case of a separator consisting of one single byte (instead of an array of size one), but the general idea would apply to a separator consisting of a byte array:
public byte[] ArrayJoin(byte separator, List<byte[]> arrays)
{
using (MemoryStream result = new MemoryStream())
{
byte[] first = arrays.First();
result.Write(first, 0, first.Length);
foreach (var array in arrays.Skip(1))
{
result.WriteByte(separator);
result.Write(array, 0, array.Length);
}
return result.ToArray();
}
}

You just need to specify the desired type for the join command.
String.join<byte>(separator_string, list_of_strings);

C# - Cast a byte array to an array of struct and vice-versa (reverse)

I would like to save a Color[] to a file. To do so, I found that saving a byte array to a file using "System.IO.File.WriteAllBytes" should be very efficient.
I would like to cast my Color[] (array of struct) to a byte array into a safe way considering:
Potential problem of little endian / big endian (I think it is impossible to happen but want to be sure)
Having 2 differents pointer to the same memory which point to different type. Does the garbage collection will know what to do - moving objects - deleting a pointer ???
If it is possible, it would be nice to have a generic way to cast array of byte to array of any struct (T struct) and vice-versa.
If not possible, why ?
Thanks,
Eric
I think that those 2 solutions make a copy that I would like to avoid and also they both uses Marshal.PtrToStructure which is specific to structure and not to array of structure:
Reading a C/C++ data structure in C# from a byte array
How to convert a structure to a byte array in C#?

Since .NET Core 2.1, yes we can! Enter MemoryMarshal.
We will treat our Color[] as a ReadOnlySpan<Color>. We reinterpret that as a ReadOnlySpan<byte>. Finally, since WriteAllBytes has no span-based overload, we use a FileStream to write the span to disk.
var byteSpan = MemoryMarshal.AsBytes(colorArray.AsSpan());
fileStream.Write(byteSpan);
As an interesting side note, you can also experiment with the [StructLayout(LayoutKind.Explicit)] as an attribute on your fields. It allows you to specify overlapping fields, effectively allowing the concept of a union.
Here is a blog post on MSDN that illustrates this. It shows the following code:
[StructLayout(LayoutKind.Explicit)]
public struct MyUnion
{
[FieldOffset(0)]
public UInt16 myInt;
[FieldOffset(0)]
public Byte byte1;
[FieldOffset(1)]
public Byte byte2;
}
In this example, the UInt16 field overlaps with the two Byte fields.
This seems to be strongly related to what you are trying to do. It gets you very close, except for the part of writing all the bytes (especially of multiple Color objects) efficiently. :)

Regarding Array Type Conversion
C# as a language intentionally makes the process of flattening objects or arrays into byte arrays difficult because this approach goes against the principals of .NET strong typing. The conventional alternatives include several serialization tools which are generally seen a safer and more robust, or manual serialization coding such as BinaryWriter.
Having two variables of different types point to the same object in memory can only be performed if the types of the variables can be cast, implicitly or explicitly. Casting from an array of one element type to another is no trivial task: it would have to convert the internal members that keep track of things such as array length, etc.
A simple way to write and read Color[] to file
void WriteColorsToFile(string path, Color[] colors)
{
BinaryWriter writer = new BinaryWriter(File.OpenWrite(path));
writer.Write(colors.Length);
foreach(Color color in colors)
{
writer.Write(color.ToArgb());
}
writer.Close();
}
Color[] ReadColorsFromFile(string path)
{
BinaryReader reader = new BinaryReader(File.OpenRead(path));
int length = reader.ReadInt32();
Colors[] result = new Colors[length];
for(int n=0; n<length; n++)
{
result[n] = Color.FromArgb(reader.ReadInt32());
}
reader.Close();
}

You could use pointers if you really want to copy each byte and not have a copy but the same object, similar to this:
var structPtr = (byte*)&yourStruct;
var size = sizeof(YourType);
var memory = new byte[size];
fixed(byte* memoryPtr = memory)
{
for(int i = 0; i < size; i++)
{
*(memoryPtr + i) = *structPtr++;
}
}
File.WriteAllBytes(path, memory);
I just tested this and after adding the fixed block and some minor corrections it looks like it is working correctly.
This is what I used to test it:
public static void Main(string[] args)
{
var a = new s { i = 1, j = 2 };
var sPtr = (byte*)&a;
var size = sizeof(s);
var mem = new byte[size];
fixed (byte* memPtr = mem)
{
for (int i = 0; i < size; i++)
{
*(memPtr + i) = *sPtr++;
}
}
File.WriteAllBytes("A:\\file.txt", mem);
}
struct s
{
internal int i;
internal int j;
}
The result is the following:
(I manually resolved the hex bytes in the second line, only the first line was produced by the program)

public struct MyX
{
public int IntValue;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 3, ArraySubType = UnmanagedType.U1)]
public byte[] Array;
MyX(int i, int b)
{
IntValue = b;
Array = new byte[3];
}
public MyX ToStruct(byte []ar)
{
byte[] data = ar;//= { 1, 0, 0, 0, 9, 8, 7 }; // IntValue = 1, Array = {9,8,7}
IntPtr ptPoit = Marshal.AllocHGlobal(data.Length);
Marshal.Copy(data, 0, ptPoit, data.Length);
MyX x = (MyX)Marshal.PtrToStructure(ptPoit, typeof(MyX));
Marshal.FreeHGlobal(ptPoit);
return x;
}
public byte[] ToBytes()
{
Byte[] bytes = new Byte[Marshal.SizeOf(typeof(MyX))];
GCHandle pinStructure = GCHandle.Alloc(this, GCHandleType.Pinned);
try
{
Marshal.Copy(pinStructure.AddrOfPinnedObject(), bytes, 0, bytes.Length);
return bytes;
}
finally
{
pinStructure.Free();
}
}
}
void function()
{
byte[] data = { 1, 0, 0, 0, 9, 8, 7 }; // IntValue = 1, Array = {9,8,7}
IntPtr ptPoit = Marshal.AllocHGlobal(data.Length);
Marshal.Copy(data, 0, ptPoit, data.Length);
var x = (MyX)Marshal.PtrToStructure(ptPoit, typeof(MyX));
Marshal.FreeHGlobal(ptPoit);
var MYstruc = x.ToStruct(data);
Console.WriteLine("x.IntValue = {0}", x.IntValue);
Console.WriteLine("x.Array = ({0}, {1}, {2})", x.Array[0], x.Array[1], x.Array[2]);
}

Working code for reference (take care, I did not need the alpha channel in my case):
// ************************************************************************
// If someday Microsoft make Color serializable ...
//public static void SaveColors(Color[] colors, string path)
//{
// BinaryFormatter bf = new BinaryFormatter();
// MemoryStream ms = new MemoryStream();
// bf.Serialize(ms, colors);
// byte[] bytes = ms.ToArray();
// File.WriteAllBytes(path, bytes);
//}
// If someday Microsoft make Color serializable ...
//public static Colors[] LoadColors(string path)
//{
// Byte[] bytes = File.ReadAllBytes(path);
// BinaryFormatter bf = new BinaryFormatter();
// MemoryStream ms2 = new MemoryStream(bytes);
// return (Colors[])bf.Deserialize(ms2);
//}
// ******************************************************************
public static void SaveColorsToFile(Color[] colors, string path)
{
var formatter = new BinaryFormatter();
int count = colors.Length;
using (var stream = File.OpenWrite(path))
{
formatter.Serialize(stream, count);
for (int index = 0; index < count; index++)
{
formatter.Serialize(stream, colors[index].R);
formatter.Serialize(stream, colors[index].G);
formatter.Serialize(stream, colors[index].B);
}
}
}
// ******************************************************************
public static Color[] LoadColorsFromFile(string path)
{
var formatter = new BinaryFormatter();
Color[] colors;
using (var stream = File.OpenRead(path))
{
int count = (int)formatter.Deserialize(stream);
colors = new Color[count];
for (int index = 0; index < count; index++)
{
byte r = (byte)formatter.Deserialize(stream);
byte g = (byte)formatter.Deserialize(stream);
byte b = (byte)formatter.Deserialize(stream);
colors[index] = Color.FromRgb(r, g, b);
}
}
return colors;
}
// ******************************************************************

Base-N encoding of a byte array

A couple of days ago I came across this CodeReview for Base-36 encoding a byte array. However, the answers that followed didn't touch on decoding back into a byte array, or possibly reusing the answer to perform encodings of different bases (radix).
The answer for the linked question uses BigInteger. So as far as implementation goes, the base and its digits could be parametrized.
The problem with BigInteger though, is that we're treating our input as an assumed integer. However, our input, a byte array, is just an opaque series of values.
If the byte array ends in a series of zero bytes, eg {0xFF,0x7F,0x00,0x00}, those bytes will be lost when using the algorithm in the answer (would only encode {0xFF,0x7F}.
If the last non-zero byte has the sign bit set then the proceeding zero byte is consumed as it's treated as the BigInt's sign delimiter. So {0xFF,0xFF,0x00,0x00} would encode only as {0xFF,0xFF,0x00}.
How could a .NET programmer use BigInteger to create a reasonably efficient and radix-agnostic encoder, with decoding support, plus the ability to handle endian-ness, and with the ability to 'work around' the ending zero bytes being lost?

edit [2020/01/26]: FWIW, the code below along with its unit test live along side my open source libraries on Github.
edit [2016/04/19]: If you're fond of exceptions, you may wish to change some of the Decode implementation code to throw InvalidDataException instead of just returning null.
edit [2014/09/14]: I've added a 'HACK' to Encode() to handle cases where the last byte in the input is signed (if you were to convert to sbyte). Only sane solution I could think of right now is to just Resize() the array by one. Additional unit tests for this case passed, but I didn't rerun perf code to account for such cases. If you can help it, always have your input to Encode() include a dummy 0 byte at the end to avoid additional allocations.
Usage
I've created a RadixEncoding class (found in the "Code" section) which initializes with three parameters:
The radix digits as a string (length determines the actual radix of course),
The assumed byte ordering (endian) of input byte arrays,
And whether or not the user wants the encode/decode logic to acknowledge ending zero bytes.
To create a Base-36 encoding, with little-endian input, and with respect given to ending zero bytes:
const string k_base36_digits = "0123456789abcdefghijklmnopqrstuvwxyz";
var base36_no_zeros = new RadixEncoding(k_base36_digits, EndianFormat.Little, false);
And then to actually perform encoding/decoding:
const string k_input = "A test 1234";
byte[] input_bytes = System.Text.Encoding.UTF8.GetBytes(k_input);
string encoded_string = base36_no_zeros.Encode(input_bytes);
byte[] decoded_bytes = base36_no_zeros.Decode(encoded_string);
Performance
Timed with Diagnostics.Stopwatch, ran on an i7 860 #2.80GHz. Timing EXE ran by itself, not under a debugger.
Encoding was initialized with the same k_base36_digits string from above, EndianFormat.Little, and with ending zero bytes acknowledged (even though the UTF8 bytes don't have any extra ending zero bytes)
To encode the UTF8 bytes of "A test 1234" 1,000,000 times takes 2.6567905secs
To decode the same string the same amount of times takes 3.3916248secs
To encode the UTF8 bytes of "A test 1234. Made slightly larger!" 100,000 times takes 1.1577325secs
To decode the same string the same amount of times takes 1.244326secs
Code
If you don't have a CodeContracts generator, you will have to reimplement the contracts with if/throw code.
using System;
using System.Collections.Generic;
using System.Numerics;
using Contract = System.Diagnostics.Contracts.Contract;
public enum EndianFormat
{
/// <summary>Least Significant Bit order (lsb)</summary>
/// <remarks>Right-to-Left</remarks>
/// <see cref="BitConverter.IsLittleEndian"/>
Little,
/// <summary>Most Significant Bit order (msb)</summary>
/// <remarks>Left-to-Right</remarks>
Big,
};
/// <summary>Encodes/decodes bytes to/from a string</summary>
/// <remarks>
/// Encoded string is always in big-endian ordering
///
/// <p>Encode and Decode take a <b>includeProceedingZeros</b> parameter which acts as a work-around
/// for an edge case with our BigInteger implementation.
/// MSDN says BigInteger byte arrays are in LSB->MSB ordering. So a byte buffer with zeros at the
/// end will have those zeros ignored in the resulting encoded radix string.
/// If such a loss in precision absolutely cannot occur pass true to <b>includeProceedingZeros</b>
/// and for a tiny bit of extra processing it will handle the padding of zero digits (encoding)
/// or bytes (decoding).</p>
/// <p>Note: doing this for decoding <b>may</b> add an extra byte more than what was originally
/// given to Encode.</p>
/// </remarks>
// Based on the answers from http://codereview.stackexchange.com/questions/14084/base-36-encoding-of-a-byte-array/
public class RadixEncoding
{
const int kByteBitCount = 8;
readonly string kDigits;
readonly double kBitsPerDigit;
readonly BigInteger kRadixBig;
readonly EndianFormat kEndian;
readonly bool kIncludeProceedingZeros;
/// <summary>Numerial base of this encoding</summary>
public int Radix { get { return kDigits.Length; } }
/// <summary>Endian ordering of bytes input to Encode and output by Decode</summary>
public EndianFormat Endian { get { return kEndian; } }
/// <summary>True if we want ending zero bytes to be encoded</summary>
public bool IncludeProceedingZeros { get { return kIncludeProceedingZeros; } }
public override string ToString()
{
return string.Format("Base-{0} {1}", Radix.ToString(), kDigits);
}
/// <summary>Create a radix encoder using the given characters as the digits in the radix</summary>
/// <param name="digits">Digits to use for the radix-encoded string</param>
/// <param name="bytesEndian">Endian ordering of bytes input to Encode and output by Decode</param>
/// <param name="includeProceedingZeros">True if we want ending zero bytes to be encoded</param>
public RadixEncoding(string digits,
EndianFormat bytesEndian = EndianFormat.Little, bool includeProceedingZeros = false)
{
Contract.Requires<ArgumentNullException>(digits != null);
int radix = digits.Length;
kDigits = digits;
kBitsPerDigit = System.Math.Log(radix, 2);
kRadixBig = new BigInteger(radix);
kEndian = bytesEndian;
kIncludeProceedingZeros = includeProceedingZeros;
}
// Number of characters needed for encoding the specified number of bytes
int EncodingCharsCount(int bytesLength)
{
return (int)Math.Ceiling((bytesLength * kByteBitCount) / kBitsPerDigit);
}
// Number of bytes needed to decoding the specified number of characters
int DecodingBytesCount(int charsCount)
{
return (int)Math.Ceiling((charsCount * kBitsPerDigit) / kByteBitCount);
}
/// <summary>Encode a byte array into a radix-encoded string</summary>
/// <param name="bytes">byte array to encode</param>
/// <returns>The bytes in encoded into a radix-encoded string</returns>
/// <remarks>If <paramref name="bytes"/> is zero length, returns an empty string</remarks>
public string Encode(byte[] bytes)
{
Contract.Requires<ArgumentNullException>(bytes != null);
Contract.Ensures(Contract.Result<string>() != null);
// Don't really have to do this, our code will build this result (empty string),
// but why not catch the condition before doing work?
if (bytes.Length == 0) return string.Empty;
// if the array ends with zeros, having the capacity set to this will help us know how much
// 'padding' we will need to add
int result_length = EncodingCharsCount(bytes.Length);
// List<> has a(n in-place) Reverse method. StringBuilder doesn't. That's why.
var result = new List<char>(result_length);
// HACK: BigInteger uses the last byte as the 'sign' byte. If that byte's MSB is set,
// we need to pad the input with an extra 0 (ie, make it positive)
if ( (bytes[bytes.Length-1] & 0x80) == 0x80 )
Array.Resize(ref bytes, bytes.Length+1);
var dividend = new BigInteger(bytes);
// IsZero's computation is less complex than evaluating "dividend > 0"
// which invokes BigInteger.CompareTo(BigInteger)
while (!dividend.IsZero)
{
BigInteger remainder;
dividend = BigInteger.DivRem(dividend, kRadixBig, out remainder);
int digit_index = System.Math.Abs((int)remainder);
result.Add(kDigits[digit_index]);
}
if (kIncludeProceedingZeros)
for (int x = result.Count; x < result.Capacity; x++)
result.Add(kDigits[0]); // pad with the character that represents 'zero'
// orientate the characters in big-endian ordering
if (kEndian == EndianFormat.Little)
result.Reverse();
// If we didn't end up adding padding, ToArray will end up returning a TrimExcess'd array,
// so nothing wasted
return new string(result.ToArray());
}
void DecodeImplPadResult(ref byte[] result, int padCount)
{
if (padCount > 0)
{
int new_length = result.Length + DecodingBytesCount(padCount);
Array.Resize(ref result, new_length); // new bytes will be zero, just the way we want it
}
}
#region Decode (Little Endian)
byte[] DecodeImpl(string chars, int startIndex = 0)
{
var bi = new BigInteger();
for (int x = startIndex; x < chars.Length; x++)
{
int i = kDigits.IndexOf(chars[x]);
if (i < 0) return null; // invalid character
bi *= kRadixBig;
bi += i;
}
return bi.ToByteArray();
}
byte[] DecodeImplWithPadding(string chars)
{
int pad_count = 0;
for (int x = 0; x < chars.Length; x++, pad_count++)
if (chars[x] != kDigits[0]) break;
var result = DecodeImpl(chars, pad_count);
DecodeImplPadResult(ref result, pad_count);
return result;
}
#endregion
#region Decode (Big Endian)
byte[] DecodeImplReversed(string chars, int startIndex = 0)
{
var bi = new BigInteger();
for (int x = (chars.Length-1)-startIndex; x >= 0; x--)
{
int i = kDigits.IndexOf(chars[x]);
if (i < 0) return null; // invalid character
bi *= kRadixBig;
bi += i;
}
return bi.ToByteArray();
}
byte[] DecodeImplReversedWithPadding(string chars)
{
int pad_count = 0;
for (int x = chars.Length - 1; x >= 0; x--, pad_count++)
if (chars[x] != kDigits[0]) break;
var result = DecodeImplReversed(chars, pad_count);
DecodeImplPadResult(ref result, pad_count);
return result;
}
#endregion
/// <summary>Decode a radix-encoded string into a byte array</summary>
/// <param name="radixChars">radix string</param>
/// <returns>The decoded bytes, or null if an invalid character is encountered</returns>
/// <remarks>
/// If <paramref name="radixChars"/> is an empty string, returns a zero length array
///
/// Using <paramref name="IncludeProceedingZeros"/> has the potential to return a buffer with an
/// additional zero byte that wasn't in the input. So a 4 byte buffer was encoded, this could end up
/// returning a 5 byte buffer, with the extra byte being null.
/// </remarks>
public byte[] Decode(string radixChars)
{
Contract.Requires<ArgumentNullException>(radixChars != null);
if (kEndian == EndianFormat.Big)
return kIncludeProceedingZeros ? DecodeImplReversedWithPadding(radixChars) : DecodeImplReversed(radixChars);
else
return kIncludeProceedingZeros ? DecodeImplWithPadding(radixChars) : DecodeImpl(radixChars);
}
};
Basic Unit Tests
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
static bool ArraysCompareN<T>(T[] input, T[] output)
where T : IEquatable<T>
{
if (output.Length < input.Length) return false;
for (int x = 0; x < input.Length; x++)
if(!output[x].Equals(input[x])) return false;
return true;
}
static bool RadixEncodingTest(RadixEncoding encoding, byte[] bytes)
{
string encoded = encoding.Encode(bytes);
byte[] decoded = encoding.Decode(encoded);
return ArraysCompareN(bytes, decoded);
}
[TestMethod]
public void TestRadixEncoding()
{
const string k_base36_digits = "0123456789abcdefghijklmnopqrstuvwxyz";
var base36 = new RadixEncoding(k_base36_digits, EndianFormat.Little, true);
var base36_no_zeros = new RadixEncoding(k_base36_digits, EndianFormat.Little, true);
byte[] ends_with_zero_neg = { 0xFF, 0xFF, 0x00, 0x00 };
byte[] ends_with_zero_pos = { 0xFF, 0x7F, 0x00, 0x00 };
byte[] text = System.Text.Encoding.ASCII.GetBytes("A test 1234");
Assert.IsTrue(RadixEncodingTest(base36, ends_with_zero_neg));
Assert.IsTrue(RadixEncodingTest(base36, ends_with_zero_pos));
Assert.IsTrue(RadixEncodingTest(base36_no_zeros, text));
}

Interestingly, I was able to port Kornman's techniques across to Java and got expected output up to and including base36. Whereas when running his? code from c# using C:\Windows\Microsoft.NET\Framework\v4.0.30319 csc, the output was not as expected.
For example, trying to base16 encode the obtained MD5 hashBytes for the String "hello world" below using Kornman's RadixEncoding encode, I could see the groups of two bytes per characters had the bytes in wrong order.
Rather than 5eb63bbbe01eeed093cb22bb8f5acdc3
I saw something like e56bb3bb0ee1....
This was on Windows 7.
const string input = "hello world";
public static void Main(string[] args)
{
using (System.Security.Cryptography.MD5 md5 = System.Security.Cryptography.MD5.Create())
{
byte[] inputBytes = System.Text.Encoding.ASCII.GetBytes(input);
byte[] hashBytes = md5.ComputeHash(inputBytes);
// Convert the byte array to hexadecimal string
StringBuilder sb = new StringBuilder();
for (int i = 0; i < hashBytes.Length; i++)
{
sb.Append(hashBytes[i].ToString("X2"));
}
Console.WriteLine(sb.ToString());
}
}
Java code is below for anyone interested. As mentioned above, it only works to base 36.
private static final char[] BASE16_CHARS = "0123456789abcdef".toCharArray();
private static final BigInteger BIGINT_16 = BigInteger.valueOf(16);
private static final char[] BASE36_CHARS = "0123456789abcdefghijklmnopqrstuvwxyz".toCharArray();
private static final BigInteger BIGINT_36 = BigInteger.valueOf(36);
public static String toBaseX(byte[] bytes, BigInteger base, char[] chars)
{
if (bytes == null) {
return null;
}
final int bitsPerByte = 8;
double bitsPerDigit = Math.log(chars.length) / Math.log(2);
// Number of chars to encode specified bytes
int size = (int) Math.ceil((bytes.length * bitsPerByte) / bitsPerDigit);
StringBuilder sb = new StringBuilder(size);
for (BigInteger value = new BigInteger(bytes); !value.equals(BigInteger.ZERO);) {
BigInteger[] quotientAndRemainder = value.divideAndRemainder(base);
sb.insert(0, chars[Math.abs(quotientAndRemainder[1].intValue())]);
value = quotientAndRemainder[0];
}
return sb.toString();
}

Best way to combine two or more byte arrays in C#

I have 3 byte arrays in C# that I need to combine into one. What would be the most efficient method to complete this task?

For primitive types (including bytes), use System.Buffer.BlockCopy instead of System.Array.Copy. It's faster.
I timed each of the suggested methods in a loop executed 1 million times using 3 arrays of 10 bytes each. Here are the results:
New Byte Array using System.Array.Copy - 0.2187556 seconds
New Byte Array using System.Buffer.BlockCopy - 0.1406286 seconds
IEnumerable<byte> using C# yield operator - 0.0781270 seconds
IEnumerable<byte> using LINQ's Concat<> - 0.0781270 seconds
I increased the size of each array to 100 elements and re-ran the test:
New Byte Array using System.Array.Copy - 0.2812554 seconds
New Byte Array using System.Buffer.BlockCopy - 0.2500048 seconds
IEnumerable<byte> using C# yield operator - 0.0625012 seconds
IEnumerable<byte> using LINQ's Concat<> - 0.0781265 seconds
I increased the size of each array to 1000 elements and re-ran the test:
New Byte Array using System.Array.Copy - 1.0781457 seconds
New Byte Array using System.Buffer.BlockCopy - 1.0156445 seconds
IEnumerable<byte> using C# yield operator - 0.0625012 seconds
IEnumerable<byte> using LINQ's Concat<> - 0.0781265 seconds
Finally, I increased the size of each array to 1 million elements and re-ran the test, executing each loop only 4000 times:
New Byte Array using System.Array.Copy - 13.4533833 seconds
New Byte Array using System.Buffer.BlockCopy - 13.1096267 seconds
IEnumerable<byte> using C# yield operator - 0 seconds
IEnumerable<byte> using LINQ's Concat<> - 0 seconds
So, if you need a new byte array, use
byte[] rv = new byte[a1.Length + a2.Length + a3.Length];
System.Buffer.BlockCopy(a1, 0, rv, 0, a1.Length);
System.Buffer.BlockCopy(a2, 0, rv, a1.Length, a2.Length);
System.Buffer.BlockCopy(a3, 0, rv, a1.Length + a2.Length, a3.Length);
But, if you can use an IEnumerable<byte>, DEFINITELY prefer LINQ's Concat<> method. It's only slightly slower than the C# yield operator, but is more concise and more elegant.
IEnumerable<byte> rv = a1.Concat(a2).Concat(a3);
If you have an arbitrary number of arrays and are using .NET 3.5, you can make the System.Buffer.BlockCopy solution more generic like this:
private byte[] Combine(params byte[][] arrays)
{
byte[] rv = new byte[arrays.Sum(a => a.Length)];
int offset = 0;
foreach (byte[] array in arrays) {
System.Buffer.BlockCopy(array, 0, rv, offset, array.Length);
offset += array.Length;
}
return rv;
}
*Note: The above block requires you adding the following namespace at the the top for it to work.
using System.Linq;
To Jon Skeet's point regarding iteration of the subsequent data structures (byte array vs. IEnumerable<byte>), I re-ran the last timing test (1 million elements, 4000 iterations), adding a loop that iterates over the full array with each pass:
New Byte Array using System.Array.Copy - 78.20550510 seconds
New Byte Array using System.Buffer.BlockCopy - 77.89261900 seconds
IEnumerable<byte> using C# yield operator - 551.7150161 seconds
IEnumerable<byte> using LINQ's Concat<> - 448.1804799 seconds
The point is, it is VERY important to understand the efficiency of both the creation and the usage of the resulting data structure. Simply focusing on the efficiency of the creation may overlook the inefficiency associated with the usage. Kudos, Jon.

Many of the answers seem to me to be ignoring the stated requirements:
The result should be a byte array
It should be as efficient as possible
These two together rule out a LINQ sequence of bytes - anything with yield is going to make it impossible to get the final size without iterating through the whole sequence.
If those aren't the real requirements of course, LINQ could be a perfectly good solution (or the IList<T> implementation). However, I'll assume that Superdumbell knows what he wants.
(EDIT: I've just had another thought. There's a big semantic difference between making a copy of the arrays and reading them lazily. Consider what happens if you change the data in one of the "source" arrays after calling the Combine (or whatever) method but before using the result - with lazy evaluation, that change will be visible. With an immediate copy, it won't. Different situations will call for different behaviour - just something to be aware of.)
Here are my proposed methods - which are very similar to those contained in some of the other answers, certainly :)
public static byte[] Combine(byte[] first, byte[] second)
{
byte[] ret = new byte[first.Length + second.Length];
Buffer.BlockCopy(first, 0, ret, 0, first.Length);
Buffer.BlockCopy(second, 0, ret, first.Length, second.Length);
return ret;
}
public static byte[] Combine(byte[] first, byte[] second, byte[] third)
{
byte[] ret = new byte[first.Length + second.Length + third.Length];
Buffer.BlockCopy(first, 0, ret, 0, first.Length);
Buffer.BlockCopy(second, 0, ret, first.Length, second.Length);
Buffer.BlockCopy(third, 0, ret, first.Length + second.Length,
third.Length);
return ret;
}
public static byte[] Combine(params byte[][] arrays)
{
byte[] ret = new byte[arrays.Sum(x => x.Length)];
int offset = 0;
foreach (byte[] data in arrays)
{
Buffer.BlockCopy(data, 0, ret, offset, data.Length);
offset += data.Length;
}
return ret;
}
Of course the "params" version requires creating an array of the byte arrays first, which introduces extra inefficiency.

I took Matt's LINQ example one step further for code cleanliness:
byte[] rv = a1.Concat(a2).Concat(a3).ToArray();
In my case, the arrays are small, so I'm not concerned about performance.

If you simply need a new byte array, then use the following:
byte[] Combine(byte[] a1, byte[] a2, byte[] a3)
{
byte[] ret = new byte[a1.Length + a2.Length + a3.Length];
Array.Copy(a1, 0, ret, 0, a1.Length);
Array.Copy(a2, 0, ret, a1.Length, a2.Length);
Array.Copy(a3, 0, ret, a1.Length + a2.Length, a3.Length);
return ret;
}
Alternatively, if you just need a single IEnumerable, consider using the C# 2.0 yield operator:
IEnumerable<byte> Combine(byte[] a1, byte[] a2, byte[] a3)
{
foreach (byte b in a1)
yield return b;
foreach (byte b in a2)
yield return b;
foreach (byte b in a3)
yield return b;
}

I actually ran into some issues with using Concat... (with arrays in the 10-million, it actually crashed).
I found the following to be simple, easy and works well enough without crashing on me, and it works for ANY number of arrays (not just three) (It uses LINQ):
public static byte[] ConcatByteArrays(params byte[][] arrays)
{
return arrays.SelectMany(x => x).ToArray();
}

The memorystream class does this job pretty nicely for me. I couldn't get the buffer class to run as fast as memorystream.
using (MemoryStream ms = new MemoryStream())
{
ms.Write(BitConverter.GetBytes(22),0,4);
ms.Write(BitConverter.GetBytes(44),0,4);
ms.ToArray();
}

public static byte[] Concat(params byte[][] arrays) {
using (var mem = new MemoryStream(arrays.Sum(a => a.Length))) {
foreach (var array in arrays) {
mem.Write(array, 0, array.Length);
}
return mem.ToArray();
}
}

public static bool MyConcat<T>(ref T[] base_arr, ref T[] add_arr)
{
try
{
int base_size = base_arr.Length;
int size_T = System.Runtime.InteropServices.Marshal.SizeOf(base_arr[0]);
Array.Resize(ref base_arr, base_size + add_arr.Length);
Buffer.BlockCopy(add_arr, 0, base_arr, base_size * size_T, add_arr.Length * size_T);
}
catch (IndexOutOfRangeException ioor)
{
MessageBox.Show(ioor.Message);
return false;
}
return true;
}

Can use generics to combine arrays. Following code can easily be expanded to three arrays. This way you never need to duplicate code for different type of arrays. Some of the above answers seem overly complex to me.
private static T[] CombineTwoArrays<T>(T[] a1, T[] a2)
{
T[] arrayCombined = new T[a1.Length + a2.Length];
Array.Copy(a1, 0, arrayCombined, 0, a1.Length);
Array.Copy(a2, 0, arrayCombined, a1.Length, a2.Length);
return arrayCombined;
}

/// <summary>
/// Combine two Arrays with offset and count
/// </summary>
/// <param name="src1"></param>
/// <param name="offset1"></param>
/// <param name="count1"></param>
/// <param name="src2"></param>
/// <param name="offset2"></param>
/// <param name="count2"></param>
/// <returns></returns>
public static T[] Combine<T>(this T[] src1, int offset1, int count1, T[] src2, int offset2, int count2)
=> Enumerable.Range(0, count1 + count2).Select(a => (a < count1) ? src1[offset1 + a] : src2[offset2 + a - count1]).ToArray();

Here's a generalization of the answer provided by #Jon Skeet.
It is basically the same, only it is usable for any type of array, not only bytes:
public static T[] Combine<T>(T[] first, T[] second)
{
T[] ret = new T[first.Length + second.Length];
Buffer.BlockCopy(first, 0, ret, 0, first.Length);
Buffer.BlockCopy(second, 0, ret, first.Length, second.Length);
return ret;
}
public static T[] Combine<T>(T[] first, T[] second, T[] third)
{
T[] ret = new T[first.Length + second.Length + third.Length];
Buffer.BlockCopy(first, 0, ret, 0, first.Length);
Buffer.BlockCopy(second, 0, ret, first.Length, second.Length);
Buffer.BlockCopy(third, 0, ret, first.Length + second.Length,
third.Length);
return ret;
}
public static T[] Combine<T>(params T[][] arrays)
{
T[] ret = new T[arrays.Sum(x => x.Length)];
int offset = 0;
foreach (T[] data in arrays)
{
Buffer.BlockCopy(data, 0, ret, offset, data.Length);
offset += data.Length;
}
return ret;
}

All you need to pass list of Byte Arrays and this function will return you the Array of Bytes (Merged).
This is the best solution i think :).
public static byte[] CombineMultipleByteArrays(List<byte[]> lstByteArray)
{
using (var ms = new MemoryStream())
{
using (var doc = new iTextSharp.text.Document())
{
using (var copy = new PdfSmartCopy(doc, ms))
{
doc.Open();
foreach (var p in lstByteArray)
{
using (var reader = new PdfReader(p))
{
copy.AddDocument(reader);
}
}
doc.Close();
}
}
return ms.ToArray();
}
}

Concat is the right answer, but for some reason a handrolled thing is getting the most votes. If you like that answer, perhaps you'd like this more general solution even more:
IEnumerable<byte> Combine(params byte[][] arrays)
{
foreach (byte[] a in arrays)
foreach (byte b in a)
yield return b;
}
which would let you do things like:
byte[] c = Combine(new byte[] { 0, 1, 2 }, new byte[] { 3, 4, 5 }).ToArray();