add variable to hex stream - c#

in c# i want to add variable from trackbar.value to hex stream:
UdpClient udpClient = new UdpClient("192.168.1.205", 52381);=
Byte[] sendBytes = new byte[] {
0x01, 0x00, 0x00, 0x06, 0x00, 0x00, 0x00, 0xff, 0x81, 0x01,
0x04, 0x07, 0x0("trackbar.value"), 0xFF
};
try
{
udpClient.Send(sendBytes, sendBytes.Length);
}
catch (Exception)
{
Console.WriteLine(e.ToString());
}
ho do i syntax that?

If you have a fixed size byte array you can do this:
Byte[] sendBytes = new byte[] {
0x01, 0x00, 0x00, 0x06, 0x00, 0x00, 0x00, 0xff, 0x81, 0x01,
0x04, 0x07, 0x00, 0xFF
};
sendBytes[12] = (byte) trackbar.value;
BUT: This will of course lead to data loss if your trackbar.value is not a byte value and exceeds a maximum value of 255.
So you will either have to reduce precision and scale your value, something like this:
byte scaledValue = (byte) ((double) trackbar.value / [maximum_trackbar_value] * 255.0);
or increase the amount of bytes used in your message to transfer the value.
For example using a 16bit integer with a maximum value of 65535 (= 2^16 - 1) will require 2 bytes. To "split" your value into these 2 bytes you will have to do a bit shift like this
1. sendBytes[12] = (byte) trackbar.value;
2. sendBytes[13] = (byte) (trackbar.value >> 8);
The second line will shift the highest 8 bits of the value to the right so they can be put into the next byte, which can only be used for 8 bits, the rest will be cut off. For better understanding:
1. (byte) (00000101 00111001) = 00111001
2. (byte) ((00000101 00111001) >> 8) = (byte) (00000000 00000101) = 00000101
This will transfer your value in little endian order, meaning the less significant byte ("smaller value") first in your array. You will find more details on that topic here: https://en.wikipedia.org/wiki/Endianness. The receiver of your message will have to do the reverse:
int trackbarValue = (receivedBytes[12] & 0xFF) | ((receivedBytes[13] & 0xFF) << 8);
Here are two functions to make this a little more general:
public static byte[] GetBytesLittleEndian(int num, int byteCount)
{
var result = new byte[byteCount];
for (int i = 0; i < byteCount; i++)
{
result[i] = (byte)(num >> (i * 8));
}
return result;
}
public static int ParseLittleEndian(byte[] data, int byteCount)
{
int parsed = 0;
for (int i = 0; i < byteCount; i++)
{
parsed |= data[i] << (i * 8);
}
return parsed;
}
Of course there is build-in functionality for that, see BitConverter.GetBytes() for example, but I think it won't hurt to know some details ;)

Related

Trying to convert byte string to byte type in C# [duplicate]

How can you convert a byte array to a hexadecimal string and vice versa?
You can use Convert.ToHexString starting with .NET 5.
There's also a method for the reverse operation: Convert.FromHexString.
For older versions of .NET you can either use:
public static string ByteArrayToString(byte[] ba)
{
StringBuilder hex = new StringBuilder(ba.Length * 2);
foreach (byte b in ba)
hex.AppendFormat("{0:x2}", b);
return hex.ToString();
}
or:
public static string ByteArrayToString(byte[] ba)
{
return BitConverter.ToString(ba).Replace("-","");
}
There are even more variants of doing it, for example here.
The reverse conversion would go like this:
public static byte[] StringToByteArray(String hex)
{
int NumberChars = hex.Length;
byte[] bytes = new byte[NumberChars / 2];
for (int i = 0; i < NumberChars; i += 2)
bytes[i / 2] = Convert.ToByte(hex.Substring(i, 2), 16);
return bytes;
}
Using Substring is the best option in combination with Convert.ToByte. See this answer for more information. If you need better performance, you must avoid Convert.ToByte before you can drop SubString.
Performance Analysis
Note: new leader as of 2015-08-20.
I ran each of the various conversion methods through some crude Stopwatch performance testing, a run with a random sentence (n=61, 1000 iterations) and a run with a Project Gutenburg text (n=1,238,957, 150 iterations). Here are the results, roughly from fastest to slowest. All measurements are in ticks (10,000 ticks = 1 ms) and all relative notes are compared to the [slowest] StringBuilder implementation. For the code used, see below or the test framework repo where I now maintain the code for running this.
Disclaimer
WARNING: Do not rely on these stats for anything concrete; they are simply a sample run of sample data. If you really need top-notch performance, please test these methods in an environment representative of your production needs with data representative of what you will use.
Results
Lookup by byte unsafe (via CodesInChaos) (added to test repo by airbreather)
Text: 4,727.85 (105.2X)
Sentence: 0.28 (99.7X)
Lookup by byte (via CodesInChaos)
Text: 10,853.96 (45.8X faster)
Sentence: 0.65 (42.7X faster)
Byte Manipulation 2 (via CodesInChaos)
Text: 12,967.69 (38.4X faster)
Sentence: 0.73 (37.9X faster)
Byte Manipulation (via Waleed Eissa)
Text: 16,856.64 (29.5X faster)
Sentence: 0.70 (39.5X faster)
Lookup/Shift (via Nathan Moinvaziri)
Text: 23,201.23 (21.4X faster)
Sentence: 1.24 (22.3X faster)
Lookup by nibble (via Brian Lambert)
Text: 23,879.41 (20.8X faster)
Sentence: 1.15 (23.9X faster)
BitConverter (via Tomalak)
Text: 113,269.34 (4.4X faster)
Sentence: 9.98 (2.8X faster)
{SoapHexBinary}.ToString (via Mykroft)
Text: 178,601.39 (2.8X faster)
Sentence: 10.68 (2.6X faster)
{byte}.ToString("X2") (using foreach) (derived from Will Dean's answer)
Text: 308,805.38 (2.4X faster)
Sentence: 16.89 (2.4X faster)
{byte}.ToString("X2") (using {IEnumerable}.Aggregate, requires System.Linq) (via Mark)
Text: 352,828.20 (2.1X faster)
Sentence: 16.87 (2.4X faster)
Array.ConvertAll (using string.Join) (via Will Dean)
Text: 675,451.57 (1.1X faster)
Sentence: 17.95 (2.2X faster)
Array.ConvertAll (using string.Concat, requires .NET 4.0) (via Will Dean)
Text: 752,078.70 (1.0X faster)
Sentence: 18.28 (2.2X faster)
{StringBuilder}.AppendFormat (using foreach) (via Tomalak)
Text: 672,115.77 (1.1X faster)
Sentence: 36.82 (1.1X faster)
{StringBuilder}.AppendFormat (using {IEnumerable}.Aggregate, requires System.Linq) (derived from Tomalak's answer)
Text: 718,380.63 (1.0X faster)
Sentence: 39.71 (1.0X faster)
Lookup tables have taken the lead over byte manipulation. Basically, there is some form of precomputing what any given nibble or byte will be in hex. Then, as you rip through the data, you simply look up the next portion to see what hex string it would be. That value is then added to the resulting string output in some fashion. For a long time byte manipulation, potentially harder to read by some developers, was the top-performing approach.
Your best bet is still going to be finding some representative data and trying it out in a production-like environment. If you have different memory constraints, you may prefer a method with fewer allocations to one that would be faster but consume more memory.
Testing Code
Feel free to play with the testing code I used. A version is included here but feel free to clone the repo and add your own methods. Please submit a pull request if you find anything interesting or want to help improve the testing framework it uses.
Add the new static method (Func<byte[], string>) to /Tests/ConvertByteArrayToHexString/Test.cs.
Add that method's name to the TestCandidates return value in that same class.
Make sure you are running the input version you want, sentence or text, by toggling the comments in GenerateTestInput in that same class.
Hit F5 and wait for the output (an HTML dump is also generated in the /bin folder).
static string ByteArrayToHexStringViaStringJoinArrayConvertAll(byte[] bytes) {
return string.Join(string.Empty, Array.ConvertAll(bytes, b => b.ToString("X2")));
}
static string ByteArrayToHexStringViaStringConcatArrayConvertAll(byte[] bytes) {
return string.Concat(Array.ConvertAll(bytes, b => b.ToString("X2")));
}
static string ByteArrayToHexStringViaBitConverter(byte[] bytes) {
string hex = BitConverter.ToString(bytes);
return hex.Replace("-", "");
}
static string ByteArrayToHexStringViaStringBuilderAggregateByteToString(byte[] bytes) {
return bytes.Aggregate(new StringBuilder(bytes.Length * 2), (sb, b) => sb.Append(b.ToString("X2"))).ToString();
}
static string ByteArrayToHexStringViaStringBuilderForEachByteToString(byte[] bytes) {
StringBuilder hex = new StringBuilder(bytes.Length * 2);
foreach (byte b in bytes)
hex.Append(b.ToString("X2"));
return hex.ToString();
}
static string ByteArrayToHexStringViaStringBuilderAggregateAppendFormat(byte[] bytes) {
return bytes.Aggregate(new StringBuilder(bytes.Length * 2), (sb, b) => sb.AppendFormat("{0:X2}", b)).ToString();
}
static string ByteArrayToHexStringViaStringBuilderForEachAppendFormat(byte[] bytes) {
StringBuilder hex = new StringBuilder(bytes.Length * 2);
foreach (byte b in bytes)
hex.AppendFormat("{0:X2}", b);
return hex.ToString();
}
static string ByteArrayToHexViaByteManipulation(byte[] bytes) {
char[] c = new char[bytes.Length * 2];
byte b;
for (int i = 0; i < bytes.Length; i++) {
b = ((byte)(bytes[i] >> 4));
c[i * 2] = (char)(b > 9 ? b + 0x37 : b + 0x30);
b = ((byte)(bytes[i] & 0xF));
c[i * 2 + 1] = (char)(b > 9 ? b + 0x37 : b + 0x30);
}
return new string(c);
}
static string ByteArrayToHexViaByteManipulation2(byte[] bytes) {
char[] c = new char[bytes.Length * 2];
int b;
for (int i = 0; i < bytes.Length; i++) {
b = bytes[i] >> 4;
c[i * 2] = (char)(55 + b + (((b - 10) >> 31) & -7));
b = bytes[i] & 0xF;
c[i * 2 + 1] = (char)(55 + b + (((b - 10) >> 31) & -7));
}
return new string(c);
}
static string ByteArrayToHexViaSoapHexBinary(byte[] bytes) {
SoapHexBinary soapHexBinary = new SoapHexBinary(bytes);
return soapHexBinary.ToString();
}
static string ByteArrayToHexViaLookupAndShift(byte[] bytes) {
StringBuilder result = new StringBuilder(bytes.Length * 2);
string hexAlphabet = "0123456789ABCDEF";
foreach (byte b in bytes) {
result.Append(hexAlphabet[(int)(b >> 4)]);
result.Append(hexAlphabet[(int)(b & 0xF)]);
}
return result.ToString();
}
static readonly uint* _lookup32UnsafeP = (uint*)GCHandle.Alloc(_Lookup32, GCHandleType.Pinned).AddrOfPinnedObject();
static string ByteArrayToHexViaLookup32UnsafeDirect(byte[] bytes) {
var lookupP = _lookup32UnsafeP;
var result = new string((char)0, bytes.Length * 2);
fixed (byte* bytesP = bytes)
fixed (char* resultP = result) {
uint* resultP2 = (uint*)resultP;
for (int i = 0; i < bytes.Length; i++) {
resultP2[i] = lookupP[bytesP[i]];
}
}
return result;
}
static uint[] _Lookup32 = Enumerable.Range(0, 255).Select(i => {
string s = i.ToString("X2");
return ((uint)s[0]) + ((uint)s[1] << 16);
}).ToArray();
static string ByteArrayToHexViaLookupPerByte(byte[] bytes) {
var result = new char[bytes.Length * 2];
for (int i = 0; i < bytes.Length; i++)
{
var val = _Lookup32[bytes[i]];
result[2*i] = (char)val;
result[2*i + 1] = (char) (val >> 16);
}
return new string(result);
}
static string ByteArrayToHexViaLookup(byte[] bytes) {
string[] hexStringTable = new string[] {
"00", "01", "02", "03", "04", "05", "06", "07", "08", "09", "0A", "0B", "0C", "0D", "0E", "0F",
"10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "1A", "1B", "1C", "1D", "1E", "1F",
"20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "2A", "2B", "2C", "2D", "2E", "2F",
"30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "3A", "3B", "3C", "3D", "3E", "3F",
"40", "41", "42", "43", "44", "45", "46", "47", "48", "49", "4A", "4B", "4C", "4D", "4E", "4F",
"50", "51", "52", "53", "54", "55", "56", "57", "58", "59", "5A", "5B", "5C", "5D", "5E", "5F",
"60", "61", "62", "63", "64", "65", "66", "67", "68", "69", "6A", "6B", "6C", "6D", "6E", "6F",
"70", "71", "72", "73", "74", "75", "76", "77", "78", "79", "7A", "7B", "7C", "7D", "7E", "7F",
"80", "81", "82", "83", "84", "85", "86", "87", "88", "89", "8A", "8B", "8C", "8D", "8E", "8F",
"90", "91", "92", "93", "94", "95", "96", "97", "98", "99", "9A", "9B", "9C", "9D", "9E", "9F",
"A0", "A1", "A2", "A3", "A4", "A5", "A6", "A7", "A8", "A9", "AA", "AB", "AC", "AD", "AE", "AF",
"B0", "B1", "B2", "B3", "B4", "B5", "B6", "B7", "B8", "B9", "BA", "BB", "BC", "BD", "BE", "BF",
"C0", "C1", "C2", "C3", "C4", "C5", "C6", "C7", "C8", "C9", "CA", "CB", "CC", "CD", "CE", "CF",
"D0", "D1", "D2", "D3", "D4", "D5", "D6", "D7", "D8", "D9", "DA", "DB", "DC", "DD", "DE", "DF",
"E0", "E1", "E2", "E3", "E4", "E5", "E6", "E7", "E8", "E9", "EA", "EB", "EC", "ED", "EE", "EF",
"F0", "F1", "F2", "F3", "F4", "F5", "F6", "F7", "F8", "F9", "FA", "FB", "FC", "FD", "FE", "FF",
};
StringBuilder result = new StringBuilder(bytes.Length * 2);
foreach (byte b in bytes) {
result.Append(hexStringTable[b]);
}
return result.ToString();
}
Update (2010-01-13)
Added Waleed's answer to analysis. Quite fast.
Update (2011-10-05)
Added string.Concat Array.ConvertAll variant for completeness (requires .NET 4.0). On par with string.Join version.
Update (2012-02-05)
Test repo includes more variants such as StringBuilder.Append(b.ToString("X2")). None upset the results any. foreach is faster than {IEnumerable}.Aggregate, for instance, but BitConverter still wins.
Update (2012-04-03)
Added Mykroft's SoapHexBinary answer to analysis, which took over third place.
Update (2013-01-15)
Added CodesInChaos's byte manipulation answer, which took over first place (by a large margin on large blocks of text).
Update (2013-05-23)
Added Nathan Moinvaziri's lookup answer and the variant from Brian Lambert's blog. Both rather fast, but not taking the lead on the test machine I used (AMD Phenom 9750).
Update (2014-07-31)
Added #CodesInChaos's new byte-based lookup answer. It appears to have taken the lead on both the sentence tests and the full-text tests.
Update (2015-08-20)
Added airbreather's optimizations and unsafe variant to this answer's repo. If you want to play in the unsafe game, you can get some huge performance gains over any of the prior top winners on both short strings and large texts.
There's a class called SoapHexBinary that does exactly what you want.
using System.Runtime.Remoting.Metadata.W3cXsd2001;
public static byte[] GetStringToBytes(string value)
{
SoapHexBinary shb = SoapHexBinary.Parse(value);
return shb.Value;
}
public static string GetBytesToString(byte[] value)
{
SoapHexBinary shb = new SoapHexBinary(value);
return shb.ToString();
}
When writing crypto code it's common to avoid data dependent branches and table lookups to ensure the runtime doesn't depend on the data, since data dependent timing can lead to side-channel attacks.
It's also pretty fast.
static string ByteToHexBitFiddle(byte[] bytes)
{
char[] c = new char[bytes.Length * 2];
int b;
for (int i = 0; i < bytes.Length; i++) {
b = bytes[i] >> 4;
c[i * 2] = (char)(55 + b + (((b-10)>>31)&-7));
b = bytes[i] & 0xF;
c[i * 2 + 1] = (char)(55 + b + (((b-10)>>31)&-7));
}
return new string(c);
}
Ph'nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn
Abandon all hope, ye who enter here
An explanation of the weird bit fiddling:
bytes[i] >> 4 extracts the high nibble of a byte
bytes[i] & 0xF extracts the low nibble of a byte
b - 10
is < 0 for values b < 10, which will become a decimal digit
is >= 0 for values b > 10, which will become a letter from A to F.
Using i >> 31 on a signed 32 bit integer extracts the sign, thanks to sign extension.
It will be -1 for i < 0 and 0 for i >= 0.
Combining 2) and 3), shows that (b-10)>>31 will be 0 for letters and -1 for digits.
Looking at the case for letters, the last summand becomes 0, and b is in the range 10 to 15. We want to map it to A(65) to F(70), which implies adding 55 ('A'-10).
Looking at the case for digits, we want to adapt the last summand so it maps b from the range 0 to 9 to the range 0(48) to 9(57). This means it needs to become -7 ('0' - 55).
Now we could just multiply with 7. But since -1 is represented by all bits being 1, we can instead use & -7 since (0 & -7) == 0 and (-1 & -7) == -7.
Some further considerations:
I didn't use a second loop variable to index into c, since measurement shows that calculating it from i is cheaper.
Using exactly i < bytes.Length as upper bound of the loop allows the JITter to eliminate bounds checks on bytes[i], so I chose that variant.
Making b an int allows unnecessary conversions from and to byte.
If you want more flexibility than BitConverter, but don't want those clunky 1990s-style explicit loops, then you can do:
String.Join(String.Empty, Array.ConvertAll(bytes, x => x.ToString("X2")));
Or, if you're using .NET 4.0:
String.Concat(Array.ConvertAll(bytes, x => x.ToString("X2")));
(The latter from a comment on the original post.)
Another lookup table based approach. This one uses only one lookup table for each byte, instead of a lookup table per nibble.
private static readonly uint[] _lookup32 = CreateLookup32();
private static uint[] CreateLookup32()
{
var result = new uint[256];
for (int i = 0; i < 256; i++)
{
string s=i.ToString("X2");
result[i] = ((uint)s[0]) + ((uint)s[1] << 16);
}
return result;
}
private static string ByteArrayToHexViaLookup32(byte[] bytes)
{
var lookup32 = _lookup32;
var result = new char[bytes.Length * 2];
for (int i = 0; i < bytes.Length; i++)
{
var val = lookup32[bytes[i]];
result[2*i] = (char)val;
result[2*i + 1] = (char) (val >> 16);
}
return new string(result);
}
I also tested variants of this using ushort, struct{char X1, X2}, struct{byte X1, X2} in the lookup table.
Depending on the compilation target (x86, X64) those either had the approximately same performance or were slightly slower than this variant.
And for even higher performance, its unsafe sibling:
private static readonly uint[] _lookup32Unsafe = CreateLookup32Unsafe();
private static readonly uint* _lookup32UnsafeP = (uint*)GCHandle.Alloc(_lookup32Unsafe,GCHandleType.Pinned).AddrOfPinnedObject();
private static uint[] CreateLookup32Unsafe()
{
var result = new uint[256];
for (int i = 0; i < 256; i++)
{
string s=i.ToString("X2");
if(BitConverter.IsLittleEndian)
result[i] = ((uint)s[0]) + ((uint)s[1] << 16);
else
result[i] = ((uint)s[1]) + ((uint)s[0] << 16);
}
return result;
}
public static string ByteArrayToHexViaLookup32Unsafe(byte[] bytes)
{
var lookupP = _lookup32UnsafeP;
var result = new char[bytes.Length * 2];
fixed(byte* bytesP = bytes)
fixed (char* resultP = result)
{
uint* resultP2 = (uint*)resultP;
for (int i = 0; i < bytes.Length; i++)
{
resultP2[i] = lookupP[bytesP[i]];
}
}
return new string(result);
}
Or if you consider it acceptable to write into the string directly:
public static string ByteArrayToHexViaLookup32UnsafeDirect(byte[] bytes)
{
var lookupP = _lookup32UnsafeP;
var result = new string((char)0, bytes.Length * 2);
fixed (byte* bytesP = bytes)
fixed (char* resultP = result)
{
uint* resultP2 = (uint*)resultP;
for (int i = 0; i < bytes.Length; i++)
{
resultP2[i] = lookupP[bytesP[i]];
}
}
return result;
}
You can use the BitConverter.ToString method:
byte[] bytes = {0, 1, 2, 4, 8, 16, 32, 64, 128, 256}
Console.WriteLine( BitConverter.ToString(bytes));
Output:
00-01-02-04-08-10-20-40-80-FF
More information: BitConverter.ToString Method (Byte[])
I just encountered the very same problem today, and I came across this code:
private static string ByteArrayToHex(byte[] barray)
{
char[] c = new char[barray.Length * 2];
byte b;
for (int i = 0; i < barray.Length; ++i)
{
b = ((byte)(barray[i] >> 4));
c[i * 2] = (char)(b > 9 ? b + 0x37 : b + 0x30);
b = ((byte)(barray[i] & 0xF));
c[i * 2 + 1] = (char)(b > 9 ? b + 0x37 : b + 0x30);
}
return new string(c);
}
Source: Forum post byte[] Array to Hex String (see the post by PZahra). I modified the code a little to remove the 0x prefix.
I did some performance testing to the code and it was almost eight times faster than using BitConverter.ToString() (the fastest according to patridge's post).
As of .NET 5 RC2 you can use:
Convert.ToHexString(byte[] inArray) which returns a string and
Convert.FromHexString(string s) which returns a byte[].
Overloads are available that take span parameters.
This is an answer to revision 4 of Tomalak's highly popular answer (and subsequent edits).
I'll make the case that this edit is wrong, and explain why it could be reverted. Along the way, you might learn a thing or two about some internals, and see yet another example of what premature optimization really is and how it can bite you.
tl;dr: Just use Convert.ToByte and String.Substring if you're in a hurry ("Original code" below), it's the best combination if you don't want to re-implement Convert.ToByte. Use something more advanced (see other answers) that doesn't use Convert.ToByte if you need performance. Do not use anything else other than String.Substring in combination with Convert.ToByte, unless someone has something interesting to say about this in the comments of this answer.
warning: This answer may become obsolete if a Convert.ToByte(char[], Int32) overload is implemented in the framework. This is unlikely to happen soon.
As a general rule, I don't much like to say "don't optimize prematurely", because nobody knows when "premature" is. The only thing you must consider when deciding whether to optimize or not is: "Do I have the time and resources to investigate optimization approaches properly?". If you don't, then it's too soon, wait until your project is more mature or until you need the performance (if there is a real need, then you will make the time). In the meantime, do the simplest thing that could possibly work instead.
Original code:
public static byte[] HexadecimalStringToByteArray_Original(string input)
{
var outputLength = input.Length / 2;
var output = new byte[outputLength];
for (var i = 0; i < outputLength; i++)
output[i] = Convert.ToByte(input.Substring(i * 2, 2), 16);
return output;
}
Revision 4:
public static byte[] HexadecimalStringToByteArray_Rev4(string input)
{
var outputLength = input.Length / 2;
var output = new byte[outputLength];
using (var sr = new StringReader(input))
{
for (var i = 0; i < outputLength; i++)
output[i] = Convert.ToByte(new string(new char[2] { (char)sr.Read(), (char)sr.Read() }), 16);
}
return output;
}
The revision avoids String.Substring and uses a StringReader instead. The given reason is:
Edit: you can improve performance for long strings by using a single
pass parser, like so:
Well, looking at the reference code for String.Substring, it's clearly "single-pass" already; and why shouldn't it be? It operates at byte-level, not on surrogate pairs.
It does allocate a new string however, but then you need to allocate one to pass to Convert.ToByte anyway. Furthermore, the solution provided in the revision allocates yet another object on every iteration (the two-char array); you can safely put that allocation outside the loop and reuse the array to avoid that.
public static byte[] HexadecimalStringToByteArray(string input)
{
var outputLength = input.Length / 2;
var output = new byte[outputLength];
var numeral = new char[2];
using (var sr = new StringReader(input))
{
for (var i = 0; i < outputLength; i++)
{
numeral[0] = (char)sr.Read();
numeral[1] = (char)sr.Read();
output[i] = Convert.ToByte(new string(numeral), 16);
}
}
return output;
}
Each hexadecimal numeral represents a single octet using two digits (symbols).
But then, why call StringReader.Read twice? Just call its second overload and ask it to read two characters in the two-char array at once; and reduce the amount of calls by two.
public static byte[] HexadecimalStringToByteArray(string input)
{
var outputLength = input.Length / 2;
var output = new byte[outputLength];
var numeral = new char[2];
using (var sr = new StringReader(input))
{
for (var i = 0; i < outputLength; i++)
{
var read = sr.Read(numeral, 0, 2);
Debug.Assert(read == 2);
output[i] = Convert.ToByte(new string(numeral), 16);
}
}
return output;
}
What you're left with is a string reader whose only added "value" is a parallel index (internal _pos) which you could have declared yourself (as j for example), a redundant length variable (internal _length), and a redundant reference to the input string (internal _s). In other words, it's useless.
If you wonder how Read "reads", just look at the code, all it does is call String.CopyTo on the input string. The rest is just book-keeping overhead to maintain values we don't need.
So, remove the string reader already, and call CopyTo yourself; it's simpler, clearer, and more efficient.
public static byte[] HexadecimalStringToByteArray(string input)
{
var outputLength = input.Length / 2;
var output = new byte[outputLength];
var numeral = new char[2];
for (int i = 0, j = 0; i < outputLength; i++, j += 2)
{
input.CopyTo(j, numeral, 0, 2);
output[i] = Convert.ToByte(new string(numeral), 16);
}
return output;
}
Do you really need a j index that increments in steps of two parallel to i? Of course not, just multiply i by two (which the compiler should be able to optimize to an addition).
public static byte[] HexadecimalStringToByteArray_BestEffort(string input)
{
var outputLength = input.Length / 2;
var output = new byte[outputLength];
var numeral = new char[2];
for (int i = 0; i < outputLength; i++)
{
input.CopyTo(i * 2, numeral, 0, 2);
output[i] = Convert.ToByte(new string(numeral), 16);
}
return output;
}
What does the solution look like now? Exactly like it was at the beginning, only instead of using String.Substring to allocate the string and copy the data to it, you're using an intermediary array to which you copy the hexadecimal numerals to, then allocate the string yourself and copy the data again from the array and into the string (when you pass it in the string constructor). The second copy might be optimized-out if the string is already in the intern pool, but then String.Substring will also be able to avoid it in these cases.
In fact, if you look at String.Substring again, you see that it uses some low-level internal knowledge of how strings are constructed to allocate the string faster than you could normally do it, and it inlines the same code used by CopyTo directly in there to avoid the call overhead.
String.Substring
Worst-case: One fast allocation, one fast copy.
Best-case: No allocation, no copy.
Manual method
Worst-case: Two normal allocations, one normal copy, one fast copy.
Best-case: One normal allocation, one normal copy.
Conclusion? If you want to use Convert.ToByte(String, Int32) (because you don't want to re-implement that functionality yourself), there doesn't seem to be a way to beat String.Substring; all you do is run in circles, re-inventing the wheel (only with sub-optimal materials).
Note that using Convert.ToByte and String.Substring is a perfectly valid choice if you don't need extreme performance. Remember: only opt for an alternative if you have the time and resources to investigate how it works properly.
If there was a Convert.ToByte(char[], Int32), things would be different of course (it would be possible to do what I described above and completely avoid String).
I suspect that people who report better performance by "avoiding String.Substring" also avoid Convert.ToByte(String, Int32), which you should really be doing if you need the performance anyway. Look at the countless other answers to discover all the different approaches to do that.
Disclaimer: I haven't decompiled the latest version of the framework to verify that the reference source is up-to-date, I assume it is.
Now, it all sounds good and logical, hopefully even obvious if you've managed to get so far. But is it true?
Intel(R) Core(TM) i7-3720QM CPU # 2.60GHz
Cores: 8
Current Clock Speed: 2600
Max Clock Speed: 2600
--------------------
Parsing hexadecimal string into an array of bytes
--------------------
HexadecimalStringToByteArray_Original: 7,777.09 average ticks (over 10000 runs), 1.2X
HexadecimalStringToByteArray_BestEffort: 8,550.82 average ticks (over 10000 runs), 1.1X
HexadecimalStringToByteArray_Rev4: 9,218.03 average ticks (over 10000 runs), 1.0X
Yes!
Props to Partridge for the bench framework, it's easy to hack. The input used is the following SHA-1 hash repeated 5000 times to make a 100,000 bytes long string.
209113288F93A9AB8E474EA78D899AFDBB874355
Have fun! (But optimize with moderation.)
Complement to answer by #CodesInChaos (reversed method)
public static byte[] HexToByteUsingByteManipulation(string s)
{
byte[] bytes = new byte[s.Length / 2];
for (int i = 0; i < bytes.Length; i++)
{
int hi = s[i*2] - 65;
hi = hi + 10 + ((hi >> 31) & 7);
int lo = s[i*2 + 1] - 65;
lo = lo + 10 + ((lo >> 31) & 7) & 0x0f;
bytes[i] = (byte) (lo | hi << 4);
}
return bytes;
}
Explanation:
& 0x0f is to support also lower case letters
hi = hi + 10 + ((hi >> 31) & 7); is the same as:
hi = ch-65 + 10 + (((ch-65) >> 31) & 7);
For '0'..'9' it is the same as hi = ch - 65 + 10 + 7; which is hi = ch - 48 (this is because of 0xffffffff & 7).
For 'A'..'F' it is hi = ch - 65 + 10; (this is because of 0x00000000 & 7).
For 'a'..'f' we have to big numbers so we must subtract 32 from default version by making some bits 0 by using & 0x0f.
65 is code for 'A'
48 is code for '0'
7 is the number of letters between '9' and 'A' in the ASCII table (...456789:;<=>?#ABCD...).
Dotnet 5 Update
To convert from byte[] (byte array) to hexadecimal string, use:
System.Convert.ToHexString
var myBytes = new byte[100];
var myString = System.Convert.ToHexString(myBytes);
To convert from hexadecimal string to byte[], use:
System.Convert.FromHexString
var myString = "E10B116E8530A340BCC7B3EAC208487B";
var myBytes = System.Convert.FromHexString(myString);
This problem could also be solved using a look-up table. This would require a small amount of static memory for both the encoder and decoder. This method will however be fast:
Encoder table 512 bytes or 1024 bytes (twice
the size if both upper and lower case
is needed)
Decoder table 256 bytes or
64 KiB (either a single char look-up
or dual char look-up)
My solution uses 1024 bytes for the encoding table, and 256 bytes for decoding.
Decoding
private static readonly byte[] LookupTable = new byte[] {
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF
};
private static byte Lookup(char c)
{
var b = LookupTable[c];
if (b == 255)
throw new IOException("Expected a hex character, got " + c);
return b;
}
public static byte ToByte(char[] chars, int offset)
{
return (byte)(Lookup(chars[offset]) << 4 | Lookup(chars[offset + 1]));
}
Encoding
private static readonly char[][] LookupTableUpper;
private static readonly char[][] LookupTableLower;
static Hex()
{
LookupTableLower = new char[256][];
LookupTableUpper = new char[256][];
for (var i = 0; i < 256; i++)
{
LookupTableLower[i] = i.ToString("x2").ToCharArray();
LookupTableUpper[i] = i.ToString("X2").ToCharArray();
}
}
public static char[] ToCharLower(byte[] b, int bOffset)
{
return LookupTableLower[b[bOffset]];
}
public static char[] ToCharUpper(byte[] b, int bOffset)
{
return LookupTableUpper[b[bOffset]];
}
Comparison
StringBuilderToStringFromBytes: 106148
BitConverterToStringFromBytes: 15783
ArrayConvertAllToStringFromBytes: 54290
ByteManipulationToCharArray: 8444
TableBasedToCharArray: 5651 *
* this solution
Note
During decoding IOException and IndexOutOfRangeException could occur (if a character has a too high value > 256). Methods for de/encoding streams or arrays should be implemented, this is just a proof of concept.
Converting byte[] to a hexadecimal string - benchmark / performance analysis
Updated on: 2022-04-17
Since .NET 5 you should use Convert.ToHexString(bytes[])!
using System;
string result = Convert.ToHexString(bytesToConvert);
About this leaderboard and the benchmark
The comparison from Thymine seems to be outdated and incomplete, especially after .NET 5 with its Convert.ToHexString, so I decided to ~~fall into the bytes to hex string rabbit hole~~ create a new, updated comparison with more methods from answers to both of these two questions.
I went with BenchamrkDotNet instead of a custom-made benchmarking script, which will, hopefully, make the result more accurate.
Remember that micro-benchmarking won't ever represent the actual situation, and you should do your tests.
I ran these benchmarks on a Linux with Kernel 5.15.32 on an AMD Ryzen 5800H with 2x8 GB DDR4 # 2133 MHz.
Be aware that the whole benchmark might take a lot of time to complete - around 40 minutes on my machine.
UPPERCASE (capitalized) vs lowercase output
All methods mentioned (unless stated otherwise) focus on UPPERCASE output only. That means the output will look like B33F69, not b33f69.
The output from Convert.ToHexString is always uppercase. Still, thankfully there isn't any significant performance drop when paired with ToLower(), although both unsafe methods will be faster if that's your concern.
Making the string lowercase efficiently might be a challenge in some methods (especially the ones with bit operators magic), but in most, it's enough to change a parameter X2 to x2 or change the letters from uppercase to lowercase in a mapping.
Leaderboard
It is sorted by Mean N=100. The reference point is the StringBuilderForEachByte method.
Method (means are in nanoseconds)
Mean N=10
Ratio N=10
Mean N=100
Ratio N=100
Mean N=500
Ratio N=500
Mean N=1k
Ratio N=1k
Mean N=10k
Ratio N=10k
Mean N=100k
Ratio N=100k
StringBuilderAggregateBytesAppendFormat
364.92
1.48
3,680.00
1.74
18,928.33
1.86
38,362.94
1.87
380,994.74
1.72
42,618,861.57
1.62
StringBuilderForEachAppendFormat
309.59
1.26
3,203.11
1.52
20,775.07
2.04
41,398.07
2.02
426,839.96
1.93
37,220,750.15
1.41
StringJoinSelect
310.84
1.26
2,765.91
1.31
13,549.12
1.33
28,691.16
1.40
304,163.97
1.38
63,541,601.12
2.41
StringConcatSelect
301.34
1.22
2,733.64
1.29
14,449.53
1.42
29,174.83
1.42
307,196.94
1.39
32,877,994.95
1.25
StringJoinArrayConvertAll
279.21
1.13
2,608.71
1.23
13,305.96
1.30
27,207.12
1.32
295,589.61
1.34
62,950,871.38
2.39
StringBuilderAggregateBytesAppend
276.18
1.12
2,599.62
1.23
12,788.11
1.25
26,043.54
1.27
255,389.06
1.16
27,664,344.41
1.05
StringConcatArrayConvertAll
244.81
0.99
2,361.08
1.12
11,881.18
1.16
23,709.21
1.15
265,197.33
1.20
56,044,744.44
2.12
StringBuilderForEachByte
246.09
1.00
2,112.77
1.00
10,200.36
1.00
20,540.77
1.00
220,993.95
1.00
26,387,941.13
1.00
StringBuilderForEachBytePreAllocated
213.85
0.87
1,897.19
0.90
9,340.66
0.92
19,142.27
0.93
204,968.88
0.93
24,902,075.81
0.94
BitConverterReplace
140.09
0.57
1,207.74
0.57
6,170.46
0.60
12,438.23
0.61
145,022.35
0.66
17,719,082.72
0.67
LookupPerNibble
63.78
0.26
421.75
0.20
1,978.22
0.19
3,957.58
0.19
35,358.21
0.16
4,993,649.91
0.19
LookupAndShift
53.22
0.22
311.56
0.15
1,461.15
0.14
2,924.11
0.14
26,180.11
0.12
3,771,827.62
0.14
WhilePropertyLookup
41.83
0.17
308.59
0.15
1,473.10
0.14
2,925.66
0.14
28,440.28
0.13
5,060,341.10
0.19
LookupAndShiftAlphabetArray
37.06
0.15
290.96
0.14
1,387.01
0.14
3,087.86
0.15
29,883.54
0.14
5,136,607.61
0.19
ByteManipulationDecimal
35.29
0.14
251.69
0.12
1,180.38
0.12
2,347.56
0.11
22,731.55
0.10
4,645,593.05
0.18
ByteManipulationHexMultiply
35.45
0.14
235.22
0.11
1,342.50
0.13
2,661.25
0.13
25,810.54
0.12
7,833,116.68
0.30
ByteManipulationHexIncrement
36.43
0.15
234.31
0.11
1,345.38
0.13
2,737.89
0.13
26,413.92
0.12
7,820,224.57
0.30
WhileLocalLookup
42.03
0.17
223.59
0.11
1,016.93
0.10
1,979.24
0.10
19,360.07
0.09
4,150,234.71
0.16
LookupAndShiftAlphabetSpan
30.00
0.12
216.51
0.10
1,020.65
0.10
2,316.99
0.11
22,357.13
0.10
4,580,277.95
0.17
LookupAndShiftAlphabetSpanMultiply
29.04
0.12
207.38
0.10
985.94
0.10
2,259.29
0.11
22,287.12
0.10
4,563,518.13
0.17
LookupPerByte
32.45
0.13
205.84
0.10
951.30
0.09
1,906.27
0.09
18,311.03
0.08
3,908,692.66
0.15
LookupSpanPerByteSpan
25.69
0.10
184.29
0.09
863.79
0.08
2,035.55
0.10
19,448.30
0.09
4,086,961.29
0.15
LookupPerByteSpan
27.03
0.11
184.26
0.09
866.03
0.08
2,005.34
0.10
19,760.55
0.09
4,192,457.14
0.16
Lookup32SpanUnsafeDirect
16.90
0.07
99.20
0.05
436.66
0.04
895.23
0.04
8,266.69
0.04
1,506,058.05
0.06
Lookup32UnsafeDirect
16.51
0.07
98.64
0.05
436.49
0.04
878.28
0.04
8,278.18
0.04
1,753,655.67
0.07
ConvertToHexString
19.27
0.08
64.83
0.03
295.15
0.03
585.86
0.03
5,445.73
0.02
1,478,363.32
0.06
ConvertToHexString.ToLower()
45.66
-
175.16
-
787.86
-
1,516.65
-
13,939.71
-
2,620,046.76
-
Conclusion
The method ConvertToHexString is undoubtedly the fastest out there, and in my perspective, it should always be used if you have the option - it's swift and clean.
using System;
string result = Convert.ToHexString(bytesToConvert);
If not, I decided to highlight two other methods I consider worthy below.
I decided not to highlight unsafe methods since such code might be not only, well, unsafe, but most projects I've worked with don't allow such code.
Worthy mentions
The first one is LookupPerByteSpan.
The code is almost identical to the code in LookupPerByte by CodesInChaos from this answer. This one is the fastest not-unsafe method benchmarked. The difference between the original and this one is using stack allocation for shorter inputs (up to 512 bytes). This makes this method around 10 % faster on these inputs but around 5 % slower on larger ones. Since most of the data I work with is shorter than larger, I opted for this one. LookupSpanPerByteSpan is also very fast, but the code size of its ReadOnlySpan<byte> mapping is too large compared to all other methods.
private static readonly uint[] Lookup32 = Enumerable.Range(0, 256).Select(i =>
{
string s = i.ToString("X2");
return s[0] + ((uint)s[1] << 16);
}).ToArray();
public string ToHexString(byte[] bytes)
{
var result = bytes.Length * 2 <= 1024
? stackalloc char[bytes.Length * 2]
: new char[bytes.Length * 2];
for (int i = 0; i < bytes.Length; i++)
{
var val = Lookup32[bytes[i]];
result[2 * i] = (char)val;
result[2 * i + 1] = (char)(val >> 16);
}
return new string(result);
}
The second one is LookupAndShiftAlphabetSpanMultiply.
First, I would like to mention that this one is my creation. However, I believe this method is not only pretty fast but also simple to understand.
The speed comes from a change that happened in C# 7.3, where declared ReadOnlySpan<byte> methods returning a constant array initialization - new byte {1, 2, 3, ...} - are compiled as the program's static data, therefore omitting a redundant memory. [source]
private static ReadOnlySpan<byte> HexAlphabetSpan => new[]
{
(byte)'0', (byte)'1', (byte)'2', (byte)'3',
(byte)'4', (byte)'5', (byte)'6', (byte)'7',
(byte)'8', (byte)'9', (byte)'A', (byte)'B',
(byte)'C', (byte)'D', (byte)'E', (byte)'F'
};
public static string ToHexString(byte[] bytes)
{
var res = bytes.Length * 2 <= 1024 ? stackalloc char[bytes.Length * 2] : new char[bytes.Length * 2];
for (var i = 0; i < bytes.Length; ++i)
{
var j = i * 2;
res[j] = (char)HexAlphabetSpan[bytes[i] >> 4];
res[j + 1] = (char)HexAlphabetSpan[bytes[i] & 0xF];
}
return new string(res);
}
Source code
The source code for all methods, the benchmark, and this answer can be found here as a Gist on my GitHub.
Why make it complex? This is simple in Visual Studio 2008:
C#:
string hex = BitConverter.ToString(YourByteArray).Replace("-", "");
VB:
Dim hex As String = BitConverter.ToString(YourByteArray).Replace("-", "")
This is a great post. I like Waleed's solution. I haven't run it through patridge's test but it seems to be quite fast. I also needed the reverse process, converting a hex string to a byte array, so I wrote it as a reversal of Waleed's solution. Not sure if it's any faster than Tomalak's original solution. Again, I did not run the reverse process through patridge's test either.
private byte[] HexStringToByteArray(string hexString)
{
int hexStringLength = hexString.Length;
byte[] b = new byte[hexStringLength / 2];
for (int i = 0; i < hexStringLength; i += 2)
{
int topChar = (hexString[i] > 0x40 ? hexString[i] - 0x37 : hexString[i] - 0x30) << 4;
int bottomChar = hexString[i + 1] > 0x40 ? hexString[i + 1] - 0x37 : hexString[i + 1] - 0x30;
b[i / 2] = Convert.ToByte(topChar + bottomChar);
}
return b;
}
Safe versions:
public static class HexHelper
{
[System.Diagnostics.Contracts.Pure]
public static string ToHex(this byte[] value)
{
if (value == null)
throw new ArgumentNullException("value");
const string hexAlphabet = #"0123456789ABCDEF";
var chars = new char[checked(value.Length * 2)];
unchecked
{
for (int i = 0; i < value.Length; i++)
{
chars[i * 2] = hexAlphabet[value[i] >> 4];
chars[i * 2 + 1] = hexAlphabet[value[i] & 0xF];
}
}
return new string(chars);
}
[System.Diagnostics.Contracts.Pure]
public static byte[] FromHex(this string value)
{
if (value == null)
throw new ArgumentNullException("value");
if (value.Length % 2 != 0)
throw new ArgumentException("Hexadecimal value length must be even.", "value");
unchecked
{
byte[] result = new byte[value.Length / 2];
for (int i = 0; i < result.Length; i++)
{
// 0(48) - 9(57) -> 0 - 9
// A(65) - F(70) -> 10 - 15
int b = value[i * 2]; // High 4 bits.
int val = ((b - '0') + ((('9' - b) >> 31) & -7)) << 4;
b = value[i * 2 + 1]; // Low 4 bits.
val += (b - '0') + ((('9' - b) >> 31) & -7);
result[i] = checked((byte)val);
}
return result;
}
}
}
Unsafe versions For those who prefer performance and do not afraid of unsafeness. About 35% faster ToHex and 10% faster FromHex.
public static class HexUnsafeHelper
{
[System.Diagnostics.Contracts.Pure]
public static unsafe string ToHex(this byte[] value)
{
if (value == null)
throw new ArgumentNullException("value");
const string alphabet = #"0123456789ABCDEF";
string result = new string(' ', checked(value.Length * 2));
fixed (char* alphabetPtr = alphabet)
fixed (char* resultPtr = result)
{
char* ptr = resultPtr;
unchecked
{
for (int i = 0; i < value.Length; i++)
{
*ptr++ = *(alphabetPtr + (value[i] >> 4));
*ptr++ = *(alphabetPtr + (value[i] & 0xF));
}
}
}
return result;
}
[System.Diagnostics.Contracts.Pure]
public static unsafe byte[] FromHex(this string value)
{
if (value == null)
throw new ArgumentNullException("value");
if (value.Length % 2 != 0)
throw new ArgumentException("Hexadecimal value length must be even.", "value");
unchecked
{
byte[] result = new byte[value.Length / 2];
fixed (char* valuePtr = value)
{
char* valPtr = valuePtr;
for (int i = 0; i < result.Length; i++)
{
// 0(48) - 9(57) -> 0 - 9
// A(65) - F(70) -> 10 - 15
int b = *valPtr++; // High 4 bits.
int val = ((b - '0') + ((('9' - b) >> 31) & -7)) << 4;
b = *valPtr++; // Low 4 bits.
val += (b - '0') + ((('9' - b) >> 31) & -7);
result[i] = checked((byte)val);
}
}
return result;
}
}
}
BTW
For benchmark testing initializing alphabet every time convert function called is wrong, alphabet must be const (for string) or static readonly (for char[]). Then alphabet-based conversion of byte[] to string becomes as fast as byte manipulation versions.
And of course test must be compiled in Release (with optimization) and with debug option "Suppress JIT optimization" turned off (same for "Enable Just My Code" if code must be debuggable).
Not to pile on to the many answers here, but I found a fairly optimal (~4.5x better than accepted), straightforward implementation of the hex string parser. First, output from my tests (the first batch is my implementation):
Give me that string:
04c63f7842740c77e545bb0b2ade90b384f119f6ab57b680b7aa575a2f40939f
Time to parse 100,000 times: 50.4192 ms
Result as base64: BMY/eEJ0DHflRbsLKt6Qs4TxGfarV7aAt6pXWi9Ak58=
BitConverter'd: 04-C6-3F-78-42-74-0C-77-E5-45-BB-0B-2A-DE-90-B3-84-F1-19-F6-AB-5
7-B6-80-B7-AA-57-5A-2F-40-93-9F
Accepted answer: (StringToByteArray)
Time to parse 100000 times: 233.1264ms
Result as base64: BMY/eEJ0DHflRbsLKt6Qs4TxGfarV7aAt6pXWi9Ak58=
BitConverter'd: 04-C6-3F-78-42-74-0C-77-E5-45-BB-0B-2A-DE-90-B3-84-F1-19-F6-AB-5
7-B6-80-B7-AA-57-5A-2F-40-93-9F
With Mono's implementation:
Time to parse 100000 times: 777.2544ms
Result as base64: BMY/eEJ0DHflRbsLKt6Qs4TxGfarV7aAt6pXWi9Ak58=
BitConverter'd: 04-C6-3F-78-42-74-0C-77-E5-45-BB-0B-2A-DE-90-B3-84-F1-19-F6-AB-5
7-B6-80-B7-AA-57-5A-2F-40-93-9F
With SoapHexBinary:
Time to parse 100000 times: 845.1456ms
Result as base64: BMY/eEJ0DHflRbsLKt6Qs4TxGfarV7aAt6pXWi9Ak58=
BitConverter'd: 04-C6-3F-78-42-74-0C-77-E5-45-BB-0B-2A-DE-90-B3-84-F1-19-F6-AB-5
7-B6-80-B7-AA-57-5A-2F-40-93-9F
The base64 and 'BitConverter'd' lines are there to test for correctness. Note that they are equal.
The implementation:
public static byte[] ToByteArrayFromHex(string hexString)
{
if (hexString.Length % 2 != 0) throw new ArgumentException("String must have an even length");
var array = new byte[hexString.Length / 2];
for (int i = 0; i < hexString.Length; i += 2)
{
array[i/2] = ByteFromTwoChars(hexString[i], hexString[i + 1]);
}
return array;
}
private static byte ByteFromTwoChars(char p, char p_2)
{
byte ret;
if (p <= '9' && p >= '0')
{
ret = (byte) ((p - '0') << 4);
}
else if (p <= 'f' && p >= 'a')
{
ret = (byte) ((p - 'a' + 10) << 4);
}
else if (p <= 'F' && p >= 'A')
{
ret = (byte) ((p - 'A' + 10) << 4);
} else throw new ArgumentException("Char is not a hex digit: " + p,"p");
if (p_2 <= '9' && p_2 >= '0')
{
ret |= (byte) ((p_2 - '0'));
}
else if (p_2 <= 'f' && p_2 >= 'a')
{
ret |= (byte) ((p_2 - 'a' + 10));
}
else if (p_2 <= 'F' && p_2 >= 'A')
{
ret |= (byte) ((p_2 - 'A' + 10));
} else throw new ArgumentException("Char is not a hex digit: " + p_2, "p_2");
return ret;
}
I tried some stuff with unsafe and moving the (clearly redundant) character-to-nibble if sequence to another method, but this was the fastest it got.
(I concede that this answers half the question. I felt that the string->byte[] conversion was underrepresented, while the byte[]->string angle seems to be well covered. Thus, this answer.)
From Microsoft's developers, a nice, simple conversion:
public static string ByteArrayToString(byte[] ba)
{
// Concatenate the bytes into one long string
return ba.Aggregate(new StringBuilder(32),
(sb, b) => sb.Append(b.ToString("X2"))
).ToString();
}
While the above is clean and compact, performance junkies will scream about it using enumerators. You can get peak performance with an improved version of Tomalak's original answer:
public static string ByteArrayToString(byte[] ba)
{
StringBuilder hex = new StringBuilder(ba.Length * 2);
for(int i=0; i < ba.Length; i++) // <-- Use for loop is faster than foreach
hex.Append(ba[i].ToString("X2")); // <-- ToString is faster than AppendFormat
return hex.ToString();
}
This is the fastest of all the routines I've seen posted here so far. Don't just take my word for it... performance test each routine and inspect its CIL code for yourself.
Inverse function for Waleed Eissa code (Hex String To Byte Array):
public static byte[] HexToBytes(this string hexString)
{
byte[] b = new byte[hexString.Length / 2];
char c;
for (int i = 0; i < hexString.Length / 2; i++)
{
c = hexString[i * 2];
b[i] = (byte)((c < 0x40 ? c - 0x30 : (c < 0x47 ? c - 0x37 : c - 0x57)) << 4);
c = hexString[i * 2 + 1];
b[i] += (byte)(c < 0x40 ? c - 0x30 : (c < 0x47 ? c - 0x37 : c - 0x57));
}
return b;
}
Waleed Eissa function with lower case support:
public static string BytesToHex(this byte[] barray, bool toLowerCase = true)
{
byte addByte = 0x37;
if (toLowerCase) addByte = 0x57;
char[] c = new char[barray.Length * 2];
byte b;
for (int i = 0; i < barray.Length; ++i)
{
b = ((byte)(barray[i] >> 4));
c[i * 2] = (char)(b > 9 ? b + addByte : b + 0x30);
b = ((byte)(barray[i] & 0xF));
c[i * 2 + 1] = (char)(b > 9 ? b + addByte : b + 0x30);
}
return new string(c);
}
Extension methods (disclaimer: completely untested code, BTW...):
public static class ByteExtensions
{
public static string ToHexString(this byte[] ba)
{
StringBuilder hex = new StringBuilder(ba.Length * 2);
foreach (byte b in ba)
{
hex.AppendFormat("{0:x2}", b);
}
return hex.ToString();
}
}
etc.. Use either of Tomalak's three solutions (with the last one being an extension method on a string).
.NET 5 has added the Convert.ToHexString method.
For those using an older version of .NET
internal static class ByteArrayExtensions
{
public static string ToHexString(this byte[] bytes, Casing casing = Casing.Upper)
{
Span<char> result = stackalloc char[0];
if (bytes.Length > 16)
{
var array = new char[bytes.Length * 2];
result = array.AsSpan();
}
else
{
result = stackalloc char[bytes.Length * 2];
}
int pos = 0;
foreach (byte b in bytes)
{
ToCharsBuffer(b, result, pos, casing);
pos += 2;
}
return result.ToString();
}
private static void ToCharsBuffer(byte value, Span<char> buffer, int startingIndex = 0, Casing casing = Casing.Upper)
{
uint difference = (((uint)value & 0xF0U) << 4) + ((uint)value & 0x0FU) - 0x8989U;
uint packedResult = ((((uint)(-(int)difference) & 0x7070U) >> 4) + difference + 0xB9B9U) | (uint)casing;
buffer[startingIndex + 1] = (char)(packedResult & 0xFF);
buffer[startingIndex] = (char)(packedResult >> 8);
}
}
public enum Casing : uint
{
// Output [ '0' .. '9' ] and [ 'A' .. 'F' ].
Upper = 0,
// Output [ '0' .. '9' ] and [ 'a' .. 'f' ].
Lower = 0x2020U,
}
Adapted from the .NET repository
https://github.com/dotnet/runtime/blob/v5.0.3/src/libraries/System.Private.CoreLib/src/System/Convert.cs
https://github.com/dotnet/runtime/blob/v5.0.3/src/libraries/Common/src/System/HexConverter.cs
Fastest method for old school people... miss you pointers
static public byte[] HexStrToByteArray(string str)
{
byte[] res = new byte[(str.Length % 2 != 0 ? 0 : str.Length / 2)]; //check and allocate memory
for (int i = 0, j = 0; j < res.Length; i += 2, j++) //convert loop
res[j] = (byte)((str[i] % 32 + 9) % 25 * 16 + (str[i + 1] % 32 + 9) % 25);
return res;
}
Tests: Hex String To Byte Array
I noticed that most of tests were performed on functions that convert Bytes array to Hex string.
So, in this post I will focus on the other side: functions that convert Hex String To Byte Array.
If you are interested in result only, you could skip down to Summary section.
The test code file is supplied at the end of the post.
Labels
I would like to name the function from the accepted answer (by Tomalak) StringToByteArrayV1, or to shortcut it to V1. rest of functions will be named in same way: V2, V3, V4, ..., etc.
Index of Participating Functions
StringToByteArrayV1 by Tomalak (the accepted answer)
StringToByteArrayV2 by Mykroft (using SoapHexBinary)
StringToByteArrayV3 by drphrozen (look-up table)
StringToByteArrayV4 by CoperNick (Byte Manipulation)
StringToByteArrayV5_1 by Chris F (Byte Manipulation)
StringToByteArrayV5_2 by Chris F ( V5_1 + ehanced it based on Amir Rezaei's comment)
StringToByteArrayV5_3 by Chris F ( V5_2 + ehanced it based on Ben Voigt's comment) (you could see final shape of it at this post published test code)
StringToByteArrayV6 by Ben Mosher (Byte Manipulation)
StringToByteArrayV7 by Maratius (Byte Manipulation - the safe version)
StringToByteArrayV8 by Maratius (Byte Manipulation - the unsafe version)
StringToByteArrayV9 by Geograph
StringToByteArrayV10 by AlejandroAlis
StringToByteArrayV11 by Fredrik Hu
StringToByteArrayV12 by Maarten Bodewes
StringToByteArrayV13 by ClausAndersen
StringToByteArrayV14 by Stas Makutin
StringToByteArrayV15 by JJJ
StringToByteArrayV16 by JamieSee
StringToByteArrayV17 by spacepille
StringToByteArrayV18 by Gregory Morse
StringToByteArrayV19 by Rick
StringToByteArrayV20 by SandRock
StringToByteArrayV21 by Paul
Correctness Test
I have tested correctness by passing all 256 possible values of 1 byte, then checking output to see if correct.
Result:
V18 has issue with strings start with "00" (see Roger Stewart comment on it ). other than that it passes all tests.
if hex string alphabet letters are uppercase: all functions successfully passed
if hex string alphabet letters are lowercase then the following functions failed: V5_1, V5_2, v7, V8, V15, V19
note: V5_3 solves this issue (of V5_1 and V5_2)
Performance Test
I have done performance tests using Stopwatch class.
Performance for long strings
input length: 10,000,000 bytes
runs: 100
average elapsed time per run:
V1 = 136.4ms
V2 = 104.5ms
V3 = 22.0ms
V4 = 9.9ms
V5_1 = 10.2ms
V5_2 = 9.0ms
V5_3 = 9.3ms
V6 = 18.3ms
V7 = 9.8ms
V8 = 8.8ms
V9 = 10.2ms
V10 = 19.0ms
V11 = 12.2ms
V12 = 27.4ms
V13 = 21.8ms
V14 = 12.0ms
V15 = 14.9ms
V16 = 15.3ms
V17 = 9.5ms
V18 got excluded from this test, because it was very slow when using very long string
V19 = 222.8ms
V20 = 66.0ms
V21 = 15.4ms
V1 average ticks per run: 1363529.4
V2 is more fast than V1 by: 1.3 times (ticks ratio)
V3 is more fast than V1 by: 6.2 times (ticks ratio)
V4 is more fast than V1 by: 13.8 times (ticks ratio)
V5_1 is more fast than V1 by: 13.3 times (ticks ratio)
V5_2 is more fast than V1 by: 15.2 times (ticks ratio)
V5_3 is more fast than V1 by: 14.8 times (ticks ratio)
V6 is more fast than V1 by: 7.4 times (ticks ratio)
V7 is more fast than V1 by: 13.9 times (ticks ratio)
V8 is more fast than V1 by: 15.4 times (ticks ratio)
V9 is more fast than V1 by: 13.4 times (ticks ratio)
V10 is more fast than V1 by: 7.2 times (ticks ratio)
V11 is more fast than V1 by: 11.1 times (ticks ratio)
V12 is more fast than V1 by: 5.0 times (ticks ratio)
V13 is more fast than V1 by: 6.3 times (ticks ratio)
V14 is more fast than V1 by: 11.4 times (ticks ratio)
V15 is more fast than V1 by: 9.2 times (ticks ratio)
V16 is more fast than V1 by: 8.9 times (ticks ratio)
V17 is more fast than V1 by: 14.4 times (ticks ratio)
V19 is more SLOW than V1 by: 1.6 times (ticks ratio)
V20 is more fast than V1 by: 2.1 times (ticks ratio)
V21 is more fast than V1 by: 8.9 times (ticks ratio)
Performance of V18 for long strings
V18 took long time at the previous test,
so let's decrease length for it:
input length: 1,000,000 bytes
runs: 100
average elapsed time per run: V1 = 14.1ms , V18 = 146.7ms
V1 average ticks per run: 140630.3
V18 is more SLOW than V1 by: 10.4 times (ticks ratio)
Performance for short strings
input length: 100 byte
runs: 1,000,000
V1 average ticks per run: 14.6
V2 is more fast than V1 by: 1.4 times (ticks ratio)
V3 is more fast than V1 by: 5.9 times (ticks ratio)
V4 is more fast than V1 by: 15.7 times (ticks ratio)
V5_1 is more fast than V1 by: 15.1 times (ticks ratio)
V5_2 is more fast than V1 by: 18.4 times (ticks ratio)
V5_3 is more fast than V1 by: 16.3 times (ticks ratio)
V6 is more fast than V1 by: 5.3 times (ticks ratio)
V7 is more fast than V1 by: 15.7 times (ticks ratio)
V8 is more fast than V1 by: 18.0 times (ticks ratio)
V9 is more fast than V1 by: 15.5 times (ticks ratio)
V10 is more fast than V1 by: 7.8 times (ticks ratio)
V11 is more fast than V1 by: 12.4 times (ticks ratio)
V12 is more fast than V1 by: 5.3 times (ticks ratio)
V13 is more fast than V1 by: 5.2 times (ticks ratio)
V14 is more fast than V1 by: 13.4 times (ticks ratio)
V15 is more fast than V1 by: 9.9 times (ticks ratio)
V16 is more fast than V1 by: 9.2 times (ticks ratio)
V17 is more fast than V1 by: 16.2 times (ticks ratio)
V18 is more fast than V1 by: 1.1 times (ticks ratio)
V19 is more SLOW than V1 by: 1.6 times (ticks ratio)
V20 is more fast than V1 by: 1.9 times (ticks ratio)
V21 is more fast than V1 by: 11.4 times (ticks ratio)
Testing Code
It is good idea to read Disclaimer section down here in this post, before using any from the following code
https://github.com/Ghosticollis/performance-tests/blob/main/MTestPerformance.cs
Summary
I recommend using one of the following functions, because of the good performance, and support both upper and lower case:
StringToByteArrayV4 by CoperNick
StringToByteArrayV9 by Geograph
StringToByteArrayV17 by spacepille
StringToByteArrayV5_3 basically by Chris F (it is base on V5_1, but I have enhanced it based on Amir Rezaei's and Ben Voigt's comments).
Here is the final shape of V5_3:
static byte[] HexStringToByteArrayV5_3(string hexString) {
int hexStringLength = hexString.Length;
byte[] b = new byte[hexStringLength / 2];
for (int i = 0; i < hexStringLength; i += 2) {
int topChar = hexString[i];
topChar = (topChar > 0x40 ? (topChar & ~0x20) - 0x37 : topChar - 0x30) << 4;
int bottomChar = hexString[i + 1];
bottomChar = bottomChar > 0x40 ? (bottomChar & ~0x20) - 0x37 : bottomChar - 0x30;
b[i / 2] = (byte)(topChar + bottomChar);
}
return b;
}
Disclaimer
WARNING: I don't have proper knowledge in testing. The main purpose of these primitive tests is to give quick overview on what might be good from all of posted functions.
If you need accurate results, please use proper testing tools.
Finally, I would like to say I am new to be active at stackoverflow, sorry if my post is lacking.
comments to enhance this post would be appreciated.
And for inserting into an SQL string (if you're not using command parameters):
public static String ByteArrayToSQLHexString(byte[] Source)
{
return = "0x" + BitConverter.ToString(Source).Replace("-", "");
}
In terms of speed, this seems to be better than anything here:
public static string ToHexString(byte[] data) {
byte b;
int i, j, k;
int l = data.Length;
char[] r = new char[l * 2];
for (i = 0, j = 0; i < l; ++i) {
b = data[i];
k = b >> 4;
r[j++] = (char)(k > 9 ? k + 0x37 : k + 0x30);
k = b & 15;
r[j++] = (char)(k > 9 ? k + 0x37 : k + 0x30);
}
return new string(r);
}
I did not get the code you suggested to work, Olipro. hex[i] + hex[i+1] apparently returned an int.
I did, however have some success by taking some hints from Waleeds code and hammering this together. It's ugly as hell but it seems to work and performs at 1/3 of the time compared to the others according to my tests (using patridges testing mechanism). Depending on input size. Switching around the ?:s to separate out 0-9 first would probably yield a slightly faster result since there are more numbers than letters.
public static byte[] StringToByteArray2(string hex)
{
byte[] bytes = new byte[hex.Length/2];
int bl = bytes.Length;
for (int i = 0; i < bl; ++i)
{
bytes[i] = (byte)((hex[2 * i] > 'F' ? hex[2 * i] - 0x57 : hex[2 * i] > '9' ? hex[2 * i] - 0x37 : hex[2 * i] - 0x30) << 4);
bytes[i] |= (byte)(hex[2 * i + 1] > 'F' ? hex[2 * i + 1] - 0x57 : hex[2 * i + 1] > '9' ? hex[2 * i + 1] - 0x37 : hex[2 * i + 1] - 0x30);
}
return bytes;
}
This version of ByteArrayToHexViaByteManipulation could be faster.
From my reports:
ByteArrayToHexViaByteManipulation3: 1,68 average ticks (over 1000 runs), 17,5X
ByteArrayToHexViaByteManipulation2: 1,73 average ticks (over 1000 runs), 16,9X
ByteArrayToHexViaByteManipulation: 2,90 average ticks (over 1000 runs), 10,1X
ByteArrayToHexViaLookupAndShift: 3,22 average ticks (over 1000 runs), 9,1X
...
static private readonly char[] hexAlphabet = new char[]
{'0','1','2','3','4','5','6','7','8','9','A','B','C','D','E','F'};
static string ByteArrayToHexViaByteManipulation3(byte[] bytes)
{
char[] c = new char[bytes.Length * 2];
byte b;
for (int i = 0; i < bytes.Length; i++)
{
b = ((byte)(bytes[i] >> 4));
c[i * 2] = hexAlphabet[b];
b = ((byte)(bytes[i] & 0xF));
c[i * 2 + 1] = hexAlphabet[b];
}
return new string(c);
}
And I think this one is an optimization:
static private readonly char[] hexAlphabet = new char[]
{'0','1','2','3','4','5','6','7','8','9','A','B','C','D','E','F'};
static string ByteArrayToHexViaByteManipulation4(byte[] bytes)
{
char[] c = new char[bytes.Length * 2];
for (int i = 0, ptr = 0; i < bytes.Length; i++, ptr += 2)
{
byte b = bytes[i];
c[ptr] = hexAlphabet[b >> 4];
c[ptr + 1] = hexAlphabet[b & 0xF];
}
return new string(c);
}
I'll enter this bit fiddling competition as I have an answer that also uses bit-fiddling to decode hexadecimals. Note that using character arrays may be even faster as calling StringBuilder methods will take time as well.
public static String ToHex (byte[] data)
{
int dataLength = data.Length;
// pre-create the stringbuilder using the length of the data * 2, precisely enough
StringBuilder sb = new StringBuilder (dataLength * 2);
for (int i = 0; i < dataLength; i++) {
int b = data [i];
// check using calculation over bits to see if first tuple is a letter
// isLetter is zero if it is a digit, 1 if it is a letter
int isLetter = (b >> 7) & ((b >> 6) | (b >> 5)) & 1;
// calculate the code using a multiplication to make up the difference between
// a digit character and an alphanumerical character
int code = '0' + ((b >> 4) & 0xF) + isLetter * ('A' - '9' - 1);
// now append the result, after casting the code point to a character
sb.Append ((Char)code);
// do the same with the lower (less significant) tuple
isLetter = (b >> 3) & ((b >> 2) | (b >> 1)) & 1;
code = '0' + (b & 0xF) + isLetter * ('A' - '9' - 1);
sb.Append ((Char)code);
}
return sb.ToString ();
}
public static byte[] FromHex (String hex)
{
// pre-create the array
int resultLength = hex.Length / 2;
byte[] result = new byte[resultLength];
// set validity = 0 (0 = valid, anything else is not valid)
int validity = 0;
int c, isLetter, value, validDigitStruct, validDigit, validLetterStruct, validLetter;
for (int i = 0, hexOffset = 0; i < resultLength; i++, hexOffset += 2) {
c = hex [hexOffset];
// check using calculation over bits to see if first char is a letter
// isLetter is zero if it is a digit, 1 if it is a letter (upper & lowercase)
isLetter = (c >> 6) & 1;
// calculate the tuple value using a multiplication to make up the difference between
// a digit character and an alphanumerical character
// minus 1 for the fact that the letters are not zero based
value = ((c & 0xF) + isLetter * (-1 + 10)) << 4;
// check validity of all the other bits
validity |= c >> 7; // changed to >>, maybe not OK, use UInt?
validDigitStruct = (c & 0x30) ^ 0x30;
validDigit = ((c & 0x8) >> 3) * (c & 0x6);
validity |= (isLetter ^ 1) * (validDigitStruct | validDigit);
validLetterStruct = c & 0x18;
validLetter = (((c - 1) & 0x4) >> 2) * ((c - 1) & 0x2);
validity |= isLetter * (validLetterStruct | validLetter);
// do the same with the lower (less significant) tuple
c = hex [hexOffset + 1];
isLetter = (c >> 6) & 1;
value ^= (c & 0xF) + isLetter * (-1 + 10);
result [i] = (byte)value;
// check validity of all the other bits
validity |= c >> 7; // changed to >>, maybe not OK, use UInt?
validDigitStruct = (c & 0x30) ^ 0x30;
validDigit = ((c & 0x8) >> 3) * (c & 0x6);
validity |= (isLetter ^ 1) * (validDigitStruct | validDigit);
validLetterStruct = c & 0x18;
validLetter = (((c - 1) & 0x4) >> 2) * ((c - 1) & 0x2);
validity |= isLetter * (validLetterStruct | validLetter);
}
if (validity != 0) {
throw new ArgumentException ("Hexadecimal encoding incorrect for input " + hex);
}
return result;
}
Converted from Java code.
For performance I would go with drphrozens solution. A tiny optimization for the decoder could be to use a table for either char to get rid of the "<< 4".
Clearly the two method calls are costly. If some kind of check is made either on input or output data (could be CRC, checksum or whatever) the if (b == 255)... could be skipped and thereby also the method calls altogether.
Using offset++ and offset instead of offset and offset + 1 might give some theoretical benefit but I suspect the compiler handles this better than me.
private static readonly byte[] LookupTableLow = new byte[] {
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF
};
private static readonly byte[] LookupTableHigh = new byte[] {
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0x00, 0x10, 0x20, 0x30, 0x40, 0x50, 0x60, 0x70, 0x80, 0x90, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xA0, 0xB0, 0xC0, 0xD0, 0xE0, 0xF0, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xA0, 0xB0, 0xC0, 0xD0, 0xE0, 0xF0, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF
};
private static byte LookupLow(char c)
{
var b = LookupTableLow[c];
if (b == 255)
throw new IOException("Expected a hex character, got " + c);
return b;
}
private static byte LookupHigh(char c)
{
var b = LookupTableHigh[c];
if (b == 255)
throw new IOException("Expected a hex character, got " + c);
return b;
}
public static byte ToByte(char[] chars, int offset)
{
return (byte)(LookupHigh(chars[offset++]) | LookupLow(chars[offset]));
}
This is just off the top of my head and has not been tested or benchmarked.

Inverse linear transformation

I am implementing a square encryption algorithm, but maybe someone knows a way to inverse linear transformation. I take the input bytes of 16 bytes and put them row by row into a two-dimensional matrix input_state[4,4] doing a linear transformation by the following C# code.
private byte[,] LinerPreob(byte[,] input_state)
{
byte[,] coef = new byte[4, 4]{
{ 0x02, 0x01, 0x01, 0x03 },
{ 0x03, 0x02, 0x01, 0x01 },
{ 0x01, 0x03, 0x02, 0x01 },
{ 0x01, 0x01, 0x03, 0x02 }
};
byte[,] result = new byte[4, 4];
byte temp;
for(int i = 0; i < 4; i++)
for (int j = 0; j < 4; j++)
{
temp = 0;
for (int k = 0; k < 4; k++)
temp ^= Mult(input_state[i, k], coef[k, j]);
result[i, j] = temp;
}
return result;
}
Transformation over each byte in the matrix according to the formula temp^=multiplication in the Galois field (256) (function Mult) of the current byte by the coefficient in the coef matrix. ^ is an XOR operation.

Appending Array on top of Array to specified length

I've made a function to append an array on top of itself to a specified length. The only issue I'm having is when the array is an odd number, say 9, and the length I'm trying to get it to, for example 20, it would output with 2 zeros at the end. I'm sure there's a way to have the array fill in every byte.
Current Function:
public static byte[] AppendToLen(byte[] input, int length)
{
byte[] output = new byte[length];
if (length <= input.Length) return null;
for (int i = 0; i < length / input.Length; i++)
Array.Copy(input, 0, output, input.Length * i, input.Length);
return output;
}
Block of bytes
byte[] Block = new byte[0x10] { 0x02, 0x03, 0xFF, 0x04, 0x61, 0x37, 0x5f, 0xe8, 0x19, 0x70, 0xa2, 0x77, 0x8c, 0x94, 0x89, 0xb4 };
An example is:
foreach(byte bit in AppendToLen(Block, 56)) {
Console.WriteLine(bit.ToString("X2"));
}
Ouput: 0203FF0461375FE81970A2778C9489B40203FF0461375FE81970A2778C9489B40203FF0461375FE81970A2778C9489B40000000000000000
In the example above you are only iterating twice because the length / input.Length term rounds down to 2. Remember that this is integer division. Also, remember not to walk off the end of your array. The snippet below does what you want.
public static byte[] AppendToLen(byte[] input, int length)
{
byte[] output = new byte[length];
if (length <= input.Length) return null;
// Just use offset here since that's what you care about
for (int offset = 0; offset < length; offset += input.Length)
// Copy as much of the input array as possible to the output,
// starting at this iteration's offset
Array.Copy(input, 0, output, offset, Math.Min(length - offset, input.Length));
return output;
}
The problem is that you are not assigning any value to the remainder bytes. In your example 56 % 16 = 8, so you have 8 bytes left at the end after your loop without initialize. Now, the way you want to fill those I don't know, maybe with the first 8 bytes in your input array. If so, you would need to add an extra check after the loop to see if the remainder is > 0, and copy those 8 (remainder) bytes to the last positions of your array.

Converting Type Int to Type Byte

I have a small set of code that attempts to bit-shift a 1-bit from the 0 index to the 7 index in a byte. After each bit shift I want to take the int value and convert it to a byte:
for(int t=0; t<8; t++)
{
int ShiftBit = 0x80 >> t;
byte ShiftBitByte = Convert.ToByte(ShiftBit.ToString(),8);
}
I would expect the outputs to be:
0x80
0x40
0x20
0x10
0x08
0x04
0x02
0x01
When I run my code I encounter an exception "Additional non-parsable characters are at the end of the string." Is there a better way to capture these bytes?
Thank you
Why don't do you this?
for ( int i = 0 ; i < 8 ; ++i )
{
int s = 0x80 >> i ;
byte b = (byte) s ;
)
Or (cleaner):
for ( int i = 0x00000080 ; i != 0 ; i >>1 )
{
byte b = (byte) i ;
}
To turn a byte into a hex string, something like
byte b = ...
string s = string.Format("0x{0:X2}",b) ;
Ought to do you. But a byte is a number, it doesn't have a format (representation) until you turn it into a string a give it one.
Are you looking for this ?
for (int t = 0; t < 8; t++)
{
int ShiftBit = 0x80 >> t;
byte ShiftBitByte = (byte) ShiftBit;
Console.WriteLine("0x{0:X}",ShiftBitByte);
}
See Standard Numeric Format Strings
You're getting the error because you are erroneously specifying that the string is in base 8. The digit '8' is not a legal digit in base 8, which uses only 0 through 7.
Why not
for (byte b = 0x80; b != 0; b >>= 1)
?
Thanks for catching that ... what I really meant to write was shift a bit in int 0x10000000 from index 0 to 7 and then convert it to a byte each time for an array [0x80, 0x40, 0x20, 0x10, 0x08, 0x04, 0x02, 0x01]. Is that possible?
I don't understand what int 0x10000000 has to do with it. If you want that array, you can achieve that in any of a number of ways:
byte[] xs = { 0x80, 0x40, 0x20, 0x10, 0x08, 0x04, 0x02, 0x01 };
byte[] ys = Enumerable.Range(0, 8).Select(i => (byte)(0x80 >> i)).ToArray();
byte[] zs = new byte[8];
for (int index = 0; index < zs.Length; index++)
zs[index] = (byte)(0x80 >> index);

C# ICMPv6 checksum calculation

I need to compute ICMPv6 checksum. I have found correct manual in RFC 2460 and written code in c#, which is pasted below. In a code you can see that i add to checksum source IP, destination IP of packet, than ICMPv6 message length (which is 32 bytes for Neighbor Advertisement) and then i add next header ID which is 58 for ICMPv6. Then there is FOR cycle, which adds to checksum whole ICMPv6 message (which starts with message type i believe, e.g. 88 00 ... for Neighbor advertisement). Than i made calculations and complements checksum, but it's wrong. I am trying to compute this checksum for real sniffed Neighbor Advertisement, which is also below, but i can't get the same checksum. What could be wrong in my code?
static void Main(string[] args)
{
ICaptureDevice device = new OfflineCaptureDevice("icmp.pcap");
device.Open();
device.OnPacketArrival += new SharpPcap.PacketArrivalEventHandler(device_OnPacketArrival);
device.Capture();
Console.ReadKey();
}
private static void device_OnPacketArrival(object sender, SharpPcap.CaptureEventArgs e)
{
try
{
//packet conversions
var packet = PacketDotNet.Packet.ParsePacket(e.Packet);
var ethernetPacket = ((PacketDotNet.EthernetPacket)packet);
int dlzka_packetu = e.Packet.Data.Length;
string eth = BitConverter.ToString(ethernetPacket.Bytes);
//now its in string field format - one string item is one byte, e.g. FF
string[] eth_final2 = eth.Split('-');
foreach (string bytes in eth_final2) { Console.Write(bytes + " "); }
Console.WriteLine();
//taking out source IP
IPAddress src_ip = IPAddress.Parse(eth_final2[22]+eth_final2[23]+":"+eth_final2[24]+eth_final2[25]+":"+eth_final2[26]+eth_final2[27]+":"+eth_final2[28]+eth_final2[29]+":"+eth_final2[30]+eth_final2[31]+":"+eth_final2[32]+eth_final2[33]+":"+eth_final2[34]+eth_final2[35]+":"+eth_final2[36]+eth_final2[37]);
//destination IP
IPAddress dst_ip = IPAddress.Parse(eth_final2[38] + eth_final2[39] + ":" + eth_final2[40] + eth_final2[41] + ":" + eth_final2[42] + eth_final2[43] + ":" + eth_final2[44] + eth_final2[45] + ":" + eth_final2[46] + eth_final2[47] + ":" + eth_final2[48] + eth_final2[49] + ":" + eth_final2[50] + eth_final2[51] + ":" + eth_final2[52] + eth_final2[53]);
Console.WriteLine(src_ip);
Console.WriteLine(dst_ip);
int icmpv6_length = 32;
int next_header = 58;
Console.WriteLine();
string icmp_payload = "";
//taking out ICMPv6 message
for (int i = 54; i < 54 + 32; i++)
{
if (i == 56 || i == 57) { icmp_payload += "00"; }
else icmp_payload += eth_final2[i];
}
Console.WriteLine(icmp_payload);
byte[] icmp_bytes = GetStringToBytes(icmp_payload);
//CALLING THE FUNCTION ICMPchecksum
ushort sum = ICMPchecksum(src_ip.GetAddressBytes(), dst_ip.GetAddressBytes(), BitConverter.GetBytes(icmpv6_length), BitConverter.GetBytes(next_header), icmp_bytes);
Console.WriteLine(sum.ToString("X"));
}
catch (Exception ex)
{
Console.WriteLine("ERROR");
}
}
static byte[] GetStringToBytes(string value)
{
SoapHexBinary shb = SoapHexBinary.Parse(value);
return shb.Value;
}
static ushort ICMPchecksum(byte[] src_ip, byte[] dst_ip, byte[] length, byte[] next, byte[] payload)
{
ushort checksum = 0;
//length of byte fields
Console.WriteLine("src_ip: "+src_ip.Length+" dst_ip: "+dst_ip.Length+" length: "+length.Length+" next_header: "+next.Length+" payload: "+payload.Length);
//display all fields, which will be used for checksum calculation
Console.WriteLine(BitConverter.ToString(src_ip));
Console.WriteLine(BitConverter.ToString(dst_ip));
Console.WriteLine(BitConverter.ToString(length));
Console.WriteLine(BitConverter.ToString(next));
Console.WriteLine(BitConverter.ToString(payload));
//ADDS SOURCE IPv6 address
checksum += BitConverter.ToUInt16(src_ip, 0);
checksum += BitConverter.ToUInt16(src_ip, 2);
checksum += BitConverter.ToUInt16(src_ip, 4);
checksum += BitConverter.ToUInt16(src_ip, 6);
checksum += BitConverter.ToUInt16(src_ip, 8);
checksum += BitConverter.ToUInt16(src_ip, 10);
checksum += BitConverter.ToUInt16(src_ip, 12);
checksum += BitConverter.ToUInt16(src_ip, 14);
//ADDS DEST IPv6 address
checksum += BitConverter.ToUInt16(dst_ip, 0);
checksum += BitConverter.ToUInt16(dst_ip, 2);
checksum += BitConverter.ToUInt16(dst_ip, 4);
checksum += BitConverter.ToUInt16(dst_ip, 6);
checksum += BitConverter.ToUInt16(dst_ip, 8);
checksum += BitConverter.ToUInt16(dst_ip, 10);
checksum += BitConverter.ToUInt16(dst_ip, 12);
checksum += BitConverter.ToUInt16(dst_ip, 14);
//ADDS LENGTH OF ICMPv6 packet
checksum += BitConverter.ToUInt16(length, 0);
checksum += BitConverter.ToUInt16(length, 2);
//ADDS NEXT HEADER ID = 58
checksum += BitConverter.ToUInt16(next, 0);
checksum += BitConverter.ToUInt16(next, 2);
//ADDS WHOLE ICMPv6 message
for (int i = 0; i < payload.Length; i = i + 2)
{
Console.WriteLine(i);
checksum += BitConverter.ToUInt16(payload, i);
}
checksum += (ushort)(checksum >> 16);
return (ushort)~checksum;
}
And here is packet screen.
only link : http://img831.imageshack.us/img831/7237/icmpv6.png
plus the link, where you can download the file, i am using for testing:
http://www.2shared.com/file/wIETWTWB/icmp.html
Thanks for help.
a payload is used the ICMPv6 message, as it is signed in picture i have posted in my question
At least that is wrong. when you calculate the checksum, you should replace the bytes representing the checksum itself with 0 first.
Update
Based on your icmp.pcap I have created a standalone program to calculate the checksum. See below for the full source.
The most important error is:
ushort checksum = 0;
Although the checksum is 16-bit, this algorithm requires at least 32-bits for temporary values:
uint checksum = 0;
The checksum calculation requires ones' complement arithmetic where the carry of an addition wraps around. To improve performance, all wrapping is done at the end;
checksum += (ushort)(checksum >> 16);
But that is only possible if checksum has more than 16 bits. Otherwise this line would be useless.
But there is something else to take in account. The BitConverter.ToUInt16 depends on the endianness of your processor, which is probably Little Endian. That will make the result A5AB. Which is probably not as you expect, but if you switch bytes, you get the correct version. I inserted a function from here to fix that. That will make the result ABA5. On the other hand, if you don't set the checksum to 0 before the calculation, the ICMPchecksum will always return 0 if the checksum is valid. Also note that your function cannot handle a odd length of the payload.
class Program {
static void Main (string [] args)
{
byte[] src_ip = new byte[]
{
0xfe, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x02, 0x19, 0x55, 0xff, 0xfe, 0x27, 0x27, 0xd0
};
byte[] dst_ip = new byte[]
{
0xfe, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x02, 0x22, 0x75, 0xff, 0xfe, 0xd6, 0xfe, 0x50
};
byte[] length = new byte[] {0, 0, 0, 32};
byte [] next = new byte [] { 0, 0, 0, 58 };
byte[] payload = new byte[]
{
0x88, 0x00, 0xab, 0xa5, 0xe0, 0x00, 0x00, 0x00,
0xfe, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x02, 0x19, 0x55, 0xff, 0xfe, 0x27, 0x27, 0xd0,
0x02, 0x01, 0x00, 0x19, 0x55, 0x27, 0x27, 0xd0
};
#if true
payload [2] = 0;
payload [3] = 0;
#endif
ushort checksum = ICMPchecksum(src_ip, dst_ip, length, next, payload);
Console.WriteLine (checksum.ToString ("X"));
Console.ReadKey ();
}
public static ushort BitConverterToUInt16 (byte [] value, int startIndex)
{
#if false
return System.BitConverter.ToUInt16 (value, startIndex);
#else
return System.BitConverter.ToUInt16 (value.Reverse ().ToArray (), value.Length - sizeof (UInt16) - startIndex);
#endif
}
static ushort ICMPchecksum (byte [] src_ip, byte [] dst_ip, byte [] length, byte [] next, byte [] payload)
{
uint checksum = 0;
//length of byte fields
Console.WriteLine ("src_ip: " + src_ip.Length + " dst_ip: " + dst_ip.Length + " length: " + length.Length + " next_header: " + next.Length + " payload: " + payload.Length);
//display all fields, which will be used for checksum calculation
Console.WriteLine (BitConverter.ToString (src_ip));
Console.WriteLine (BitConverter.ToString (dst_ip));
Console.WriteLine (BitConverter.ToString (length));
Console.WriteLine (BitConverter.ToString (next));
Console.WriteLine (BitConverter.ToString (payload));
//ADDS SOURCE IPv6 address
checksum += BitConverterToUInt16 (src_ip, 0);
checksum += BitConverterToUInt16 (src_ip, 2);
checksum += BitConverterToUInt16 (src_ip, 4);
checksum += BitConverterToUInt16 (src_ip, 6);
checksum += BitConverterToUInt16 (src_ip, 8);
checksum += BitConverterToUInt16 (src_ip, 10);
checksum += BitConverterToUInt16 (src_ip, 12);
checksum += BitConverterToUInt16 (src_ip, 14);
//ADDS DEST IPv6 address
checksum += BitConverterToUInt16 (dst_ip, 0);
checksum += BitConverterToUInt16 (dst_ip, 2);
checksum += BitConverterToUInt16 (dst_ip, 4);
checksum += BitConverterToUInt16 (dst_ip, 6);
checksum += BitConverterToUInt16 (dst_ip, 8);
checksum += BitConverterToUInt16 (dst_ip, 10);
checksum += BitConverterToUInt16 (dst_ip, 12);
checksum += BitConverterToUInt16 (dst_ip, 14);
//ADDS LENGTH OF ICMPv6 packet
checksum += BitConverterToUInt16 (length, 0);
checksum += BitConverterToUInt16 (length, 2);
//ADDS NEXT HEADER ID = 58
checksum += BitConverterToUInt16 (next, 0);
checksum += BitConverterToUInt16 (next, 2);
//ADDS WHOLE ICMPv6 message
for (int i = 0; i < payload.Length; i = i + 2) {
Console.WriteLine (i);
checksum += BitConverterToUInt16 (payload, i);
}
checksum += (ushort)(checksum >> 16);
return (ushort)~checksum;
}
}

Categories

Resources