How to group bytes to message in an reactive way - c#

Assume we have an output-only Stream, which outputs bytes. The data from Stream are serialized messages, each message always starts with a byte sequence (0xAA, 0xBB, 0xCC), but the length of the message is unknown.
Currently, I created an Observable and emit each byte on the stream, then subscribe to this observable, buffer every emission, find the byte sequence, and then emit the buffer. Something like
List<byte> buffer = new List<byte>();
dataStream.subscribe(b => {
buffer.add(b);
int[] idx = SearchSequence(buffer);
if(idx.Length < 2){
// TODO: wait for more data
}
else{
messageStream.onNext(buffer.GetRange(idx[0], idx[1]));
// TODO: remove them from buffer
}
})
Is there any more elegant way to solve this issue? There are two concerns as far as I know of:
The length of message is not fixed, which invalidate Observable.buffer()
At the time when dataStream (or other observable derived from it) is subscribed, the output of Stream may in the middle of a message.
UPDATE:
How do you detect the end of a message?
There is not gap between messages, the messages are comes right next to each other. Thus, the starting sequence of a message (0xAA, 0xBB, 0xCC) is also the ending sequence of the previous message
What does your input observable look like?
My current code is like:
Observer<byte> ob = null;
var dataStream = Observable.Create<byte>(o => ob = o);
while(true){
ob.OnNext(ms.ReadByte());
}
What do you want your output observable to look like?
A observable which emits message
Observable<byte[]>

I don't know how elegant this is, but maybe it will get you (or others) started. I'm assuming you want the (0xAA, 0xBB, 0xCC) header excluded from the messages:
var s = new Subject<byte>();
IObservable<byte[]> results = s.Publish(_s => Observable.When(_s
.And(_s.Skip(1))
.And(_s.Skip(2))
.Then((a, b, c) => (a, b, c))
))
.Publish(_t => _t
.Buffer(_t.Where(t => t.a == 0xAA && t.b == 0xBB && t.c == 0xCC))
.Select(l => (l[l.Count - 1].a == 0xAA && l[l.Count - 1].b == 0xBB && l[l.Count - 1].c == 0xCC
? l.Take(l.Count - 3)
: l
)
.Select(e => e.c)
.ToArray()
)
.Skip(1)
)
;
Explanation:
We first use And/Then/When to do a double zip, so the stream (0xAA, 0xBB, 0xCC, 0x01, 0x02, 0x03, 0xAA, 0xBB, 0xCC, 0x01, 0x02, 0x03) becomes a stream of tuples that look like this:
(0xAA, 0xBB, 0xCC)
(0xBB, 0xCC, 0x01)
(0xCC, 0x01, 0x02)
(0x01, 0x02, 0x03)
(0x02, 0x03, 0xAA)
(0x03, 0xAA, 0xBB)
(0xAA, 0xBB, 0xCC)
(0xBB, 0xCC, 0x01)
(0xCC, 0x01, 0x02)
(0x01, 0x02, 0x03)
We then use .Where to sniff out when you have a tuple that looks like (0xAA, 0xBB, 0xCC), and use that as a buffer boundary.
Once you have your buffer boundaries, they actually cut off after the message starter occurs, so you end up for our sample stream of two messages, you end up with three lists of tuples:
List 1: (0xAA, 0xBB, 0xCC)
List 2: (0xBB, 0xCC, 0x01)
(0xCC, 0x01, 0x02)
(0x01, 0x02, 0x03)
(0x02, 0x03, 0xAA)
(0x03, 0xAA, 0xBB)
(0xAA, 0xBB, 0xCC)
List 3: (0xBB, 0xCC, 0x01)
(0xCC, 0x01, 0x02)
(0x01, 0x02, 0x03)
The bytes we want in each list are basically the third column, but if we want to exclude the message-introduction from the messages, then we have to do some cleanup: We have to exclude the last three elements from list 2 (and all other 'middle' lists), we have to remove list 1, and we have to preserve list 3. Removing the first list is accomplished by the .Skip(1) at the end. Stripping out the last three elements from middle lists is done by checking to see if the last element in the list is (0xAA, 0xBB, 0xCC), and if it is, taking all elements except the last three.
Given all that, I hope there's a better way to do this.
Here's some runner code:
results.Dump(); //Linqpad
s.OnNext(0xAA);
s.OnNext(0xBB);
s.OnNext(0xCC);
s.OnNext(0x01);
s.OnNext(0x02);
s.OnNext(0x03);
s.OnNext(0xAA);
s.OnNext(0xBB);
s.OnNext(0xCC);
s.OnNext(0xAA);
s.OnNext(0xBB);
s.OnNext(0xCC);
s.OnNext(0x01);
s.OnNext(0x02);
s.OnNext(0x03);
s.OnNext(0xAA);
s.OnNext(0xBB);
s.OnNext(0xAA);
s.OnNext(0xBB);
s.OnNext(0xCC);
s.OnNext(0xCC);
s.OnNext(0xAA);
s.OnNext(0xBB);
s.OnNext(0x04);
s.OnNext(0x05);
s.OnNext(0x06);
s.OnNext(0x07);
s.OnCompleted();
Output:
01 02 03
01 02 03 AA BB
CC AA BB 04 05 06 07

Here is a custom Window operator that splits a sequence to sub-sequences, according to the supplied separator array. It is based on the built-in Window operator that accepts a IObservable<TWindowBoundary> windowBoundaries parameter.
/// <summary>Projects each element of an observable sequence into non-overlapping
/// windows that are separated by the provided separator.</summary>
public static IObservable<IObservable<T>> Window<T>(
this IObservable<T> source,
T[] separator,
IEqualityComparer<T> comparer = null)
{
return Observable.Defer(() =>
{
var boundaries = new Subject<Unit>();
var queue = new Queue<T>(separator.Length);
return source
.Do(x =>
{
if (queue.Count == separator.Length) queue.Dequeue();
queue.Enqueue(x);
if (queue.SequenceEqual(separator, comparer))
{
queue.Clear();
boundaries.OnNext(default);
}
})
.Concat(Observable.Repeat(default(T), separator.Length - 1))
.SkipLast(separator.Length - 1)
.Window(boundaries)
.Select((window, i) => i == 0 ? window : window.Skip(separator.Length));
});
}
A Subject is used for notifying the Window operator that a new boundary has been detected. The detection mechanism includes a Queue that holds the last emitted elements. This queue is compared to the separator every time a new element is emitted. The Window operator is lagging intentionally behind the detection mechanism by separator.Length - 1 elements, so that the resulting windows are aligned properly.
Usage example:
IObservable<byte> dataStream = GetDataStream();
IObservable<byte[]> messageStream = dataStream
.Window(new byte[] { 0xAA, 0xBB, 0xCC })
.SelectMany(window => window.ToArray());

Related

Turn random bytes to .NET decimal 0..1 fractional range

I'm playing with a TRNG usb device, and successfully turning the random bytes into various usable data in C#.
I'd like to create a .NET decimal value between 0..1 (ex: 0.37327) directly from the bytes using binary reader (or other direct bytes -> decimal) method.
// assume: byte[] random_data of appropriate length, patterned for a 0..1 range
decimal value = new BinaryReader(new MemoryStream(random_data)).ReadDecimal();
I was looking for the byte format of decimal, but it looks like it may not be a standard?
How do I convert byte values into decimals?
.NET decimal cross-platform standard
This is for hobby work, so it's acceptable to me to use something that might change in future.
I've looked at the bytes generated for sample input decimal values - I see negative and precision flags (1Ch max in last int32?), but the min/max data values dumping compiler constants are stumping me a bit, and I'm generate above zero values or invalid values:
fractional value nearest 1 (.99...9): FFFFFF0F 6102253E 5ECE4F20 00001C00
fractional value nearest 0 (.00...1): 01000000 00000000 00000000 00001C00
Can you help me onto the correct path for generating a full fractional 0..1 range?
Edit with "final" code, thanks to everyone!
Here's the C# code I ended up with to create an unbiased random decimal of range [0..1] from a suitable random byte stream (like a TRNG device, www.Random.org, or CSPRNG algorithm). Generated values look good to the eyeball, boundary tests pass, and as long as I avoided embarassing typos and copy/paste bugs, this should be usable as-is.
Thanks for the help and interesting discussions!
private decimal RandomDecimalRange01()
{
// 96 bits of random data; we'll use 94 bits to directly map decimal's max precision 0..1 range
byte[] data = new byte[12];
decimal value = 0;
// loop until valid value is generated, discarding invalids values. Mostly controlled by top 2 bits: 11 is always invalid, 00 or 01, is always valid, 10 has valid and invalid ranges. Odds make loop generally find value in one or a few iterations.
while (true)
{
// Acquire random bytes from random source (like TRNG device or CSPRNG api)
if (!trng.GetBytes(data))
{
throw new Exception("Failed to aquire random bytes from source");
}
else
{
// Read 94 random bits (pull 96 bits, discard 2)
BinaryReader reader = new BinaryReader(new MemoryStream(data));
int low = reader.ReadInt32();
int mid = reader.ReadInt32();
int high = reader.ReadInt32() & 0x3FFFFFFF; // don't consume upper 2 random bits - out of range
// Discard invalid values and reloop (interpret special invalid value as 1)
if (high > 542101086)
{
continue;
}
else if (high == 542101086)
{
if (mid > 1042612833)
{
continue;
}
else if (mid == 1042612833)
{
if (low > 268435455)
{
// Special override to generate 1 value for inclusive [0..1] range - interpret the smallest invalid value as 1. Remove code for exclusive range [0..1)
if (low == 268435456)
{
value = 1m; // return 1.0
break;
}
continue;
}
}
}
// return random decimal created from parts - positive, maximum precision 28 (1C) scale
value = new decimal(low, mid, high, false, 28);
break;
}
}
return value;
}
Sample generated values running TrueRNGPro TRNG device bytes through algorithm
0.8086691474438979082567747041
0.4268035919422123276460607186
0.7758625805098585303332549015
0.0701321080502462116399370731
0.3127190777525873850928167447
0.6022236739048965325585049764
0.1244605652187291191393036867
Tests around interesting boundary values
// test databyte values for max & min ranges
new byte[] { 0x01, 0x00, 0x00, 0x10, 0x61, 0x02, 0x25, 0x3E, 0x5E, 0xCE, 0x4F, 0x20 }; // boundary: 1 too large for algorithm, will be discarded
new byte[] { 0x00, 0x00, 0x00, 0x10, 0x61, 0x02, 0x25, 0x3E, 0x5E, 0xCE, 0x4F, 0x20 }; // boundary: special 1 more than largest valid .99999..., interpret as 1 value
new byte[] { 0xFF, 0xFF, 0xFF, 0x0F, 0x61, 0x02, 0x25, 0x3E, 0x5E, 0xCE, 0x4F, 0x20 }; // boundary: largest valid value .9999...
new byte[] { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }; // boundary: smallest valid value, should be 0
new byte[] { 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }; // boundary: 1 more than smallest valid value, should be .000...1
From https://msdn.microsoft.com/en-us/library/system.decimal.getbits(v=vs.110).aspx
The binary representation of a Decimal number consists of a 1-bit
sign, a 96-bit integer number, and a scaling factor used to divide the
integer number and specify what portion of it is a decimal fraction.
The scaling factor is implicitly the number 10, raised to an exponent
ranging from 0 to 28.
The return value is a four-element array of 32-bit signed integers.
The first, second, and third elements of the returned array contain
the low, middle, and high 32 bits of the 96-bit integer number.
The fourth element of the returned array contains the scale factor and
sign. It consists of the following parts:
Bits 0 to 15, the lower word, are unused and must be zero.
Bits 16 to 23 must contain an exponent between 0 and 28, which
indicates the power of 10 to divide the integer number.
Bits 24 to 30 are unused and must be zero.
Bit 31 contains the sign: 0 mean positive, and 1 means negative.
You are looking for 0 <= 96bitinteger / 10exponent <= 1
Multiplying through, this is the same as 0 <= 96bitinteger <= 10^exponent
This would enumerate every possibile pair of 96bitinteger and exponent that would produce a decimal value between 0 and 1 (assuming the sign bit is set to 0).
for (int exponent=0; exponent<=28; exponent++) {
BigInteger max = BigInteger.Pow(10, exponent);
for (int i = 0; i <= max; i++) {
var fmt = "96bitinteger: {0}, exponent: {1}";
Console.WriteLine(String.Format(fmt, i, exponent));
}
}
Using exponent=28
1028 in hex is 204fce5e 3e250261 10000000. So once you place the 32-bit numbers according to the docs, and then create fhe final 32-bits accounting for the fact for some reason when they say bit 0 they mean the highest order bit, it isn't too hard to construct the decimal number 1 from bytes.
int[] data = new int[] { 0x10000000, 0x3e250261, 0x204fce5e, 0x1C0000 };
var random_data = data.SelectMany(BitConverter.GetBytes).ToArray();
decimal value = new BinaryReader(new MemoryStream(random_data)).ReadDecimal();
Console.WriteLine(value);
Consider the more general new int[] { a, b, c, 0x1C0000 }. The constraints on a b and c, for creating a decimal number 0 <= d <= 1 are
if c < 0x204fce5e:
a can be anything
b can be anything
elif c = 0x204fce5e:
if b < 0x3e250261:
a can be anything
elif b = 0x3e250261
constrain a <= 10000000
b can not be greater than 0x3e250261
c can not be greater than 0x204fce5e.
Do you want uniform distribution between 0 and 1 (including both ends)?
Do the original data is also uniform at the byte level?
How much precision do you really need?
Do you need decimal or float/double would do?
If you use decimal, I don't think you would get uniform distribution as you cannot use whole range of all bytes.
With floating points, I think it might be easier to do bits manipulations if you create a number between 1 and 2 and then subtract 1. However, you would never get exactly 1.0.

How to write hex values in byte array?

I have a hex value of 0x1047F71 and I want to put in byte array of 4 bytes. Is this the right way to do it:
byte[] sync_welcome_sent = new byte[4] { 0x10, 0x47, 0xF7, 0x01 };
or
byte[] sync_welcome_sent = new byte[4] { 0x01, 0x04, 0x7F, 0x71 };
I would appreciate any help.
If you want to be compatible with Intel little-endian, the answer is "None of the above", because the answer would be "71h, 7fh, 04h, 01h".
For big-endian, the second answer above is correct: "01h, 04h, 7fh, 71h".
You can get the bytes with the following code:
uint test = 0x1047F71;
var bytes = BitConverter.GetBytes(test);
If you want big-endian, you can just reverse the bytes using Linq like so:
var bytes = BitConverter.GetBytes(test).Reverse();
However, if you are running the code on a Big Endian system, reversing the bytes will not be necessary, since BitConverter.GetBytes()will return them as big endian on a big endian system.
Therefore you should write the code as follows:
uint test = 0x1047F71;
var bytes = BitConverter.GetBytes(test);
if (BitConverter.IsLittleEndian)
bytes = bytes.Reverse().ToArray();
// now bytes[] are big-endian no matter what system the code is running on.

What is the Standard for Handling Endianness With Interop?

I created a very simple DLL for Speck in (granted, probably inefficient) ASM. I connected to it in C# using InteropServices.
When I tested this crypto with the test vectors provided in the paper describing the algorithm, I found that the only way to get them to come out right was to "flip" the key and the plain text, and then to "flip" the crypto at the end for a match. So an endianness issue I guess. I have seen the same, for example, between a reference implementation of Serpent and TrueCrypt's version -- they produce the same result only with the bytes in the reverse order.
I will post my assembly code and my C# code for reference, though it may not be critical to see the code in order to understand my question. In the C# code is a click event handler that checks the DLL for consistency with the test vectors. As you can also see there, the program has to do a lot of array flipping in that handler to get the match.
So the question I have been working towards is this. Should I "flip" those arrays inside the DLL to account for endianness? Or should I leave it to the caller (also me, but C# side)? Or am I making mountains out of molehills and I should just ignore endianness at this point? I am not planning to sell the silly thing, so there is no worry about compatibility issues, but I am a stickler for doing things right, so I am hoping you all can guide me on the best practice here if there is one.
ASM:
.code ; the beginning of the code
; section
WinMainCRTStartup proc h:DWORD, r:DWORD, u:DWORD ; the dll entry point
mov rax, 1 ; if eax is 0, the dll won't
; start
ret ; return
WinMainCRTStartup Endp ; end of the dll entry
_DllMainCRTStartup proc h:DWORD, r:DWORD, u:DWORD ; the dll entry point
mov rax, 1 ; if eax is 0, the dll won't
; start
ret ; return
_DllMainCRTStartup Endp
SpeckEncrypt proc plaintText:QWORD, cipherText:QWORD, Key:QWORD
; Pass in 3 addresses pointing to the base of the plainText, cipherText, and Key arrays
; These come in as RCX, RDX, and R8, respectively
; I will use These, RAX, and R9 through R15 for my working space. Will do 128 bit block, 128 bit key sizes, but they will fit nicely in 64 bit registers
; simple prologue, pushing ebp and ebx and the R# registers, and moving the value of esp into ebp for the duration of the proc
push rbp
mov rbp,rsp
push rbx
push R9
push R10
push R11
push R12
push R13
push R14
push R15
; Move data into the registers for processing
mov r9,[rcx] ; rcx holds the memory location of the first 64 bits of plainText. Move this into R9. This is plainText[0]
mov r10,[rcx+8] ; put next 64 bits into R10. This is plainText[1]
;NOTE that the address of the cipherText is in RDX but we will fill r11 and r12 with values pointed at by RCX. This is per the algorithm. We will use RDX to output the final bytes
mov r11,[rcx] ; cipherText[0] = plainText[0]
mov r12,[rcx+8] ; cipherText[1] = plainText[1]
mov r13, [r8] ;First 64 bits of key. This is Key[0]
mov r14, [r8+8] ; Next 64 bits of key. This is Key[1]
push rcx ; I could get away without this and loop in another register, but I want to count my loop in rcx so I free it up for that
mov rcx, 0 ; going to count up from here to 32. Would count down but the algorithm uses the counter value in one permutation, so going to count up
EncryptRoundFunction:
ror r12,8
add r12,r11
xor r12,r13
rol r11,3
xor r11,r12
ror r14,8
add r14,r13
xor r14,rcx
rol r13,3
xor r13,r14
inc rcx
cmp rcx, 32
jne EncryptRoundFunction
pop rcx
; Move cipherText into memory pointed at by RDX. We won't bother copying the Key or plainText back out
mov [rdx],r11
mov [rdx+8],r12
; Now the epilogue, returning values from the stack into non-volatile registers.
pop R15
pop R14
pop R13
pop R12
pop R11
pop R10
pop R9
pop rbx
pop rbp
ret ; return eax
SpeckEncrypt endp ; end of the function
SpeckDecrypt proc cipherText:QWORD, plainText:QWORD, Key:QWORD
; Pass in 3 addresses pointing to the base of the cipherText, plainText, and Key arrays
; These come in as RCX, RDX, and R8, respectively
; I will use These, RAX, and R9 through R15 for my working space. Will do 128 bit block, 128 bit key sizes, but they will fit nicely in 64 bit registers
; simple prologue, pushing ebp and ebx and the R# registers, and moving the value of esp into ebp for the duration of the proc
push rbp
mov rbp,rsp
push rbx
push R9
push R10
push R11
push R12
push R13
push R14
push R15
; Move data into the registers for processing
mov r9,[rcx] ; rcx holds the memory location of the first 64 bits of cipherText. Move this into R9. This is cipherText[0]
mov r10,[rcx+8] ; put next 64 bits into R10. This is cipherText[1]
;NOTE that the address of the plainText is in RDX but we will fill r11 and r12 with values pointed at by RCX. This is per the algorithm. We will use RDX to output the final bytes
mov r11,[rcx] ; plainText[0] = cipherText[0]
mov r12,[rcx+8] ; plainText[1] = cipherText[1]
mov r13, [r8] ;First 64 bits of key. This is Key[0]
mov r14, [r8+8] ; Next 64 bits of key. This is Key[1]
push rcx ; I could get away without this and loop in another register, but I want to count my loop in rcx so I free it up for that
mov rcx, 0 ; We will count up while making the round keys
DecryptMakeRoundKeys:
; On encrypt we could make each key just as we needed it. But here we need the keys in reverse order. To undo round 31 of encryption, for example, we need round key 31.
; So we will make them all and push them on the stack, pop them off again as we need them in the main DecryptRoundFunction
; I should pull this off and call it for encrypt and decrypt to save space, but for now will have it separate
; push r13 at the beginning of the process because we need a "raw" key by the time we reach decrypt round 0
; We will not push r14 because that half of the key is only used here in the round key generation function.
; We don't need it in the decrypt rounds
push r13
ror r14,8
add r14,r13
xor r14,rcx
rol r13,3
xor r13,r14
inc rcx
cmp rcx, 32
jne DecryptMakeRoundKeys
mov rcx, 32
DecryptRoundFunction:
dec rcx
pop r13
xor r11,r12
ror r11,3
xor r12,r13
sub r12,r11
rol r12,8
cmp rcx, 0
jne DecryptRoundFunction
pop rcx
; Move cipherText into memory pointed at by RDX. We won't bother copying the Key or plainText back out
mov [rdx],r11
mov [rdx+8],r12
; Now the epilogue, returning values from the stack into non-volatile registers.
pop R15
pop R14
pop R13
pop R12
pop R11
pop R10
pop R9
pop rbx
pop rbp
ret ; return eax
SpeckDecrypt endp ; end of the function
End ; end of the dll
And the C#:
using System;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading;
using System.Windows.Forms;
namespace SpeckDLLTest
{
public partial class Form1 : Form
{
byte[] key = { 0x0f, 0x0e, 0x0d, 0x0c, 0x0b, 0x0a, 0x09, 0x08, 0x07, 0x06, 0x05, 0x04, 0x03, 0x02, 0x01, 0x00 };
public Form1()
{
InitializeComponent();
Array.Reverse(key);
}
private void richTextBox1_TextChanged(object sender, EventArgs e)
{
textBox1.Text = richTextBox1.Text.Length.ToString();
if (richTextBox1.Text != "")
{
byte[] plainText = ASCIIEncoding.ASCII.GetBytes(richTextBox1.Text);
byte[] cipherText = new byte[plainText.Length];
Thread t = new Thread(() =>
{
cipherText = Encrypt(plainText);
BeginInvoke(new Action(() => richTextBox2.Text = Convert.ToBase64String(cipherText)));
});
t.Start();
t.Join();
t.Abort();
byte[] plainAgain = new byte[cipherText.Length];
t = new Thread(() =>
{
plainAgain = Decrypt(cipherText);
BeginInvoke(new Action(() => richTextBox3.Text = ASCIIEncoding.ASCII.GetString(plainAgain)));
});
t.Start();
t.Join();
t.Abort();
}
else
{
richTextBox2.Text = "";
richTextBox3.Text = "";
}
}
private byte[] Decrypt(byte[] cipherText)
{
int blockCount = cipherText.Length / 16;
if (cipherText.Length % 16 != 0) blockCount++;
Array.Resize(ref cipherText, blockCount * 16);
byte[] plainText = new byte[cipherText.Length];
unsafe
{
fixed (byte* plaintextPointer = plainText, ciphertextPointer = cipherText, keyPointer = key)
{
for (int i = 0; i < blockCount; i++)
{
for (int j = 0; j < 1; j++)
{
UnsafeMethods.SpeckDecrypt(ciphertextPointer + i * 16, plaintextPointer + i * 16, keyPointer);
}
}
}
}
return plainText;
}
private byte[] Encrypt(byte[] plainText)
{
int blockCount = plainText.Length / 16;
if (plainText.Length % 16 != 0) blockCount++;
Array.Resize(ref plainText, blockCount * 16);
byte[] cipherText = new byte[plainText.Length];
unsafe
{
fixed (byte* plaintextPointer = plainText, ciphertextPointer = cipherText, keyPointer = key)
{
for (int i = 0; i < blockCount; i++)
{
for (int j = 0; j < 1; j++)
{
UnsafeMethods.SpeckEncrypt(plaintextPointer + i * 16, ciphertextPointer + i * 16, keyPointer);
}
}
}
}
return cipherText;
}
private void button1_Click(object sender, EventArgs e)
{
byte[] plainText = { 0x6c, 0x61, 0x76, 0x69, 0x75, 0x71, 0x65, 0x20, 0x74, 0x69, 0x20, 0x65, 0x64, 0x61, 0x6d, 0x20 };
byte[] key = { 0x0f, 0x0e, 0x0d, 0x0c, 0x0b, 0x0a, 0x09, 0x08, 0x07, 0x06, 0x05, 0x04, 0x03, 0x02, 0x01, 0x00 };
byte[] testVector = { 0xa6, 0x5d, 0x98, 0x51, 0x79, 0x78, 0x32, 0x65, 0x78, 0x60, 0xfe, 0xdf, 0x5c, 0x57, 0x0d, 0x18 };
Array.Reverse(key);
Array.Reverse(plainText);
byte[] cipherText = new byte[16];
unsafe
{
fixed (byte* plaintextPointer = plainText, ciphertextPointer = cipherText, keyPointer = key)
{
UnsafeMethods.SpeckEncrypt(plaintextPointer, ciphertextPointer, keyPointer);
Array.Reverse(cipherText);
bool testBool = true;
for (int i = 0; i < cipherText.Length; i++)
{
if (testVector[i] != cipherText[i]) testBool = false;
}
if (testBool == false) MessageBox.Show("Failed!");
else MessageBox.Show("Passed!");
}
}
}
}
public static class UnsafeMethods
{
[DllImport("Speck.dll")]
unsafe public extern static void SpeckEncrypt(byte* plainText, byte* cipherText, byte* Key);
[DllImport("Speck.dll")]
unsafe public extern static void SpeckDecrypt(byte* cipherText, byte* plainText, byte* Key);
}
}
Whether someone might like it or not, the de facto standard for byte order when it comes to networking and cryptography is big-endian (most significant byte first — the “natural” order). This applies not only to serialization of data for inter-system exchange, but to intra-system API as well and for any other case where caller is not supposed to be aware of callee internals. This convention does not have anything to do with endianness of particular hardware and popularity of such hardware. It just sets the default format for exchanged data, so that both lower-level and higher-level programs may pass data around without regard to their degree of awareness of what this data contains and how it is processed.
However, if the caller is supposed to be tightly coupled with the callee, it may be more convenient and performance-wise to pass the data in a more preprocessed form, especially if some of that data remains constant across invocations. For example, if we are dealing with asymmetric cryptography, it may be easier and faster to call the core functions with all data already translated to big integers, and for those we may prefer little-endian digit order (a “digit” or a “limb” is usually a half of largest available register) even on a big-endian byte order hardware — simply because such an order of digits is more useful for arbitrary-precision math library. But those details should not be visible to the outside world — for anyone else, we are accepting and returning big-endian bytestream.
Regarding your specific task.
As #RossRidge already pointed out, you are probably very wrong if your are simply flipping entire arrays, — you should swap bytes (BSWAP) in particular pieces being processed rather than inverting the order of those pieces besides that.
Chances are high that you are very overestimating your ability to write efficient machine code: for example, you don't interleave instructions with unrelated registers for better out-of-order execution, your loop is not aligned, you use counter increase to N instead of decrease to zero. Of course, that code will still be 10x faster than .Net anyway, but I strongly recommend you to write an implementation in C and benchmark — to get amazed of how good a compiler (MSVC, GCC) may be at optimizing even a straight-though written program (believe me, I once committed the same mistake when trying to accomplish the same task). If performance is not a big issue, do not mess with unmanaged code at all, — because it is just an external non-portable dependency that increases required trust level for you .Net application.
Use .Net functions dealing with bytes with caution, because they are very inconsistent with regard to endianness: BitConverter uses host byte order, StreamReader always sticks to little-endian, and String is all about the encoding given (of all UTF encodings, only UTF-8 is endian-agnostic).
That are the issues I noticed at first glance. There may be more of them.

Byte array initialisation syntax - specify first 20 bits and pad rest with 0s

My goal is to get a 64bit value hence a byte array of size 8. However my problem is that I want to set the first 20 bits myself and then have the rest to be 0s. Can this be done with the shorthand byte array initialisation?
E.g. if I wanted all 0s I would say:
byte[] test = new byte[] {0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
What I've thought about/tried:
So each hexadecimal digit corresponds to 4 binary digits. Hence, if I want to specify the first 20bits, then I specify the first 5 hexadecimal digits? But I'm not sure of how to do this:
byte[] test = new byte[] {0xAF, 0x17, 0x10, 0x00, 0x00, 0x00, 0x00, 0x00};
That would mean that I've specified the first 24 bits right? And not 20.
I could use a BitArray and do it that way but I'm just wondering whether it can be done in the above way.
How about:
byte byte1 = 0xFF;
byte byte2 = 0xFF;
byte byte3 = 0xFF;
// 8bits 8bits 4bits : total = 20 bits
// 11111111 11111111 11110000
byte[] test = new byte[] { byte1, byte2, (byte)(byte3 & 0xF0), 0x00, 0x00, 0x00, 0x00, 0x00 };
You can write your bytes backward, and use BitConverter.GetBytes(long):
var bytes = BitConverter.GetBytes(0x117AF);
Demo.
Since each hex digit corresponds to a single four-bit nibble, you can initialize data in "increments" of four bits. However, the data written in reverse will be almost certainly less clear to human readers of your code.

Execute .NET IL code in C#

Is there any way to execute an array of IL codes in C# like shell codes in C/C++?
I want to create a method, convert it to IL code, obfuscate it and store in an array of bytes and finally want to execute it decrypt the array content and execute IL code.
For example this is my C# code:
static int MyMethod(string A, int B)
{
B += 10;
if (A.Equals("A"))
B = 0;
return B;
}
Now I convert it to IL code :
private static int MyMethod(string A, int B)
{
locals: int local_0,
bool local_1
/* 0000025C 00 */ nop
/* 0000025D 03 */ ldarg_1 // int B
/* 0000025E 1F 0A */ ldc_i4_s 10
/* 00000260 58 */ add
/* 00000261 10 01 */ starg_s arg_1 // int B
/* 00000263 02 */ ldarg_0 // string A
/* 00000264 72 01 00 00 70 */ ldstr "A"
/* 00000269 6F 11 00 00 0A */ callvirt System.String::Equals(string) // returns bool
/* 0000026E 16 */ ldc_i4_0
/* 0000026F FE 01 */ ceq
/* 00000271 0B */ stloc_1 // bool local_1
/* 00000272 07 */ ldloc_1 // bool local_1
/* 00000273 2D 03 */ brtrue_s loc_28
/* 00000275 16 */ ldc_i4_0
/* 00000276 10 01 */ starg_s arg_1 // int B
loc_28:
/* 00000278 03 */ ldarg_1 // int B
/* 00000279 0A */ stloc_0 // int local_0
/* 0000027A 2B 00 */ br_s loc_32
loc_32:
/* 0000027C 06 */ ldloc_0 // int local_0
/* 0000027D 2A */ ret
}
And finally this is a byte array :
private byte[] ilcode =
{
0x00, 0x03, 0x1F, 0x0A, 0x58, 0x10, 0x01, 0x02, 0x72, 0x01, 0x00, 0x00, 0x70, 0x6F, 0x11, 0x00, 0x0, 0x0A, 0x16,
0xFE, 0x01, 0x0B, 0x07, 0x2D, 0x03, 0x16, 0x10, 0x01, 0x03, 0x0A, 0x2B, 0x00, 0x06, 0x2A
};
You will need to use the infrastructure in the System.Reflection.Emit namespace. Specifically, you should look at the docs for MethodBuilder.CreateMethodBody which takes an array of bytes representing MSIL instructions. There is a full example there, but below is short snippet of its use. Here, I will create a delegate to the dynamic method as well.
I will note that this is only supported in a very limited way, which is called out in the docs:
This is currently not fully supported. The user cannot supply the location of token fix ups and exception handlers.
The issue is that metadata tokens used in IL to reference types, methods, string literals, etc are resolved at the module level. Thus IL is not completely portable in the sense that you can't take arbitrary, raw IL from a method in one module and just drop it into another method in another module. You need the proper metadata tokens in the new module. However, if you know that your IL contains no metadata tokens you can do it, but this severely limits what you can do with this. (HT: svick, Simon Svensson)
class Program
{
static void Main(string[] args)
{
// opcodes for pushing two arguments to the stack, adding, and returning the result.
byte[] ilcodes = { 0x02, 0x03, 0x58, 0x2A };
var method = CreateFromILBytes(ilcodes);
Console.WriteLine(method(2, 3));
}
private static Func<int, int, int> CreateFromILBytes(byte[] bytes)
{
var asmName = new AssemblyName("DynamicAssembly");
var asmBuilder = AppDomain.CurrentDomain.DefineDynamicAssembly(asmName, AssemblyBuilderAccess.RunAndSave);
var module = asmBuilder.DefineDynamicModule("DynamicModule");
var typeBuilder = module.DefineType("DynamicType");
var method = typeBuilder.DefineMethod("DynamicMethod",
MethodAttributes.Public | MethodAttributes.Static,
typeof(int),
new[] { typeof(int), typeof(int) });
method.CreateMethodBody(bytes, bytes.Length);
var type = typeBuilder.CreateType();
return (Func<int, int, int>)type.GetMethod("DynamicMethod").CreateDelegate(typeof(Func<int, int, int>));
}
}
Note the use of the RunAndSave option. This will save the dynamic assembly to disk in a temporary location. It may be more desirable to use RunAndCollect which will generate the assembly in memory only and allow it to be collected later when all references to it are dead. However, there are some caveats to so called collectible assemblies, detailed here.

Categories

Resources