Develop Reference

C# - Reading Sequence of Hex Bytes in Binary

So I've been googling & googling for this, but I can't find a solution for my case. I could find things about byte arrays. but I hope there's also a simpler solution for my case.
Maybe it's just me using the wrong search terms, don't know.
Anyways, I already have a kinda working code which is:
static void Main(string[] args)
{
// Open the file to search in
BinaryReader br = new BinaryReader(File.OpenRead("D:/Users/Joey/Desktop/prod"));
for (int i = 0; i <= br.BaseStream.Length; i++)
{
// Search the file for the given byte
if (br.BaseStream.ReadByte() == (byte)0xC0)
{
Console.WriteLine("Found the byte at offset " + i); //write to the console on which offset it has been found
}
}
}
This example works.
However, I need it to be able to search for more than just one byte.
For example: C0035FD6
I feel like I'm missing something so simple, but I just can't seem to figure it out.
If anyone has gotten a solution for me, that would be great :D

You can use this extension to search for AOB:
public static class StreamExtensions
{
public static IEnumerable<long> ScanAOB(this Stream stream, params byte[] aob)
{
long position;
byte[] buffer = new byte[aob.Length - 1];
while ((position = stream.Position) < stream.Length)
{
if (stream.ReadByte() != aob[0]) continue;
if (stream.Read(buffer, 0, aob.Length - 1) == 0) continue;
if (buffer.SequenceEqual(aob.Skip(1)))
{
yield return position;
}
}
}
public static IEnumerable<long> ScanAOB(this Stream stream, params byte?[] aob)
{
long position;
byte[] buffer = new byte[aob.Length - 1];
while ((position = stream.Position) < stream.Length)
{
if (stream.ReadByte() != aob[0]) continue;
if (stream.Read(buffer, 0, aob.Length - 1) == 0) continue;
if (buffer.Cast<byte?>().SequenceEqual(aob.Skip(1), new AobComparer()))
{
yield return position;
}
}
}
private class AobComparer : IEqualityComparer<byte?>
{
public bool Equals(byte? x, byte? y) => x == null || y == null || x == y;
public int GetHashCode(byte? obj) => obj?.GetHashCode() ?? 0;
}
}
Example:
void Main()
{
using (var stream = new MemoryStream(FakeData().ToArray()))
{
stream.ScanAOB(0x1, 0x2).Dump("Addresses of: 01 02");
stream.Position = 0;
stream.ScanAOB(0x03, 0x12).Dump("Addresses of: 03 12");
stream.Position = 0;
stream.ScanAOB(0x04, null, 0x06).Dump("Addresses of: 04 ?? 06");
}
}
// Define other methods and classes here
IEnumerable<byte> FakeData()
{
return Enumerable.Range(0, 2)
.SelectMany(_ => Enumerable.Range(0, 255))
.Select(x => (byte)x);
}

Give this a shot. You will need to verify the arrays are correct. In a binary stream, a byte array is just a collection of bytes starting at offset with count bytes as its size.
//here is where you initialize your array. you may need to tweak the values to match your byte range (array)
byte[] dataArray = new byte[9] { 0x93, 0x0E, 0x40, 0xF9, 0x53, 0x00, 0x00, 0xB5, 0xDE };
//here is where you initialize the NEW array you want to write where your matching array lives
byte[] newArray = new byte[9] { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
// Open the file to search in
BinaryReader br = new BinaryReader(File.OpenRead("D:/Users/Joey/Desktop/prod"));
for (int i = 0; i <= br.BaseStream.Length; i++)
{
// Search the file for the STARTING byte of my match
if (br.BaseStream.ReadByte() == (byte)0x93)
{
Console.WriteLine("Found the starting byte at offset " + i); //write to the console on which offset it has been found
byte[] tempArray = new byte[9];
tempArray = br.ReadBytes(9);
//now compare the arrays to see if you have a full match:
int matched = 0;
for (int j=0; j<tempArray.Length; j++)
{
if(tempArray[j] == dataArray[j])
{
matched++;
}
}
//if the arrays match, write your new values:
if(matched == tempArray.Length-1)
{
br.BaseStream.Write(newArray, i, 9);
break; //exit the loop when finished
}
}
}

Want to write to a file converting an integer list of Hex to Char

So I have a list full of integers. These integers are hexadecimals. I would like to convert this list to ASCII Chars. Once that is done I would like to write the ASCII chars to a file. Here is what I have so far:
public byte[] buffer;
public List<int> list= new List<int>(new int[3]);
list[0] = 5445535420; //AKA header[0] represents the hex integers for Test_ where _ is a space
list[1] = 0; // so the char would be null
list[2] = 4a4153; // would be JAS
System.IO.FileStream fs;
fs = new FileStream(filename, FileMode.OpenOrCreate);
if (fs.CanWrite)
{
for (int i=0;i<list.Count();i++)
{
buffer = Encoding.ASCII.GetBytes(list[i].ToString());
Convert.ToChar(header[i]);
fs.Write(buffer, 0, buffer.Length);
}
}

Would this work for you? Hope comments are self-explenatory
int intFromHexLiteral = 0x4a4153;
var hexString = intFromHexLiteral.ToString("X"); // "4A4153"
var hexCharsList = Split(hexString, 2).ToList(); // ["4A", "41", "53"]
var charsArray = hexCharsList
.Select(hexChar => Convert.ToInt32(hexChar, 16)) // [74, 65, 83]
.Select(i => (char) i) // ['J', 'A', 'S']
.ToArray();
var word = new string(charsArray); // "JAS"
private static IEnumerable<string> Split(string str, int chunkSize) =>
Enumerable.Range(0, str.Length / chunkSize)
.Select(i => str.Substring(i * chunkSize, chunkSize));

An integer does not equal the bytes of the ASCII characters. I.e 1010 is not 0x1010 in hex. In your case it would make more sense to use byte[] an write each hex character explicitly.
class Program
{
static void Main(string[] args)
{
List<byte[]> list = new List<byte[]>();
list.Add(new byte[]{0x54, 0x45, 0x53, 0x54, 0x20}); //AKA header[0] represents the hex integers for Test_ where _ is a space
list.Add(new byte[]{0x0}); // so the char would be null
list.Add(new byte[]{ 0x4a, 0x41, 0x53 }); // would be JAS
foreach (var b in list)
{
var chars = Encoding.ASCII.GetChars(b);
var s = new string(chars);
Console.WriteLine(s);
}
}
}

I see that you've selected an answer, but I wanted to show you this method of solving your problem dealing with your data strictly as numeric data. The values you stick into your List<int> are small enough to fit into a long so I changed it to a List<long>. If they ever become bigger than that, than this solution would not work.
See how I broke each long element, byte by byte, in reverse and stored the conversion into a StringBuilder before writing it to the screen. In your case, you would write to a file instead, but could use the same conversion method.
using System;
using System.Collections.Generic;
using System.Text;
public class Program
{
public static void Main(string[] args)
{
List<long> list = new List<long>(new long[3]);
list[0] = 0x5445535420; // AKA header[0] represents the hex integers for Test_ where _ is a space
list[1] = 0; // so the char would be null
list[2] = 0x4a4153; // would be JAS
for (int i = 0; i < list.Count; i++)
{
StringBuilder sb = new StringBuilder();
// Break down each element byte by byte in reverse
while (list[i] > 0)
{
// Anding against 0xFF to only have the least significant byte to convert into a char
sb.Insert(0, Convert.ToChar(list[i] & 0xFF));
list[i] >>= 8; // Remove the least significant byte
}
Console.WriteLine(sb);
}
}
}
Result:
TEST
JAS

Packing bytes manually to send on network

I have an object that has the following variables:
bool firstBool;
float firstFloat; (0.0 to 1.0)
float secondFloat (0.0 to 1.0)
int firstInt; (0 to 10,000)
I was using a ToString method to get a string that I can send over the network. Scaling up I have encountered issues with the amount of data this is taking up.
the string looks like this at the moment:
"false:1.0:1.0:10000" this is 19 characters at 2 bytes per so 38 bytes
I know that I can save on this size by manually storing the data in 4 bytes like this:
A|B|B|B|B|B|B|B
C|C|C|C|C|C|C|D
D|D|D|D|D|D|D|D
D|D|D|D|D|X|X|X
A = bool(0 or 1), B = int(0 to 128), C = int(0 to 128), D = int(0 to 16384), X = Leftover bits
I convert the float(0.0 to 1.0) to int(0 to 128) since I can rebuild them on the other end and the accuracy isn't super important.
I have been experimenting with BitArray and byte[] to convert the data into and out of the binary structure.
After some experiments I ended up with this serialization process(I know it needs to be cleaned up and optimized)
public byte[] Serialize() {
byte[] firstFloatBytes = BitConverter.GetBytes(Mathf.FloorToInt(firstFloat * 128)); //Convert the float to int from (0 to 128)
byte[] secondFloatBytes = BitConverter.GetBytes(Mathf.FloorToInt(secondFloat * 128)); //Convert the float to int from (0 to 128)
byte[] firstIntData = BitConverter.GetBytes(Mathf.FloorToInt(firstInt)); // Get the bytes for the int
BitArray data = new BitArray(32); // create the size 32 bitarray to hold all the data
int i = 0; // create the index value
data[i] = firstBool; // set the 0 bit
BitArray ffBits = new BitArray(firstFloatBytes);
for(i = 1; i < 8; i++) {
data[i] = ffBits[i-1]; // Set bits 1 to 7
}
BitArray sfBits = new BitArray(secondFloatBytes);
for(i = 8; i < 15; i++) {
data[i] = sfBits[i-8]; // Set bits 8 to 14
}
BitArray fiBits = new BitArray(firstIntData);
for(i = 15; i < 29; i++) {
data[i] = fiBits[i-15]; // Set bits 15 to 28
}
byte[] output = new byte[4]; // create a byte[] to hold the output
data.CopyTo(output,0); // Copy the bits to the byte[]
return output;
}
Getting the information back out of this structure is much more complicated than getting it into this form. I figure I can probably workout something using the bitwise operators and bitmasks.
This is proving to be more complicated than I was expecting. I thought it would be very easy to access the bits of a byte[] to manipulate the data directly, extract ranges of bits, then convert back to the values required to rebuild the object. Are there best practices for this type of data serialization? Does anyone know of a tutorial or example reference I could read?

Standard and efficient serialization methods are:
Using BinaryWriter / BinaryReader:
public byte[] Serialize()
{
using(var s = new MemoryStream())
using(var w = new BinaryWriter(s))
{
w.Write(firstBool);
w.Write(firstFloat);
...
return s.ToArray();
}
}
public void Deserialize(byte[] bytes)
{
using(var s = new MemoryStream(bytes))
using(var r = new BinaryReader(s))
{
firstBool = r.ReadBool();
firstFload = r.ReadFloat();
...
}
}
Using protobuf.net
BinaryWriter / BinaryReader is much faster (around 7 times). Protobuf is more flexible, easy to use, very popular and serializes into around 33% fewer bytes. (of course these numbers are orders of magnitude and depend on what you serialize and how).
Now basically BinaryWriter will write 1 + 4 + 4 + 4 = 13 bytes. You shrink it to 5 bytes by converting the values to bool, byte, byte, short first by rounding it the way you want. Finally it's easy to merge the bool with one of your bytes to get 4 bytes if you really want to.
I don't really discourage manual serialization. But it has to be worth the price in terms of performance. The code is quite unreadable. Use bit masks and binary shifts on bytes directly but keep it as simple as possible. Don't use BitArray. It's slow and not more readable.

Here is a simple method for pack/unpack. But you loose accuracy converting a float to only 7/8 bits
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
foreach (Data data in Data.input)
{
Data.Print(data);
Data results = Data.Unpack(Data.Pack(data));
Data.Print(results);
}
Console.ReadLine();
}
}
public class Data
{
public static List<Data> input = new List<Data>() {
new Data() { firstBool = true, firstFloat = 0.2345F, secondFloat = 0.432F, firstInt = 12},
new Data() { firstBool = true, firstFloat = 0.3445F, secondFloat = 0.432F, firstInt = 11},
new Data() { firstBool = false, firstFloat = 0.2365F, secondFloat = 0.432F, firstInt = 9},
new Data() { firstBool = false, firstFloat = 0.545F, secondFloat = 0.432F, firstInt = 8},
new Data() { firstBool = true, firstFloat = 0.2367F, secondFloat = 0.432F, firstInt = 7}
};
public bool firstBool { get; set; }
public float firstFloat {get; set; } //(0.0 to 1.0)
public float secondFloat {get; set; } //(0.0 to 1.0)
public int firstInt { get; set; } //(0 to 10,000)
public static byte[] Pack(Data data)
{
byte[] results = new byte[4];
results[0] = (byte)((data.firstBool ? 0x80 : 0x00) | (byte)(data.firstFloat * 128));
results[1] = (byte)(data.secondFloat * 256);
results[2] = (byte)((data.firstInt >> 8) & 0xFF);
results[3] = (byte)(data.firstInt & 0xFF);
return results;
}
public static Data Unpack(byte[] data)
{
Data results = new Data();
results.firstBool = ((data[0] & 0x80) == 0) ? false : true;
results.firstFloat = ((float)(data[0] & 0x7F)) / 128.0F;
results.secondFloat = (float)data[1] / 256.0F;
results.firstInt = (data[2] << 8) | data[3];
return results;
}
public static void Print(Data data)
{
Console.WriteLine("Bool : '{0}', 1st Float : '{1}', 2nd Float : '{2}', Int : '{3}'",
data.firstBool,
data.firstFloat,
data.secondFloat,
data.firstInt
);
}
}
}

int to byte[] consistency over network

I have a struct that gets used all over the place and that I store as byteArray on the hd and also send to other platforms.
I used to do this by getting a string version of the struct and using getBytes(utf-8) and getString(utf-8) during serialization. With that I guess I avoided the little and big endian problems?
However that was quite a bit of overhead and I am now using this:
public static explicit operator byte[] (Int3 self)
{
byte[] int3ByteArr = new byte[12];//4*3
int x = self.x;
int3ByteArr[0] = (byte)x;
int3ByteArr[1] = (byte)(x >> 8);
int3ByteArr[2] = (byte)(x >> 0x10);
int3ByteArr[3] = (byte)(x >> 0x18);
int y = self.y;
int3ByteArr[4] = (byte)y;
int3ByteArr[5] = (byte)(y >> 8);
int3ByteArr[6] = (byte)(y >> 0x10);
int3ByteArr[7] = (byte)(y >> 0x18);
int z = self.z;
int3ByteArr[8] = (byte)z;
int3ByteArr[9] = (byte)(z >> 8);
int3ByteArr[10] = (byte)(z >> 0x10);
int3ByteArr[11] = (byte)(z >> 0x18);
return int3ByteArr;
}
public static explicit operator Int3(byte[] self)
{
int x = self[0] + (self[1] << 8) + (self[2] << 0x10) + (self[3] << 0x18);
int y = self[4] + (self[5] << 8) + (self[6] << 0x10) + (self[7] << 0x18);
int z = self[8] + (self[9] << 8) + (self[10] << 0x10) + (self[11] << 0x18);
return new Int3(x, y, z);
}
It works quite well for me, but I am not quite sure how little/big endian works,. do I still have to take care of something here to be safe when some other machine receives an int I sent as a bytearray?

Your current approach will not work for the case when your application running on system which use Big-Endian. In this situation you don't need reordering at all.
You don't need to reverse byte arrays by your self
And you don't need check for endianess of the system by your self
Static method IPAddress.HostToNetworkOrder will convert integer to the integer with big-endian order.
Static method IPAddress.NetworkToHostOrder will convert integer to the integer with order your system using
Those methods will check for Endianness of the system and will do/or not reordering of integers.
For getting bytes from integer and back use BitConverter
public struct ThreeIntegers
{
public int One;
public int Two;
public int Three;
}
public static byte[] ToBytes(this ThreeIntegers value )
{
byte[] bytes = new byte[12];
byte[] bytesOne = IntegerToBytes(value.One);
Buffer.BlockCopy(bytesOne, 0, bytes, 0, 4);
byte[] bytesTwo = IntegerToBytes(value.Two);
Buffer.BlockCopy(bytesTwo , 0, bytes, 4, 4);
byte[] bytesThree = IntegerToBytes(value.Three);
Buffer.BlockCopy(bytesThree , 0, bytes, 8, 4);
return bytes;
}
public static byte[] IntegerToBytes(int value)
{
int reordered = IPAddress.HostToNetworkOrder(value);
return BitConverter.GetBytes(reordered);
}
And converting from bytes to struct
public static ThreeIntegers GetThreeIntegers(byte[] bytes)
{
int rawValueOne = BitConverter.ToInt32(bytes, 0);
int valueOne = IPAddress.NetworkToHostOrder(rawValueOne);
int rawValueTwo = BitConverter.ToInt32(bytes, 4);
int valueTwo = IPAddress.NetworkToHostOrder(rawValueTwo);
int rawValueThree = BitConverter.ToInt32(bytes, 8);
int valueThree = IPAddress.NetworkToHostOrder(rawValueThree);
return new ThreeIntegers(valueOne, valueTwo, valueThree);
}
If you will use BinaryReader and BinaryWriter for saving and sending to another platforms then BitConverter and byte array manipulating can be dropped off.
// BinaryWriter.Write have overload for Int32
public static void SaveThreeIntegers(ThreeIntegers value)
{
using(var stream = CreateYourStream())
using (var writer = new BinaryWriter(stream))
{
int reordredOne = IPAddress.HostToNetworkOrder(value.One);
writer.Write(reorderedOne);
int reordredTwo = IPAddress.HostToNetworkOrder(value.Two);
writer.Write(reordredTwo);
int reordredThree = IPAddress.HostToNetworkOrder(value.Three);
writer.Write(reordredThree);
}
}
For reading value
public static ThreeIntegers LoadThreeIntegers()
{
using(var stream = CreateYourStream())
using (var writer = new BinaryReader(stream))
{
int rawValueOne = reader.ReadInt32();
int valueOne = IPAddress.NetworkToHostOrder(rawValueOne);
int rawValueTwo = reader.ReadInt32();
int valueTwo = IPAddress.NetworkToHostOrder(rawValueTwo);
int rawValueThree = reader.ReadInt32();
int valueThree = IPAddress.NetworkToHostOrder(rawValueThree);
}
}
Of course you can refactor methods above and get more cleaner solution.
Or add as extension methods for BinaryWriter and BinaryReader.

Yes you do. With changes endianness your serialization which preserves bit ordering will run into trouble.
Take the int value 385
In a bigendian system it would be stored as
000000000000000110000001
Interpreting it as littleendian would read it as
100000011000000000000000
And reverse translate to 8486912
If you use the BitConverter class there will be a book property desiring the endianness of the system. The bitconverter can also produce the bit arrays for you.
You will have to decide to use either endianness and reverse the byte arrays according to the serializing or deserializing systems endianness.
The description on MSDN is actually quite detailed. Here they use Array.Reverse for simplicity. I am not certain that your casting to/from byte in order to do the bit manipulation is in fact the fastest way of converting, but that is easily benchmarked.

C# Find offset of byte pattern, check specific byte, change byte, export part of byte array

This could be long one. I do have a binary file, that contains some information.
What I want to do:
File (Binary) is read from OpenFileDialog
I'm now searching for specific bytes in this file
I'm getting offset of that byte, and then I'm checking byte value of offset+2
Basic if for (if offset+2 value is 0x08, then do this, if not, then do something else)
Now, search for offset for another byte pattern.
Copy everything from that offset till the end of file
Save copied byte array to file.
So, here're my codes for every step.
Step one:
1.
Byte[] bytes;
OpenFileDialog ofd = new OpenFileDialog();
ofd.ShowDialog();
path = ofd.FileName;
bytes = File.ReadAllBytes(path);
Step two, search specific pattern in this file. I used some help here on Stackoverflow, and end up with this:
VOID from stackoverflow:
static public List<int> SearchBytePattern(byte[] pattern, byte[] bytes)
{
List<int> positions = new List<int>();
int patternLength = pattern.Length;
int totalLength = bytes.Length;
byte firstMatchByte = pattern[0];
for (int i = 0; i < totalLength; i++)
{
if (firstMatchByte == bytes[i] && totalLength - i >= patternLength)
{
byte[] match = new byte[patternLength];
Array.Copy(bytes, i, match, 0, patternLength);
if (match.SequenceEqual<byte>(pattern))
{
positions.Add(i);
i += patternLength - 1;
}
}
}
return positions;
}
My void to search for pattern:
void CheckCamera()
{
Byte[] szukajkamera = { 0x02, 0x00, 0x08, 0x00, 0x20};
List<int> positions = SearchBytePattern(szukajkamera, bytes);
foreach (var item in positions){
MessageBox.Show(item.ToString("X2"));
IndexCamera = item;
}
int OffsetCameraCheck = IndexCamera + 2;
}
Item is now my offset, where 02 00 08 00 20 is in file.
Now, how do I check, if bytes(offset=IndexCamera+2) == 0x08 ?
I can do array.IndexOf, but there's plenty of 08 before that 08 I'm looking for.
For step 5 I'm also doing the thing, but it gets impossible for me, when Buffer.BlockCopy ask me for length.
For step 5 and forward I need to search again in this same file for another pattern, get it's offset and copy from that offset till the end. If I want so, then I need to buffer.blockcopy to non-empty byte array, but I just need it empty! I totally lost it. Please, help me.
Thank you!

how do I do bytes(offset=IndexCamera+2) == 0x08 ?
if(bytes[IndexCamera+2] == 0x08)....

When doing pattern searching the above answer does work, however you need to adapt it to search for more of the pattern.
Eg:
If you are looking for the location of 08 1D 1A AA 43 88 33
then you would need something like:
public static unsafe long IndexOf(this byte[] haystack, byte[] needle, long startOffset = 0)
{
fixed (byte* h = haystack) fixed (byte* n = needle)
{
for (byte* hNext = h + startOffset, hEnd = h + haystack.LongLength + 1 - needle.LongLength, nEnd = n + needle.LongLength; hNext < hEnd; hNext++)
for (byte* hInc = hNext, nInc = n; *nInc == *hInc; hInc++)
if (++nInc == nEnd)
return hNext - h;
return -1;
}
}
Note : Credit to Dylan Nicholson who wrote this code.