Develop Reference

How would I reverse this simple-looking algorithm?

I got some old LED board to which you'd send some text and hang it up somewhere... it was manufactured in 1994/95 and it communicates over a serial port, with a 16-bit MS-DOS application in which you can type in some text.
So, because you probably couldn't run it anywhere except by using DOSBox or similar tricks, I decided to rewrite it in C#.
After port-monitoring the original dos-exe I've found that it's really not interested in you rebuilding it - requests must be answered suitable, varying bytes, pre-sent "ping" messages, etc...
Maybe you know a similar checksum routine/pattern as my dos-exe uses or you could give any tips in trying to reverse-engineer this... Additionally, because I am only familiar with programming and didn't spend much time on reversing methods and/or analyzing protocols, please don't judge me if this topic is a bit of a stupid idea - I'll be glad about any help I get...
The message really containing the text that should be displayed is 143 bytes long (just that long because it puts filler bytes if you don't use up all the space with your text), and in that msg I noticed the following patterns:
The fourth byte (which still belongs to the msg header) varies from a list of 6 or 7 repeating values (in my examples, that byte will always be 0F).
The two last bytes function as a checksum
Some examples:
displayed text: "123" (hex: "31 32 33"), checksum hex: "45 52"
text: "132" ("31 33 32"), checksum hex: "55 FF"
text: "122" ("31 32 32"), checksum hex: "95 F4"
text: "133" ("31 33 33"), checksum hex: "85 59"
text: "112" ("31 31 32"), checksum hex: "C5 C8"
text: "124" ("31 32 34"), checksum hex: "56 62"
text: "134" ("31 33 34"), checksum hex: "96 69"
text: "211" ("32 31 31"), checksum hex: "5D 63"
text: "212" ("32 31 32"), checksum hex: "3C A8"
text: {empty}, checksum hex: "DB BA"
text: "1" ("31"), checksum hex: "AE 5F"
So far I am completely sure that the checksum really does depend on this fourth byte in the header, because if it changes, the checksums will be completely different for the same text to be displayed.
Here's an an example of a full 143 bytes-string displaying "123", just for giving you a better orientation:
02 86 04 0F 05 03 01 03 01 03 01 03 00 01 03 00 ...............
00 31 00 32 00 33 00 20 00 20 00 20 00 20 00 20 .1.2.3. . . . .
00 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 . . . . . . . .
00 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 . . . . . . . .
00 20 00 20 00 20 00 20 00 20 00 FE 03 01 03 01 . . . . . .þ....
04 01 03 00 01 03 00 00 20 00 20 00 20 00 20 00 ........ . . . .
20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00 . . . . . . . .
20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00 . . . . . . . .
20 00 20 00 20 00 20 00 20 00 20 00 20 45 52
(the text information starts with 2nd byte in 2nd line "31 00 32 00 33 00 (...)"
Unfortunately on the whole web, there are no user manuals, documentations, not even a real evidence that this info board-device ever existed.

I'll write F(s) for the checksum you get when feeding in string s.
Observe that:
F("122") xor F("123") = 95 F4 xor 45 52 = D0 A6
F("132") xor F("133") = 55 FF xor 85 59 = D0 A6
F("123") xor F("124") = 45 52 xor 56 62 = 13 30
F("133") xor F("134") = 85 59 xor 96 69 = 13 30
all of which is consistent with the checksum having the following property, which checksums not infrequently have: changing a given bit in the input always XORs the output with the same thing.
I predict, e.g., that F("210") = F("211") xor D0 A6 = 8D C5, and similarly that F("222") = 3C A8 xor C5 C8 xor 95 F4 = 6C 94.
If this is true, then the following gives you a brute-force-y way to figure out the checksum in general, provided you have a black box that computes checksums for you (which apparently you have):
Find the checksum of an input all of whose bits are 0. Call this a.
For each bit position k, find the checksum of an input all of whose bits are 0 except for bit k which is 1. Call this a XOR b(k).
Now the checksum of an arbitrary input is a XOR each b(k) where bit k is set in the input.
Usually the b(k) will be closely related to one another -- the usual pattern is that you're feeding bits into a shift register -- so the above is more brute-force-y than you'd need given an understanding of the algorithm. But I expect it works, if you are able to feed in arbitrarily chosen bit patterns as input.
If not, you may still be able to do it. E.g., suppose all you actually get to choose is 29 7-bit ASCII character values, at positions 17,19,...73 of your input. Then you can first of all feed in all spaces (0x20) and then XOR each in turn with 1-bits in positions 0..6. That won't give you all the b(k) but it will give you enough for arbitrary 29-ASCII-character inputs.

Serialized .NET class to PHP

I am getting some serialized .NET class string data from a source and I just need to turn it into something readable in PHP. Doesn't necessarily have to be turned into an "object" or JSON but I need to read it somehow. I think the .NET string is just a class with some set properties but it is binary and not portable obviously. I'm not looking to convert .NET code to PHP code. Here is an example of the data:
U:?�S�#��-��v�Y��?������An�#AMAUI������
I realize this is actually binary and not printable text. I'm just using this as an example of what I see when catting the file.

Short answer:
I would really suggest NOT implementing the interpretation of the binary representation yourself. I would use another format instead (JSON, XML, etc.).
Long answer:
However, if this is not possible there is of course a way...
The actual question is: What does the binary format of serialized .NET objects look like and how can we interpret it correctly?
I have based all my research on the .NET Remoting: Binary Format Data Structure specification.
Example class:
To have a working example, I have created a simple class called A which contains 2 properties, one string and one integer value, they are called SomeString and SomeValue.
Class A looks like this:
[Serializable()]
public class A
{
public string SomeString
{
get;
set;
}
public int SomeValue
{
get;
set;
}
}
For the serialization I used the BinaryFormatter of course:
BinaryFormatter bf = new BinaryFormatter();
StreamWriter sw = new StreamWriter("test.txt");
bf.Serialize(sw.BaseStream, new A() { SomeString = "abc", SomeValue = 123 });
sw.Close();
As can be seen, I passed a new instance of class A containing abc and 123 as values.
Example result data:
If we look at the serialized result in an hex editor, we get something like this:
Let us interpret the example result data:
According to the above mentioned specification (here is the direct link to the PDF: [MS-NRBF].pdf) every record within the stream is identified by the RecordTypeEnumeration. Section 2.1.2.1 RecordTypeNumeration states:
This enumeration identifies the type of the record. Each record (except for MemberPrimitiveUnTyped) starts with a record type enumeration. The size of the enumeration is one BYTE.
SerializationHeaderRecord:
So if we look back at the data we got, we can start interpreting the first byte:
As stated in 2.1.2.1 RecordTypeEnumeration a value of 0 identifies the SerializationHeaderRecord which is specified in 2.6.1 SerializationHeaderRecord:
The SerializationHeaderRecord record MUST be the first record in a binary serialization. This record has the major and minor version of the format and the IDs of the top object and the headers.
It consists of:
RecordTypeEnum (1 byte)
RootId (4 bytes)
HeaderId (4 bytes)
MajorVersion (4 bytes)
MinorVersion (4 bytes)
With that knowledge we can interpret the record containing 17 bytes:
00 represents the RecordTypeEnumeration which is SerializationHeaderRecord in our case.
01 00 00 00 represents the RootId
If neither the BinaryMethodCall nor BinaryMethodReturn record is present in the serialization stream, the value of this field MUST contain the ObjectId of a Class, Array, or BinaryObjectString record contained in the serialization stream.
So in our case this should be the ObjectId with the value 1 (because the data is serialized using little-endian) which we will hopefully see again ;-)
FF FF FF FF represents the HeaderId
01 00 00 00 represents the MajorVersion
00 00 00 00 represents the MinorVersion
BinaryLibrary:
As specified, each record must begin with the RecordTypeEnumeration. As the last record is complete, we must assume that a new one begins.
Let us interpret the next byte:
As we can see, in our example the SerializationHeaderRecord it is followed by the BinaryLibrary record:
The BinaryLibrary record associates an INT32 ID (as specified in [MS-DTYP] section 2.2.22) with a Library name. This allows other records to reference the Library name by using the ID. This approach reduces the wire size when there are multiple records that reference the same Library name.
It consists of:
RecordTypeEnum (1 byte)
LibraryId (4 bytes)
LibraryName (variable number of bytes (which is a LengthPrefixedString))
As stated in 2.1.1.6 LengthPrefixedString...
The LengthPrefixedString represents a string value. The string is prefixed by the length of the UTF-8 encoded string in bytes. The length is encoded in a variable-length field with a minimum of 1 byte and a maximum of 5 bytes. To minimize the wire size, length is encoded as a variable-length field.
In our simple example the length is always encoded using 1 byte. With that knowledge we can continue the interpretation of the bytes in the stream:
0C represents the RecordTypeEnumeration which identifies the BinaryLibrary record.
02 00 00 00 represents the LibraryId which is 2 in our case.
Now the LengthPrefixedString follows:
42 represents the length information of the LengthPrefixedString which contains the LibraryName.
In our case the length information of 42 (decimal 66) tell's us, that we need to read the next 66 bytes and interpret them as the LibraryName.
As already stated, the string is UTF-8 encoded, so the result of the bytes above would be something like: _WorkSpace_, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
ClassWithMembersAndTypes:
Again, the record is complete so we interpret the RecordTypeEnumeration of the next one:
05 identifies a ClassWithMembersAndTypes record. Section 2.3.2.1 ClassWithMembersAndTypes states:
The ClassWithMembersAndTypes record is the most verbose of the Class records. It contains metadata about Members, including the names and Remoting Types of the Members. It also contains a Library ID that references the Library Name of the Class.
It consists of:
RecordTypeEnum (1 byte)
ClassInfo (variable number of bytes)
MemberTypeInfo (variable number of bytes)
LibraryId (4 bytes)
ClassInfo:
As stated in 2.3.1.1 ClassInfo the record consists of:
ObjectId (4 bytes)
Name (variable number of bytes (which is again a LengthPrefixedString))
MemberCount(4 bytes)
MemberNames (which is a sequence of LengthPrefixedString's where the number of items MUST be equal to the value specified in the MemberCount field.)
Back to the raw data, step by step:
01 00 00 00 represents the ObjectId. We've already seen this one, it was specified as the RootId in the SerializationHeaderRecord.
0F 53 74 61 63 6B 4F 76 65 72 46 6C 6F 77 2E 41 represents the Name of the class which is represented by using a LengthPrefixedString. As mentioned, in our example the length of the string is defined with 1 byte so the first byte 0F specifies that 15 bytes must be read and decoded using UTF-8. The result looks something like this: StackOverFlow.A - so obviously I used StackOverFlow as name of the namespace.
02 00 00 00 represents the MemberCount, it tell's us that 2 members, both represented with LengthPrefixedString's will follow.
Name of the first member:
1B 3C 53 6F 6D 65 53 74 72 69 6E 67 3E 6B 5F 5F 42 61 63 6B 69 6E 67 46 69 65 6C 64 represents the first MemberName, 1B is again the length of the string which is 27 bytes in length an results in something like this: <SomeString>k__BackingField.
Name of the second member:
1A 3C 53 6F 6D 65 56 61 6C 75 65 3E 6B 5F 5F 42 61 63 6B 69 6E 67 46 69 65 6C 64 represents the second MemberName, 1A specifies that the string is 26 bytes long. It results in something like this: <SomeValue>k__BackingField.
MemberTypeInfo:
After the ClassInfo the MemberTypeInfo follows.
Section 2.3.1.2 - MemberTypeInfo states, that the structure contains:
BinaryTypeEnums (variable in length)
A sequence of BinaryTypeEnumeration values that represents the Member Types that are being transferred. The Array MUST:
Have the same number of items as the MemberNames field of the ClassInfo structure.
Be ordered such that the BinaryTypeEnumeration corresponds to the Member name in the MemberNames field of the ClassInfo structure.
AdditionalInfos (variable in length), depending on the BinaryTpeEnum additional info may or may not be present.
| BinaryTypeEnum | AdditionalInfos |
|----------------+--------------------------|
| Primitive | PrimitiveTypeEnumeration |
| String | None |
So taking that into consideration we are almost there...
We expect 2 BinaryTypeEnumeration values (because we had 2 members in the MemberNames).
Again, back to the raw data of the complete MemberTypeInfo record:
01 represents the BinaryTypeEnumeration of the first member, according to 2.1.2.2 BinaryTypeEnumeration we can expect a String and it is represented using a LengthPrefixedString.
00 represents the BinaryTypeEnumeration of the second member, and again, according to the specification, it is a Primitive. As stated above, Primitive's are followed by additional information, in this case a PrimitiveTypeEnumeration. That's why we need to read the next byte, which is 08, match it with the table stated in 2.1.2.3 PrimitiveTypeEnumeration and be surprised to notice that we can expect an Int32 which is represented by 4 bytes, as stated in some other document about basic datatypes.
LibraryId:
After the MemerTypeInfo the LibraryId follows, it is represented by 4 bytes:
02 00 00 00 represents the LibraryId which is 2.
The values:
As specified in 2.3 Class Records:
The values of the Members of the Class MUST be serialized as records that follow this record, as specified in section 2.7. The order of the records MUST match the order of MemberNames as specified in the ClassInfo (section 2.3.1.1) structure.
That's why we can now expect the values of the members.
Let us look at the last few bytes:
06 identifies an BinaryObjectString. It represents the value of our SomeString property (the <SomeString>k__BackingField to be exact).
According to 2.5.7 BinaryObjectString it contains:
RecordTypeEnum (1 byte)
ObjectId (4 bytes)
Value (variable length, represented as a LengthPrefixedString)
So knowing that, we can clearly identify that
03 00 00 00 represents the ObjectId.
03 61 62 63 represents the Value where 03 is the length of the string itself and 61 62 63 are the content bytes that translate to abc.
Hopefully you can remember that there was a second member, an Int32. Knowing that the Int32 is represented by using 4 bytes, we can conclude, that
must be the Value of our second member. 7B hexadecimal equals 123 decimal which seems to fit our example code.
So here is the complete ClassWithMembersAndTypes record:
MessageEnd:
Finally the last byte 0B represents the MessageEnd record.

How to convert string to null-terminated one?

How to convert a simple string to a null-terminated one?
Example:
Example string: "Test message"
Here are the bytes:
54 65 73 74 20 6D 65 73 73 61 67 65
I need string with bytes like follows:
54 00 65 00 73 00 74 00 20 00 6D 00 65 00 73 00 73 00 61 00 67 00 65 00 00
I could use loops, but will be too ugly code. How can I make this conversion by native methods?

It looks like you want a null-terminated Unicode string. If the string is stored in a variable str, this should work:
var bytes = System.Text.Encoding.Unicode.GetBytes(str + "\0");
(See it run.)
Note that the resulting array will have three zero bytes at the end. This is because Unicode represents characters using two bytes. The first zero is half of the last character in the original string, and the next two are how Unicode encodes the null character '\0'. (In other words, there is one extra null character using my code than what you originally specified, but this is probably what you actually want.)

A little background on c# strings is a good place to start.
The internal structure of a C# string is different from a C string.
a) It is unicode, as is a 'char'
b) It is not null terminated
c) It includes many utility functions that in C/C++ you would require for.
How does it get away with no null termination? Simple! Internally a C# String manages a char array. C# arrays are structures, not pointers (as in C/C++). As such, they are aware of their own length. The Null termination in C/C++ is required so that string utility functions like strcmp() are able to detect the end of the string in memory.
The null character does exist in c#.
string content = "This is a message!" + '\0';
This will give you a string that ends with a null terminator. Importantly, the null character is invisible and will not show up in any output. It will show in the debug windows. It will also be present when you convert the string to a byte array (for saving to disk and other IO operations) but if you do Console.WriteLine(content) it will not be visible.
You should understand why you want that null terminator, and why you want to avoid using a loop construct to get what you are after. A null terminated string is fairly useless in c# unless you end up converting to a byte array. Generally you will only do that if you want to send your string to a native method, over a network or to a usb device.
It is also important to be aware of how you are getting your bytes. In C/C++, a char is stored as 1 bytes (8bit) and the encoding is ANSI. In C# the encoding is unicode, it is two bytes (16bit). Jon Skeet's answer shows you how to get the bytes in unicode.

Tongue in cheek but potentially useful answer.
If you are after output on your screen in hex as you have shown there you want to follow two steps:
Convert string (with null character '\0' on the end) to byte array
Convert bytes strings representations encoded in hex
Interleave with spaces
Print to screen
Try this:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace stringlulz
{
class Program
{
static void Main(string[] args)
{
string original = "Test message";
byte[] bytes = System.Text.Encoding.Unicode.GetBytes(original + '\0');
var output = bytes.Aggregate(new StringBuilder(), (s, p) => s.Append(p.ToString("x2") + ' '), s => { s.Length--; return s; });
Console.WriteLine(output.ToString().ToUpper());
Console.ReadLine();
}
}
}
The output is:
54 00 65 00 73 00 74 00 20 00 6D 00 65 00 73 00 73 00 61 00 67 00 65 00 00 00

Here's a tested C# sample of an xml command null terminated and works great.
strCmd = #"<?xml version=""1.0"" encoding=""utf-8""?><Command name=""SerialNumber"" />";
sendB = System.Text.Encoding.UTF8.GetBytes(strCmd+"\0");
sportin.Send = sendB;

Are null terminators part of text encoding?

I'm trying to read a null terminated string from a byte array; the parameter to the function is the encoding.
string ReadString(Encoding encoding)
For example, "foo" in the following encodings are:
UTF-32: 66 00 00 00 6f 00 00 00 6f 00 00 00
UTF-8: 66 6f 6f
UTF-7: 66 6f 6f 2b 41 41 41 2d
If I copied all the bytes into an array (reading up to the null terminator) and passed that array into encoding.GetString(), it wouldn't work because if the string was UTF-32 encoded my algorithm would reach the "null terminator" after the second byte.
So I sort of have a double question: Are null terminators part of the encoding? If not, how could I decode the string character by character and check the following byte for the null terminator?
Thanks in advance
(suggestions are also appreciated)
Edit:
If "foo" was null terminated and utf-32 encoded, which would it be?:
1. 66 00 00 00 6f 00 00 00 6f 00 00 00 00
2. 66 00 00 00 6f 00 00 00 6f 00 00 00 00 00 00 00

The null terminator is not "logically" part of the string; it's not considered payload. It's widely used in C/C++ to indicate where the string ends.
Having said that you can have strings with embedded \0's but then you have to be careful to ensure the string doesn't appear truncated. For example std::string doesn't have a problem with embedded \0's. But if do a c_str() and and not account for the reported length() your string will appear cut off.

Null terminators are not part of the encoding, but the string representation used by some programming language, such as C. In .NET, System.String is prefixed by the string length as a 32-bit integer and is not null-terminated. Internally System.String is always UTF-16, but you can use the encoding to output different representations.
For the second part... Use the classes in System.Text such as UTF8Encoding and UTF32Encoding to read the string. You just have to select the right one based on your parameter...

This seems to work well for me (sample from actual code that reads a unicode, null terminated string from a byte array):
//trim null-termination from end of string
byte[] languageId = ...
string language = Encoding.Unicode.GetString(languageId,
0,
languageId.Length).Trim('\0');

How to edit a binary file's hex value using C#

So here's my issue. I have a binary file that I want to edit. I can use a hex editor to edit it of course, but I need to make a program to edit this particular file. Say that I know a certain hex I want to edit, I know it's address etc. Let's say that it's a 16-bit binary, and the address is 00000000, it's on row 04 and it has a value of 02. How could I create a program that would change the value of that hex, and only that hex with the click of a button?
I've found resources that talk about similar things, but I can't for the life of me find help with the exact issue.
Any help would be appreciated, and please, don't just tell me the answer if there is one but try and explain a bit.

I think this is best explained with a specific example. Here are the first 32 bytes of an executable file as shown in Visual Studio's hex editor:
00000000 4D 5A 90 00 03 00 00 00 04 00 00 00 FF FF 00 00
00000010 B8 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00
Now a file is really just a linear sequence of bytes. The rows that you see in a hex editor are just there to make things easier to read. When you want to manipulate the bytes in a file using code, you need to identify the bytes by their 0-based positions. In the above example, the positions of the non-zero bytes are as follows:
Position Value
-------- ------
0 0x4D
1 0x5A
2 0x90
4 0x03
8 0x04
12 0xFF
13 0xFF
16 0xB8
24 0x40
In the hex editor representation shown above, the numbers on the left represent the positions of the first byte in the corresponding line. The editor is showing 16 bytes per line, so they increment by 16 (0x10) at each line.
If you simply want to take one of the bytes in the file and change its value, the most efficient approach that I see would be to open the file using a FileStream, seek to the appropriate position, and overwrite the byte. For example, the following will change the 0x40 at position 24 to 0x04:
using (var stream = new FileStream(path, FileMode.Open, FileAccess.ReadWrite)) {
stream.Position = 24;
stream.WriteByte(0x04);
}

Well the first thing would probably be to understand the conversions. Hex to decimal probably isn't as important (unless of course you need to change the value from a decimal first, but that's a simple conversion formula), but hex to binary will be important seeing as each hex character (0-9,A-F) corresponds to a specific binary output.
After understanding that stuff, the next step is to figure out exactly what you are searching for, make the proper conversion, and replace that exact string. I would recommend (if the buffer wouldn't be too large) to take the entire hex dump and replace whatever you're searching for in there to avoid overwriting a duplicate binary sequence.
Hope that helps!
Regards,
Dennis M.