Develop Reference

How to read a stream of bytes when you already know the structure of the bytes in C#?

I am attempting to parse a .blend (Blender 3D Modelling Program Save File) in C#.
.blend follows a TLV pattern and using 3rd party tools I've managed to extract the majority of the data:
.blend File Format
Shown in the image; I have deserialized the .blend FileBlocks (The actual file data) and the Sdna Structs (The types of data present in a .blend file)
For example Block[1799] is sdna type sdnas[173] which means Block[1799] is a mesh.
Each block has a property "Body" which is a System.Byte[] and contains that FileBlocks data (In the case of a mesh, it's name and mesh properties, with pointers to other file blocks like it's vertices)
In this particular case I'm reading the body of a "Vec2s" sdna struct where the first 4 bytes are [31,5,246,25...]
I also know the Vec2 sdna has 2 properties (x and y) which are both of type short. So my intuition tells me that the first 4 bytes of my object are 2 shorts representing x and y. Reading these out returns (1311,9206)
All the fields of the sdna structs are of type uint, short etc. So how am I supposed to get the actual values like names, floats, vector2s, pointers etc?
As you can tell I really don't know what I'm doing.

How to serialize very large files to a byte array?

I have a custom object. One of the properties on the object is a byte array with the contents of a file. This file can be VERY large (800+ MB in some instances). Since using the JsonSerializer and XmlSerializer are out of the question (the resulting string is too large), I think going with a byte array is the next best option.
I've looked through some other solutions (like this) but currently have no luck with what I need. We're working out of .NET 5 and things like the BinaryFormatter are no-go.
Is it somehow possible to take my object and write it to a stream so I can deal with it in a byte array?
Thanks

Protobuf-net sirialization/deserialization c# vs Linux c++

I'm passing messages between Windows C# client and Linux C++ server via TCP sockets. C# code uses protobuf-net v2, Linux Google's version of protobuf. Small test object that I pass has 6 fields ( Enum, Int, String). I need help with two problems:
C# portion unable to deserialize data sent from Linux, unless the Memory stream used as the storage for the data is initialized with the binary array in the constructor. Array can not be larger than the data sent from Linux ( 9 bytes in my case ). Code sample - byte[] data = new byte[9], copy data from the socket into the array.
MemoryStream myStream = new MemoryStream(data), pass myStream to Serializer.Deserialize...
If I initialize MemoryStream without bynary buffer or with array of 1024 bytes, Deserialize will create empty object, without processing data.
When I try to serialize the same object with the same values as Linux same in C# the size of the data is 11 bytes vs 9 on Linux. I checked the byte array in the debugger, C# version has the same 9 fields as Linux data in the indexes 2-11 of the array. Index 0 is 8 and index 1 is 9. I can try to get around the problem, by modifying Linux deserialization code, just need to know, if I always have to deal with two extra fields at the beginning of the message. Also, I can add two extra fields to messages generated on Linux if it going to fix my deserialization in C#, just need to know how to generate values for these two fields.
Thanks.
Alex.

Protobuf data is not self-terminating, simply. You can, however, create either a MemoryStream or ProtoReader that takes a larger payload, but is limited to a virtual length. If you are sending multiple messages, you will need to know the length of the payload - that is inescapable. This is often achieved via a length prefix. I would expect this to throw random errors - most likely "invalid field header 0" - and I wonder if you are swallowing that exception
impossible to comment without the specific example; most likely something to do with the default value of field number 1 (header 8 === "field 1 varint")

Can I convert a file to a byte array then save the byte array to a file keeping its properties?

Is it possible to convert a file to a byte array and save it at a later time while keeping all of its properties (name, type, etc)?
I'm reading email attachments and saving the bytes for future use. Issue with this is I don't know how to determine what file extension I need to use when saving the file.
If not what are the alternatives?

name, attributes, timestamps, etc are all meta data of the file and not part of the file contents.. so you need a container format.. You could use XML, or MIME messages, or any encapsulation scheme you desire.

You need to store the filename (a string) along with its contents.

You will need to store the properties, as well. If the byte array is merely the contents of the file, the properties would be lost.
You could, easily enough, just store all of the above information together. A single serializable class with the byte[] data and all relevant properties could be serialized into a byte array and used, for example.

How to write a file format handler

Today i'm cutting video at work (yea me!), and I came across a strange video format, an MOD file format with an companion MOI file.
I found this article online from the wiki, and I wanted to write a file format handler, but I'm not sure how to begin.
I want to write a file format handler to read the information files, has anyone ever done this and how would I begin?
Edit:
Thanks for all the suggestions, I'm going to attempt this tonight, and I'll let you know. The MOI files are not very large, maybe 5KB in size at most (I don't have them in front of me).

You're in luck in that the MOI format at least spells out the file definition. All you need to do is read in the file and interpret the results based on the file definition.
Following the definition, you should be able to create a class that could read and interpret a file which returns all of the file format definitions as properties in their respective types.
Reading the file requires opening the file and generally reading it on a byte-by-byte progression, such as:
using(FileStream fs = File.OpenRead(path-to-your-file)) {
while(true) {
int b = fs.ReadByte();
if(b == -1) {
break;
}
//Interpret byte or bytes here....
}
}
Per the wiki article's referenced PDF, it looks like someone already reverse engineered the format. From the PDF, here's the first entry in the format:
Hex-Address: 0x00
Data Type: 2 Byte ASCII
Value (Hex): "V6"
Meaning: Version
So, a simplistic implementation could pull the first 2 bytes of data from the file stream and convert to ASCII, which would provide a property value for the Version.
Next entry in the format definition:
Hex-Address: 0x02
Data Type: 4 Byte Unsigned Integer
Value (Hex):
Meaning: Total size of MOI-file
Interpreting the next 4 bytes and converting to an unsigned int would provide a property value for the MOI file size.
Hope this helps.

If the files are very large and just need to be streamed in, I would create a new reader object that uses an unmanagedmemorystream to read the information in.
I've done a lot of different file format processing like this. More recently, I've taken to making a lot of my readers more functional where reading tends to use 'yield return' to return read only objects from the file.
However, it all depends on what you want to do. If you are trying to create a general purpose format for use in other applications or create an API, you probably want to conform to an existing standard. If however you just want to get data into your own application, you are free to do it however you want. You could use a binaryreader on the stream and construct the information you need within your app, or get the reader to return objects representing the contents of the file.
The one thing I would recommend. Make sure it implements IDisposable and you wrap it in a using!