Develop Reference

Determining DLLImport arguments and safely calling an unmanaged C function

As a follow-up to my previous question, I finally got the C dll exported and usable in C#, but I'm stuck trying to figure out the proper argument types and calling method.
I've researched here on SO but there doesn't seem to be a pattern to how variable types are assigned.
I see some people suggest a StringBuilder for uchar*, others a byte[], some references to 'unsafe' code, etc. Can anyone recommend a solution based on this specific use-case?
Also note the exception generated as the code stands now, right after the call to the C function.
C function import:
[DllImport("LZFuncs.dll")]
internal static extern long LZDecomp(ref IntPtr outputBuffer, byte[] compressedBuffer, UInt32 compBufferLength); //Originally two uchar*, return is size of uncompressed data.
C function signature:
long LZDecomp(unsigned char *OutputBuffer, unsigned char *CompressedBuffer, unsigned long CompBufferLength)
Used as below:
for (int dataNum = 0; dataNum < _numEntries; dataNum++)
{
br.BaseStream.Position = _dataSizes[dataNum]; //Return to start of data.
if (_compressedFlags[dataNum] == 1)
{
_uncompressedSize = br.ReadInt32();
byte[] compData = br.ReadBytes(_dataSizes[dataNum] - 4);
IntPtr outData = IntPtr.Zero;
LZFuncs.LZDecomp(ref outData, compData, Convert.ToUInt32(compData.Length));
var uncompData = new byte[_uncompressedSize]; //System.ExecutionEngineException was unhandled
Marshal.Copy(outData, uncompData, 0, Convert.ToInt32(_uncompressedSize));
BinaryWriter bw = new BinaryWriter(new FileStream("compData" + dataNum + ".txt", FileMode.CreateNew));
bw.Write(uncompData);
bw.Close();
}
else
{
BinaryWriter bw = new BinaryWriter(new FileStream("uncompData" + dataNum + ".txt", FileMode.CreateNew));
bw.Write(br.ReadBytes(_dataSizes[dataNum]));
bw.Close();
}
}
I assume the C code is clobbering the memory pretty severely if it's breaking the C# caller with a CLR exception like that, but due to how the C code is written, there's absolutely no way to modify it without breaking the functionality, it's effectively a black box. (Written in assembly, for the most part.)
For reference, just a few questions I've read over in an effort to solve this myself:
How do I return a byte array from C++ to C#
Correct way to marshall uchar[] from native dll to byte[] in c#
There have been others but those are the most recent.

OK, that's not too hard to work with. The two buffer parameters are byte arrays. You should declare them as byte[]. The calling convention is Cdecl. Remember that C++ long is only 32 bits wide on Windows, so use C# int rather than C# long since the latter is 64 bits wide.
Declare the function like this:
[DllImport("LZFuncs.dll", CallingConvention = CallingConvention.Cdecl)]
internal static extern int LZDecomp(
[Out] byte[] outputBuffer,
[In] byte[] compressedBuffer,
uint compBufferLength
);
You are decompressing compressedBuffer into outputBuffer. You'll need to know how large outputBuffer needs to be (the code in the question shows that you already handle this) and allocate a sufficiently large array. Beyond that I think it's obvious how to call this.
The calling code will this look like this:
_uncompressedSize = br.ReadInt32();
byte[] compData = br.ReadBytes(_dataSizes[dataNum] - 4);
byte[] outData = new byte[_uncompressedSize];
int len = LZFuncs.LZDecomp(outData, compData, (uint)compData.Length);

This is an old question but a real issue and can lead to serious security issues so I thought I give it my 2 cents
Whenever I use [DllImport] I always add the location you consider safe, one option is to specify is for windows DLL's
[DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)]
However, have a look at your options to have it match your needs, you might load a private DLL that is located elsewhere.

Any proper way of sending byte data to Unity3D from a C plugin?

Just a question of curiosity here.
When you write plugins for Unity on the iOS platform, the plugins have a limited native-to-managed callback functionality (from the plugin and then to Unity). Basically this documentation:
iOS plugin Unity documentation
states that the function signature you are able to call back to is this:
Only script methods that correspond to the following signature can be called from native code: function MethodName(message:string)
The signature defined in C looks like this:
void UnitySendMessage(const char* obj, const char* method, const char* msg);
So this pretty much means I can only send strings back to Unity.
Now in my plugin I'm using protobuf-net to serialize objects and send them back to unity to be deserialized. I have gotten this to work, but by a solution I feel is quite ugly and not very elegant at all:
Person* person = [[[[[Person builder] setId:123]
setName:#"Bob"]
setEmail:#"bob#example.com"] build];
NSData* data = [person data];
NSString *rawTest = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
UnitySendMessage("GameObject", "ReceiveProductRequestResponse", [rawTest cStringUsingEncoding:NSUTF8StringEncoding]);
Basically I simply encode the bytestream into a string. In Unity I then get the bytes of the string and deserialize from there:
System.Text.UTF8Encoding encoding=new System.Text.UTF8Encoding();
Byte[] bytes = encoding.GetBytes(message);
This does work. But is there really no other way of doing it? Perhaps someone have an idea of how it could be done in some alternative way?

Base-64 (or another similar base) is the correct way to do this; you cannot use an encoding here (such as UTF8) - an encoding is intended to transform:
arbitrary string <===encoding===> structured bytes
i.e. where the bytes have a defined structure; this is not the case with protobuf; what you want is:
arbitrary bytes <===transform===> structured string
and base-64 is the most convenient implementation of that in most cases. Strictly speaking, you can sometimes go a bit higher than 64, but you'd probably have to roll it manually - not pretty. Base-64 is well-understood and well-supported, making it a good choice. I don't know how you do that in C, but in Unity it should be just:
string s = Convert.ToBase64String(bytes);
Often, you can also avoid an extra buffer here, assuming you are serializing in-memory to a MemoryStream:
string s;
using(var ms = new MemoryStream()) {
// not shown: serialization steps
s = Convert.ToBase64String(ms.GetBuffer(), 0, (int)ms.Length);
}

Example based on Marc Gravell's answer:
On the ios side:
-(void)sendData:(NSData*)data
{
NSString* base64String = [data base64Encoding];
const char* utf8String = [base64String cStringUsingEncoding:NSUTF8StringEncoding];
UnitySendMessage("iOSNativeCommunicationManager", "dataReceived", utf8String);
}
and on the unity side:
public delegate void didReceivedData( byte[] data );
public static event didReceivedData didReceivedDataEvent;
public void dataReceived( string bytesString )
{
byte[] data = System.Convert.FromBase64String(bytesString);
if( didReceivedDataEvent != null )
didReceivedDataEvent(data);
}

Problem with code converted from C++ to C#

I converted some code from a C++ application I wrote a long time ago to C#. In C++ I had a library I used that was a bit buffer, but my lack of C# knowledge has somewhat complicated the conversion.
When I query my application, and I simply use a ByteWriter without casting any values properly (just like bf.Write(-1) and bf.Write("stringhere") the query programs atleast query it, just get the wrong information. When I cast the values properly (to long, byte, short, etc) it completely breaks, and the query application doesn't even see it anymore.
C++ Code Snippet
void PlayerManager::BuildReplyInfo()
{
// Delete the old packet
g_ReplyInfo.Reset();
g_ReplyInfo.WriteLong(-1);
g_ReplyInfo.WriteByte(73);
g_ReplyInfo.WriteByte(g_ProtocolVersion.GetInt());
g_ReplyInfo.WriteString(iserver->GetName());
g_ReplyInfo.WriteString(iserver->GetMapName());
g_ReplyInfo.WriteString(gGameType);
}
C# Code
public static byte[] ConvertStringToByteArray(string str)
{
System.Text.UTF8Encoding encoding = new System.Text.UTF8Encoding();
return encoding.GetBytes(str);
}
//-----------------------------------
while (true)
{
data = new byte[1024];
recv = socket.ReceiveFrom(data, ref Remote);
Console.WriteLine("Message length is " + recv);
// If the length is 25 and the 5th byte is 'T' it is a A2S_INFO QUERY
if (recv == 25 && data[4] == 84)
{
Console.WriteLine("Source Engine Query!");
data = BuildReplyInformation();
socket.SendTo(data, 0, data.Length, SocketFlags.None, Remote);
}
}
}
public static byte[] BuildReplyInformation()
{
MemoryStream stream = new MemoryStream();
BinaryWriter writer = new BinaryWriter(stream);
writer.Write((long)(-1));
writer.Write((byte)(73)); // Steam Version
writer.Write((byte)(15)); // Protocol
writer.Write(ConvertStringToByteArray("Minecraft Server\0")); // Hostname
writer.Write(ConvertStringToByteArray("Map Name\0")); // Map Name
writer.Write(ConvertStringToByteArray("tf\0")); // Game Directory
writer.Write(ConvertStringToByteArray("Minecraft Server\0")); // Game Description
writer.Write((short)(440));
writer.Write((byte)(15)); // Players
writer.Write((byte)(32)); // Max Players
writer.Write((byte)(0)); // Bots
writer.Write((byte)(100));
writer.Write((byte)(119)); // 108 Linux, 119 Windows
writer.Write((byte)(0)); // Password Boolean
writer.Write((byte)(01)); // Vac Secured
writer.Write(ConvertStringToByteArray("1.1.3.7\0"));
return stream.ToArray();
}

A couple of ideas that might get you on track:
Are you sure you need UTF8 as string encoding?
When you look at the array and compare it to the intended structure, are you able to find out at what point the array does not comply to the standard?

Just a few things to keep in mind:
UTF-8 strings sometimes start with a BOM (byte order mark), sometimes not.
Strings sometimes are serialized length prefixed, sometimes null-terminated.
My suggestion is to double-check the original C++ method WriteString(...) to find out how it behaves with respect to #1 and #2, and then to double-check the C# method GetBytes(...) for the same. If I recall, the .NET binary serializer writes length-prefixed strings for each string written, but the UTF8 encoder does not (and does not output a null character either). The UTF8 encoder may also (depending on how you use it?) output a BOM.
Also, I'm suspicious of how \0 might be written out when passing through the UTF8 encoder. You might (for kicks) try outputting the null marker separately from the string content, as just a 0-valued byte.

Long size in C# was different from C++, resolved the issue.

Is there a BinaryReader in C++ to read data written from a BinaryWriter in C#?

I've written several ints, char[]s and the such to a data file with BinaryWriter in C#. Reading the file back in (in C#) with BinaryReader, I can recreate all of the pieces of the file perfectly.
However, attempting to read them back in with C++ yields some scary results. I was using fstream to attempt to read back the data and the data was not reading in correctly. In C++, I set up an fstream with ios::in|ios::binary|ios::ate and used seekg to target my location. I then read the next four bytes, which were written as the integer "16" (and reads correctly into C#). This reads as 1244780 in C++ (not the memory address, I checked). Why would this be? Is there an equivalent to BinaryReader in C++? I noticed it mentioned on msdn, but that's Visual C++ and intellisense doesn't even look like c++, to me.
Example code for writing the file (C#):
public static void OpenFile(string filename)
{
fs = new FileStream(filename, FileMode.Create);
w = new BinaryWriter(fs);
}
public static void WriteHeader()
{
w.Write('A');
w.Write('B');
}
public static byte[] RawSerialize(object structure)
{
Int32 size = Marshal.SizeOf(structure);
IntPtr buffer = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(structure, buffer, true);
byte[] data = new byte[size];
Marshal.Copy(buffer, data, 0, size);
Marshal.FreeHGlobal(buffer);
return data;
}
public static void WriteToFile(Structures.SomeData data)
{
byte[] buffer = Serializer.RawSerialize(data);
w.Write(buffer);
}
I'm not sure how I could show you the data file.
Example of reading the data back (C#):
BinaryReader reader = new BinaryReader(new FileStream("C://chris.dat", FileMode.Open));
char[] a = new char[2];
a = reader.ReadChars(2);
Int32 numberoffiles;
numberoffiles = reader.ReadInt32();
Console.Write("Reading: ");
Console.WriteLine(a);
Console.Write("NumberOfFiles: ");
Console.WriteLine(numberoffiles);
This I want to perform in c++. Initial attempt (fails at first integer):
fstream fin("C://datafile.dat", ios::in|ios::binary|ios::ate);
char *memblock = 0;
int size;
size = 0;
if (fin.is_open())
{
size = static_cast<int>(fin.tellg());
memblock = new char[static_cast<int>(size+1)];
memset(memblock, 0, static_cast<int>(size + 1));
fin.seekg(0, ios::beg);
fin.read(memblock, size);
fin.close();
if(!strncmp("AB", memblock, 2)){
printf("test. This works.");
}
fin.seekg(2); //read the stream starting from after the second byte.
int i;
fin >> i;
Edit: It seems that no matter what location I use "seekg" to, I receive the exact same value.

You realize that a char is 16 bits in C# rather than the 8 it usually is in C. This is because a char in C# is designed to handle Unicode text rather than raw data. Therefore, writing chars using the BinaryWriter will result in Unicode being written rather than raw bytes.
This may have lead you to calculate the offset of the integer incorrectly. I recommend you take a look at the file in a hex editor, and if you cannot work out the issue post the file and the code here.
EDIT1
Regarding your C++ code, do not use the >> operator to read from a binary stream. Use read() with the address of the int that you want to read to.
int i;
fin.read((char*)&i, sizeof(int));
EDIT2
Reading from a closed stream is also going to result in undefined behavior. You cannot call fin.close() and then still expect to be able to read from it.

This may or may not be related to the problem, but...
When you create the BinaryWriter, it defaults to writing chars in UTF-8. This means that some of them may be longer than one byte, throwing off your seeks.
You can avoid this by using the 2 argument constructor to specify the encoding. An instance of System.Text.ASCIIEncoding would be the same as what C/C++ use by default.

There are many thing going wrong in your C++ snippet. You shouldn't mix binary reading with formatted reading:
// The file is closed after this line. It is WRONG to read from a closed file.
fin.close();
if(!strncmp("AB", memblock, 2)){
printf("test. This works.");
}
fin.seekg(2); // You are moving the "get pointer" of a closed file
int i;
// Even if the file is opened, you should not mix formatted reading
// with binary reading. ">>" is just an operator for reading formatted data.
// In other words, it is for reading "text" and converting it to a
// variable of a specific data type.
fin >> i;

If it's any help, I went through how the BinaryWriter writes data here.
It's been a while but I'll quote it and hope it's accurate:
Int16 is written as 2 bytes and padded.
Int32 is written as Little Endian and zero padded
Floats are more complicated: it takes the float value and dereferences it, getting the memory address's contents which is a hexadecimal

What's the 'best' way to parse a stream into a struct or class?

actually i'm working with .Net Framework 3.5, so i have all these nice little features like lambdas, linq, etc.
Given is a serial connection (or to be more abstract: a stream) where you receive some data, which will be in a format like this:
struct Packet
{
byte STX
UInt16 DataLength
string Data
byte CRC
byte ETX
}
Using a simple mapping of the incoming data doesn't help due to the fact, that you don't really know how long one packet will be, cause it's written within the structure (the DataLength).
So my first idea would be to read the stream by byte and put it into ???. Yes, that's the next question. Where to store this first Raw data? Into a simple byte array, with the maximum possible length (that would be 65540 bytes, due to the fact, that DataLength is an UInt16 plus the additional bytes from the other fields). Or should i open up a Queue and fill it up will all the incoming bytes or maybe exists there another nice possibility?
Let's assume these problems are cleared and i have some kind of local buffer, that holds all the raw bytes from the stream. What's the nicest way to interpret it by the given structure?? Just doing some kind of for- or foreach-loop or exists there a smarter (with better performance) way (e.g. with regex or linq)?
Best regards,
Oliver

How about...
struct Packet
{
public byte STX;
public UInt16 DataLength;
public string Data;
public byte CRC;
public byte ETX;
}
//Warning: Need to add error handling
class PacketReader
{
private BinaryReader _reader;
public PacketReader(Stream stream)
{
_reader = new BinaryReader(stream);
}
Packet ReadPacket()
{
var packet = new Packet()
{
STX = _reader.ReadByte(),
DataLength = _reader.ReadUInt16(),
Data = Encoding.ASCII.GetString(
_reader.ReadBytes(packet.DataLength)),
CRC = _reader.ReadByte(),
ETX = _reader.ReadByte()
};
return packet;
}
}
Please note: I have not used BinaryReader.ReadString() on purpose because it is designed to operate on strings generated by BinaryWriter.WriteString(). The encoding is a bit different even though its a length prefixed string.

I would store them in a byte array and recreate them from there, its a fast and simple way to do it!
I would read the bytes and convert them With BitConverter, Encoding.UTF8..

Check this out, anyways it boils down to using the [Serializable] attribute, and your done
http://www.ondotnet.com/pub/a/dotnet/2002/08/26/serialization.html

Another possible option, making use of C#'s yield keyword:
public struct Packet
{
public byte STX;
public UInt16 DataLength;
public string Data;
public byte CRC;
public byte ETX;
}
public static class StreamExtensions
{
public IEnumerable<Packet> ToPacketStream(this Stream stream)
{
BinaryReader reader = new BinaryReader(stream);
while(reader.PeekChar() != -1) //Optionally change this to reflect your exit conditions
{
var packet = new Packet();
packet.STX = _reader.ReadByte();
packet.DataLength = _reader.ReadUInt16();
packet.Data = Encoding.ASCII.GetString(_reader.ReadBytes(packet.DataLength));
packet.CRC = _reader.ReadByte();
packet.ETX = _reader.ReadByte();
yield return packet;
}
}
}
//Usage
foreach(var packet in stream.ToPacketStream())
{
//Handle packet
}