Develop Reference

Sending large volume of data between two C# programs

I am currently working on a C# (I could use python also) plug-in for two separate programs that need to communicate. In the first first program, I deconstruct 3D geometry into edges, points, normals, etc. Then I send all of this data to my plug-in in my second program to be rebuilt. Ideally this would happen as fast as possible to keep things i "real time".
Currently, I am converting my data with JSON, and writing the JSON to the disk. Then my second program watches for file changes, and then reads the file and uses the JSON data.
By far the biggest bottle neck of my entire plugin is the read/write process. There has to be a faster way than writing to a file.

There are several ways to use interprocess communication.
The most well known are used between different machines: WCF(.NET 3.5) and Remoting(.NET 2)
For on-machine communication you can choose to use Named pipes or Memory mapped files.
Memory mapped files are similar to your solution in that they use the page file as a backup.
I think the Named pipes solution is the most convenient:
You set up a "server" stream and wait for some "client" to connect.
Then you transfer the data just as you would through any other stream.
Here's NamedPipeServerStream.
And this is NamedPipeClientStream.
The code example there pretty much covers it.

I think WCF with named pipes would do the job, you just have to create transfer objects, which will be serialized and it will be all done by WCF automagically, or you can just prepare your existing objects to be transfered by named pipe with not really big overhead. Using json would be nice but it creates additional layer, and with WCF you transfer objects which can be used right away without translation by json. (Really they are translated to xml but you are not doing it by yourself so it is better then you could do with parsing json I think).

Server - client project (messanging protocols)

I want to do a project, that will include server and clients sides, using TcpSocket network communication (I use TcpListener for server and TcpClient for client side) and threading. But threading is not giving me any problems so far any longer.
But what it does, is something else... because the project will not include only chat, (but also creating new game, joining game, making moves, leaving game), I need to define somehow the message format.
I have read about messaging protocols and about using first few bytes of each message to tell the server what they are trying to do. The problem is that I do NOT know how to do it. So can someone show me an example of creating formated message?
Maybe its good to mention I use StreamReader and StreamWriter classes to pass data between server and clients. Is this a good way?
To add:
My problem now, is how to seperate this data, so that the server will know what to do with it. I have read about using 1nd few bytes to "be reserved" for the type of the message. But the problem is I don't know how to solve this issue. So far I was only using StreamReader and StreamWriter classes to pass only strings. If I use these kind of coding, it will all become too messy (not recognizable), if you know what I mean.
So I need to do something like it:
To send bytes:
1st few bytes the type of the data (but I don't know which class to use, maybe a BinaryWriter, and BinaryReader on the other side??)
the rest of the message
on server I have to have some code that will recognize these "1st few bytes" so the code will know what to do with the test of the message.
and based on these "1st few bytes" the code has to send data back to clients
Do you have any ideas on what this might look like, I mean as a skeleton (something basic, so I can work on with it).
Every bit of help would be very much appreciated.
I have found one example here on stackOverflow.com. It seems to be a code into a right direction. What do you think guys?

One option that might be much easier than defining your own protocol would be to use some existing library, like WCF or JSON-RPC.
Libraries like this have their disadvantages (they often produce output that is relatively big). If if you build your applications in a modular way, you can easily switch the communication backend later, when you find your first solution wasn't good enough.

You have a few options:
Make the message format textual - including its header. So, the header contains the message size, written out as text string, and ends with some terminator (\r\n for example.) HTTP takes this approach - the headers are text. When receiving the message, you process it with ReadLine, and parse header lines with stuff like int.TryParse(str).
Send message:
output.WriteLine(message.Length);
output.Write(message);
Receive message:
int len = int.Parse(input.ReadLine());
//message is the next 'len' characters
A second option: entire message is in a binary format - the message header is binary, and specifies amongst other things the message length. Message content is written as a stream of bytes, encoded using for example Encoding.UTF8.GetBytes(str).
Or, perhaps use the HTTP protocol? I believe there is basic support for HTTP server in the form of HttpListener. Here is a simple intro to HttpListener
Whatever you do, avoid using WCF :)

Transferring binary data using sockets?

I am thinking of making a server/client application that does this: the client(s) will connect to the server and the server will send back a list of folders (probably music or videos) that the client will be able to stream or download. I am going to make a GUI for this so that it won't be a text-only interface. I want it so that when the client connects to the server and gets the information about the media files that can be streamed/downloaded, the client will show the folders/files in an explorer-like interface (folders will show up with a folder icon, videos will show up with a video icon, etc.)
This seems really big to me but I really want to learn more about socket programming with C# and feel that getting my hands really dirty is the best way. As you can imagine, I have a lot of confusion about how to start this and what it is that I need to do to make this work.
The first part of my question is: If I want to let a client receive a list of files/folders and also have the ability to download/stream them, do I need to do something with transferring binary data using sockets or am I completely wrong on that? If I'm wrong, do I need to use something else like file readers and somehow use them over sockets?
The second part to my question is: when I transfer this information over to a client, how can I make the client show the appropriate icons for videos, folders, mp3s, etc.? I'm thinking that if I have to transfer binary data I will have to use that data to somehow populate the client GUI with the correct icons/data. I am stuck on exactly how to handle that. Are there certain methods/classes that I can use to parse that data and do what I want to do with it? Again, I don't know if I'm wrong about needing to transfer binary data but either way I am confused on what it is that's necessary to handle this.
I apologize if any of my terminology is wrong. I am obviously not an expert in C# yet ;)

Any time you are sending data between machines (or between processes), you are already talking binary data (you could argue that everything is binary data, but when working with it as an OO model it doesn't feel like it). The point here is that all you need to do is serialize your messages. There are a myriad of serialization frameworks - some aimed at convenience, some aimed to range-of-features, and som can use aimed at performance (minimum bandwidth etc).
Even the inbuilt ones would work - for example, you can use XmlSerializer etc to write to a Stream (such as the NetworkStream you typically use with a socket). But please - not BinaryFormatter (it will hurt you, honest). Personally I'd use something a bit less verbose such as protobuf-net, but I'm somewhat biased since I wrote it ;p
However, note that when working with sockets, you are typically sending multiple (separate) messages as part of the conversation. In order for each end to correctly and conveniently read the right data per-message from the stream, it is common to prefix each message with the length (number of bytes) - for example, as a little-endian int taking a fixed 4 bytes. So the reading process is:
read 4 bytes
interpret that as an int, let's say n
read n bytes
run that data through your deserializer, perhaps via MemoryStream
process the message
and the sending process might be:
serialize the message, perhaps to a MemoryStream
query the length and form the 4 byte prefix
write the prefix
write the data

You don't have to go to as low level as Socket, unless you just want to learn of course.
I suggest using WCF. It will completely meet your requirements, will be flexible and you will not have to deal with sockets directly.
WCF is a big framework, but you can quickly learn how to use it.
Introduction to WCF
Here you can find how to retrieve and show file icon in Windows.
If you client is hosted by Windows, you don't need to transfer icons, you can just retrieve image on the client by filename/extension.

Send files between python+django and C#

i would like to know, what is the best way to send files between python and C# and vice versa.
I have my own protocol which work on socket level, and i can send string and numbers in both ways. Loops works too. With this i can send pretty much anything, like package of users id, if it is simple data. But soon i will start sending whole files, maybe xml or executables.
Simple server with files is no an option because i want sending files from client too.
I was thinking about serialization but i don't know it is the best solution, but if it is i will love some tips from stackoverflow community.
EDIT:
I added django to question and chose http.

RPC may be a good idea for you, because it's relatively very high level. Instead of defining your own server and protocols, you can simply remotely execute routines over the network, pass in arguments and get back results.
For example, both languages have libraries for XML-RPC.

The easier way on my use case was send files using HTTP because with python i have additionaly running django.

Serialize in C++ then deserialize in C#?

Is there an easy way to serialize data in c++ (either to xml or binary), and then deserialize the data in C#?
I'm working with some remote WINNT machines that won't run .Net. My server app is written entirely in C#, so I want an easy way to share simple data (key value pairs mostly, and maybe some representation of a SQL result set). I figure the best way is going to be to write the data to xml in some predefined format on the client, transfer the xml file to my server, and have a C# wrapper read the xml into a usable c# object.
The client and server are communicating over a tcp connection, and what I really want is to serialize the data in memory on the client, transfer the binary data over the socket to a c# memory stream that I can deserialize into a c# object (eliminating file creation, transfer, etc), but I don't think anything like that exists. Feel free to enlighten me.
Edit
I know I can create a struct in the c++ app and define it in c# and transfer data that way, but in my head, that feels like I'm limiting what can be sent. I'd have to set predefined sizes for objects, etc

Protocol Buffers might be useful to you.
Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages – Java, C++, or Python.
.NET ports are available from Marc Gravell and Jon Skeet.

I checked out all mentioned projects like prottocol buffers, json, xml, etc. but after I have found BSON I use this because of the following reasons:
Easy to use API
Available in many languages (C, C++, Haskell, Go, Erlang, Perl, PHP, Python, Ruby, C#, ...)
Binary therefore very space efficient and fast (less bytes->less time)
constistent over platforms (no problems with endianess, etc)
hierarchical. The data model is comparable to json (what the name suggests) so most data modelling tasks should be solvable.
No precompiler necessary
wideley used (Mongodb, many languages)

C++ doesn't have structural introspection (you can't find out the fields of a class at runtime), so there aren't general mechanisms to write a C++ object. You either have to adopt a convention and use code generation, or (more typically) write the serialisation yourself.
There are some libraries for standard formats such as ASN.1, HDF5, and so on which are implementation language neutral. There are proprietary libraries which serve the same purpose (eg protocol buffers).
If you're targeting a particular architecture and compiler, then you can also just dump the C++ object as raw bytes, and create a parser on the C# side.
Quite what is better depends how tightly coupled you want your endpoints to be, and whether the data is mainly numerical (HDF5), tree and sequence structures (ASN.1), or simple plain data objects (directly writing the values in memory)

Other options would be:
creating a binary file that contains the data in the way you need it
( not a easy & portable solution )
XML
YAML
plain text files

There are a lot of options you can choose from. Named pipes, shared
memory, DDE, remoting... Depends on your particular need.
Quick googling gave the following:
Named pipes
Named Shared Memory
DDE

As mentioned already, Protocol Buffers are a good option.
If that option doesn't suit your needs, then I would look at sending the XML over to the client (you would have to prefix the message with the length so you know how much to read) and then using an implementation of IXmlSerializer or use the DataContract/DataMember attributes in conjunction with the DataContractSerializer to get your representation in .NET.
I would recommend against using the marshaling attributes, as they aren't supported on things like List<T> and a number of other standard .NET classes which you would use normally.