I have multiple topics produces by a external application that share the same keys and they share this model:
Topic1: Produces messages using the unix ms timestamp as key.
Topic2: Reads from Topic1 and produce messages with the same key
Topic3: Reads from Topic2 and produce messages with the same key
.. and so on
My application relies on the final message in Topic3(They're produced within 1-2ms) but I also want the value in Topic1. I've created a class that uses several consumers but I feel like it's not very efficient. How should this be done in c#?
I've heard of the Kafka Streams java API but I've not managed to find anything similar in c#.
confluent-kafka-dotnet client does not support Kafka Streams yet. For more information you can refer to issue #344.
However, you can use kafka-streams-dotnet which is a .NET stream processing library for Apache Kafka.
An alternative option is ksqldb that lets you join streams.
I am currently working on a C# (I could use python also) plug-in for two separate programs that need to communicate. In the first first program, I deconstruct 3D geometry into edges, points, normals, etc. Then I send all of this data to my plug-in in my second program to be rebuilt. Ideally this would happen as fast as possible to keep things i "real time".
Currently, I am converting my data with JSON, and writing the JSON to the disk. Then my second program watches for file changes, and then reads the file and uses the JSON data.
By far the biggest bottle neck of my entire plugin is the read/write process. There has to be a faster way than writing to a file.
There are several ways to use interprocess communication.
The most well known are used between different machines: WCF(.NET 3.5) and Remoting(.NET 2)
For on-machine communication you can choose to use Named pipes or Memory mapped files.
Memory mapped files are similar to your solution in that they use the page file as a backup.
I think the Named pipes solution is the most convenient:
You set up a "server" stream and wait for some "client" to connect.
Then you transfer the data just as you would through any other stream.
Here's NamedPipeServerStream.
And this is NamedPipeClientStream.
The code example there pretty much covers it.
I think WCF with named pipes would do the job, you just have to create transfer objects, which will be serialized and it will be all done by WCF automagically, or you can just prepare your existing objects to be transfered by named pipe with not really big overhead. Using json would be nice but it creates additional layer, and with WCF you transfer objects which can be used right away without translation by json. (Really they are translated to xml but you are not doing it by yourself so it is better then you could do with parsing json I think).
We create client-server erp system with huge amount(in the future) of data and use c# for client and c for server. We started with xml for small requests/responses and it looks ok for now. But what is the best data exchange format for increasing amount of data per response(up to 100MB i think)?
P.S.
Highest priority is encode/decode speed.
We use Sockets to transfer data.
It really depends on what kind of data you are going to send to and receive back from the server. If “data” is some sort of a buffer with the known length and the usual operations are to put / get an object to / from the server, then I would recommend you to take a look at HTTP: it's a very simple protocol, there are many libraries and applications that support it, you can easily extend the protocol, add an encryption (HTTPS) and compression (gzip), and this protocol is easy to debug and work with.
If you want to send network packets that contain many data fields of different types, then you want to encode and decode such packet (serialize) before sending to the network. There are a plenty of open source libraries in the Internet which support both C and C# languages (you can even write your own implementation, it's not that hard). I would recommend you to take a look at XML / JSON (text-based standards for data exchange), you will find it much more easier to debug communication problem when working with a textural data.
Hope it helps !
I would suggest to take a look at JSON:
http://www.json.org/
I'm trying to make a P2P application with WCF and so far it seems simple enough and I've managed to send simple string messages but that's about it. I'd like to send files in the same manner, but I can't find any useful tutorials on it. All I find is different ways to build chat applications. Are there any useful resources on how to send files in a P2P mesh?
Currently I'm going off of a slightly modified version of this. I've found a similar example on how to send and retrieve files from a server with WCF, but I don't know if it's in any way compatible with the structure I already have since it uses a different binding.
The file transfer example that you link to uses streaming
There are only 4 bindings that support streaming, unfortunatly the peer binding that you are using is not among them.
What you can do is to create a WCF contract that has 2 properties, file_name and file_contents. The file name is a string and the file contents is a byte array. Then you can convert the file to a byte array and send it over the same way as you send over a string.
Is there an easy way to serialize data in c++ (either to xml or binary), and then deserialize the data in C#?
I'm working with some remote WINNT machines that won't run .Net. My server app is written entirely in C#, so I want an easy way to share simple data (key value pairs mostly, and maybe some representation of a SQL result set). I figure the best way is going to be to write the data to xml in some predefined format on the client, transfer the xml file to my server, and have a C# wrapper read the xml into a usable c# object.
The client and server are communicating over a tcp connection, and what I really want is to serialize the data in memory on the client, transfer the binary data over the socket to a c# memory stream that I can deserialize into a c# object (eliminating file creation, transfer, etc), but I don't think anything like that exists. Feel free to enlighten me.
Edit
I know I can create a struct in the c++ app and define it in c# and transfer data that way, but in my head, that feels like I'm limiting what can be sent. I'd have to set predefined sizes for objects, etc
Protocol Buffers might be useful to you.
Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages – Java, C++, or Python.
.NET ports are available from Marc Gravell and Jon Skeet.
I checked out all mentioned projects like prottocol buffers, json, xml, etc. but after I have found BSON I use this because of the following reasons:
Easy to use API
Available in many languages (C, C++, Haskell, Go, Erlang, Perl, PHP, Python, Ruby, C#, ...)
Binary therefore very space efficient and fast (less bytes->less time)
constistent over platforms (no problems with endianess, etc)
hierarchical. The data model is comparable to json (what the name suggests) so most data modelling tasks should be solvable.
No precompiler necessary
wideley used (Mongodb, many languages)
C++ doesn't have structural introspection (you can't find out the fields of a class at runtime), so there aren't general mechanisms to write a C++ object. You either have to adopt a convention and use code generation, or (more typically) write the serialisation yourself.
There are some libraries for standard formats such as ASN.1, HDF5, and so on which are implementation language neutral. There are proprietary libraries which serve the same purpose (eg protocol buffers).
If you're targeting a particular architecture and compiler, then you can also just dump the C++ object as raw bytes, and create a parser on the C# side.
Quite what is better depends how tightly coupled you want your endpoints to be, and whether the data is mainly numerical (HDF5), tree and sequence structures (ASN.1), or simple plain data objects (directly writing the values in memory)
Other options would be:
creating a binary file that contains the data in the way you need it
( not a easy & portable solution )
XML
YAML
plain text files
There are a lot of options you can choose from. Named pipes, shared
memory, DDE, remoting... Depends on your particular need.
Quick googling gave the following:
Named pipes
Named Shared Memory
DDE
As mentioned already, Protocol Buffers are a good option.
If that option doesn't suit your needs, then I would look at sending the XML over to the client (you would have to prefix the message with the length so you know how much to read) and then using an implementation of IXmlSerializer or use the DataContract/DataMember attributes in conjunction with the DataContractSerializer to get your representation in .NET.
I would recommend against using the marshaling attributes, as they aren't supported on things like List<T> and a number of other standard .NET classes which you would use normally.