I have multiple topics produces by a external application that share the same keys and they share this model:
Topic1: Produces messages using the unix ms timestamp as key.
Topic2: Reads from Topic1 and produce messages with the same key
Topic3: Reads from Topic2 and produce messages with the same key
.. and so on
My application relies on the final message in Topic3(They're produced within 1-2ms) but I also want the value in Topic1. I've created a class that uses several consumers but I feel like it's not very efficient. How should this be done in c#?
I've heard of the Kafka Streams java API but I've not managed to find anything similar in c#.
confluent-kafka-dotnet client does not support Kafka Streams yet. For more information you can refer to issue #344.
However, you can use kafka-streams-dotnet which is a .NET stream processing library for Apache Kafka.
An alternative option is ksqldb that lets you join streams.
Related
I have a a C# consumer of a Kafka topic but I would like to know the version/id of the message that is being read. Is this attribute available somehow in the ComsumeResult? or is there some other way to know what the version was of the produced record. The reason I ask is that the Producer keeps changing the version and I what like to track from the consumed message the version it was produced with.
So, Kafka doesn't have schemas, by default. I assume you are using Confluent Schema Registry?
That being said, after deserialization, no.
You would need to consume the data as bytes, then observe bytes 1-4 as an integer
https://docs.confluent.io/current/schema-registry/serdes-develop/index.html#wire-format
We are using protocol buffers messages over grpc to implement a micro service architecture between several components developed in several languages (c# - c++ - java).
One of our component has the responsibility to persist those messages in a document store. Currently we are using mongodb with the c# driver and protocol buffers c# generated classes required hand written serialization/deserialization code to work with c# driver).
Is there any document store / database that could store directly protocol buffers message (not as binary blob) and enable query on those messages properties ?
You could write a "redis module" (4.0) to do the job, similar to how rejson is implemented; redis modules allow you to put your own code 100% inside the server. You'd need to think about how schemas should work, though, and what operations you want to support against the data; essentially: can you leave it raw in a single binary string, or will you need to extract the data inside the server, just using the raw API on the boundary?
We create client-server erp system with huge amount(in the future) of data and use c# for client and c for server. We started with xml for small requests/responses and it looks ok for now. But what is the best data exchange format for increasing amount of data per response(up to 100MB i think)?
P.S.
Highest priority is encode/decode speed.
We use Sockets to transfer data.
It really depends on what kind of data you are going to send to and receive back from the server. If “data” is some sort of a buffer with the known length and the usual operations are to put / get an object to / from the server, then I would recommend you to take a look at HTTP: it's a very simple protocol, there are many libraries and applications that support it, you can easily extend the protocol, add an encryption (HTTPS) and compression (gzip), and this protocol is easy to debug and work with.
If you want to send network packets that contain many data fields of different types, then you want to encode and decode such packet (serialize) before sending to the network. There are a plenty of open source libraries in the Internet which support both C and C# languages (you can even write your own implementation, it's not that hard). I would recommend you to take a look at XML / JSON (text-based standards for data exchange), you will find it much more easier to debug communication problem when working with a textural data.
Hope it helps !
I would suggest to take a look at JSON:
http://www.json.org/
I'm trying to make a P2P application with WCF and so far it seems simple enough and I've managed to send simple string messages but that's about it. I'd like to send files in the same manner, but I can't find any useful tutorials on it. All I find is different ways to build chat applications. Are there any useful resources on how to send files in a P2P mesh?
Currently I'm going off of a slightly modified version of this. I've found a similar example on how to send and retrieve files from a server with WCF, but I don't know if it's in any way compatible with the structure I already have since it uses a different binding.
The file transfer example that you link to uses streaming
There are only 4 bindings that support streaming, unfortunatly the peer binding that you are using is not among them.
What you can do is to create a WCF contract that has 2 properties, file_name and file_contents. The file name is a string and the file contents is a byte array. Then you can convert the file to a byte array and send it over the same way as you send over a string.
I am in need of a PGP service for .NET that will provide the following:
Encryption/decryption of files provided as byte arrays and/or streams (e.g. writing to hard drive and having the service read it is unacceptable)
Use of arbitrary keys passed in as byte arrays and/or streams
Needs to work for a headless service running on a server with nobody watching it (no modal popups or user input required)
We've felt out a couple of products but not been totally pleased with how any of them worked. Are there any suggestions? Thanks!
It's hard to guess what you could try as there are not much OpenPGP implementations for .NET. Namely, OpenPGPBlackbox package of our SecureBlackbox product is the only comprehensive self-contained implementation for .NET (BouncyCastle offers something as well, but they seem to be limited to older RFC 2440). You are welcome to check OpenPGPBlackbox and if you have problems with it, contact our technical support as described on product pages.