Develop Reference

NetMQ/PyZMQ - Async poller sometimes doesn't recieve message until disposal

I have a DEALER/DEALER connection.
For some reason, there seems to be an importance to who runs first. If I run the C# one (NetMQ), followed by the python one, the message sent via C# will block until the python dealer is running (as expected - Send blocks until it can send). However...
sometimes, the response I send from the python dealer is not received by the C# poller (the method added to ReceiveReady isn't called). This seems to happen when I send a message from C#, then wait a few (tried ~10) seconds before I run the python client.
However the moment I either dispose of the poller on the C# end or the socket on the python end, I get the message (in the C# app, I seem to only get the first message. But if I dispose of the socket on the python end, I get all messages at once).
I'm not sure what's going on. I would have guessed some kind of timeout on the messages, if it weren't for the fact they end up being received.
C# code (about half the lines are just for disposing):
using UnityEngine;
using AsyncIO;
using NetMQ;
using NetMQ.Sockets;
using System;
using System.Threading.Tasks;
public class MsgClient_BASIC : MonoBehaviour
{
DealerSocket socket;
NetMQPoller poller;
private bool requesterIsStarted;
public static MsgClient_BASIC this_msgclient;
private void OnReceive(object s, NetMQ.NetMQSocketEventArgs a)
{
bool more;
string FS = a.Socket.ReceiveFrameString(out more);
Debug.Log(FS);
}
void Start()
{
this_msgclient = this;
ForceDotNet.Force();
socket = new DealerSocket();
socket.Bind("tcp://localhost:5558");
poller = new NetMQPoller { socket };
socket.ReceiveReady += OnReceive;
poller.RunAsync();
}
private void Update()
{
if (Input.GetKeyDown("m"))
{
Debug.Log("Sending S");
socket.SendFrame("S");
}
}
void OnDestroy()
{
Debug.Log("Stopping async poller");
try
{
poller.StopAsync();
}
catch
{
}
Debug.Log("Disposing Poller");
try
{
poller.Dispose();
}
catch
{
}
Debug.Log("Closing socket");
try
{
socket.Close();
}
catch
{
}
Debug.Log("Disposing of socket");
try
{
((IDisposable)socket).Dispose();
}
catch
{
}
Debug.Log("Cleaning up");
NetMQConfig.Cleanup(true);
}
}
Python code:
import zmq
import json
from datetime import datetime as dt
from os import path
context = zmq.Context()
class Client:
def __init__(self,port):
self.socket = context.socket(zmq.DEALER)
self.socket.setsockopt(zmq.LINGER,500)
self.socket.connect("tcp://localhost:%d" % port)
def check_msgs(self):
try:
response = self.socket.recv_string(flags=zmq.NOBLOCK)
except zmq.Again as e:
response = None
print(response)
return response
def send_dict(self,d):
jd = json.dumps(d)
self.socket.send_string(jd,flags=zmq.NOBLOCK)
def destroy(self):
self.socket.close()
context.destroy(5)
print("Destroyed")

Weird behavior from either TcpClient or network

So I have two smartbulbs from Yeelight. I am trying to turn them on/off at the same time. You can achieve that by sending some TCP messages to them.
This works completely fine from NodeJS with this little script:
const net = require("net")
send_to_lamp("192.168.178.83", '{"id": 1, "method": "set_power", "params": ["on", "sudden"]}\r\n', answ => console.log(answ))
send_to_lamp("192.168.178.84", '{"id": 1, "method": "set_power", "params": ["on", "sudden"]}\r\n', answ => console.log(answ))
console.log("sent")
function send_to_lamp(ip, body, callback) {
var con = net.createConnection({port: 55443, host: ip}, () => {
con.write(`asd\r\n`) // send two requests to wake up connection
con.write(body)
})
}
But now comes the weird behavior to play: When trying to do the exact same thing from C#,
only the first bulb that is contacted reacts to the message. For example when executing this code:
using System.Net.Sockets;
Thread t1 = new Thread(con1);
t1.Start();
await Task.Delay(1000);
Thread t2 = new Thread(con2);
t2.Start();
await Task.Delay(5000);
async static void con1() {
await TryConnect("192.168.178.83", "{\"id\": 1, \"method\": \"set_power\", \"params\": [\"on\", \"sudden\"]}\r\n");
}
async static void con2() {
await TryConnect("192.168.178.84", "{\"id\": 1, \"method\": \"set_power\", \"params\": [\"on\", \"sudden\"]}\r\n");
}
static async Task TryConnect(String server, String message)
{
try
{
using (var client = new TcpClient()){
await client.ConnectAsync(server, 55443);
using (var netstream = client.GetStream())
using (var writer = new StreamWriter(netstream))
using (var reader = new StreamReader(netstream))
{
writer.AutoFlush = true;
netstream.ReadTimeout = 3000;
await writer.WriteLineAsync(message);
Console.WriteLine("sent {0}", message);
}
}
}
catch (ArgumentNullException e)
{
Console.WriteLine("ArgumentNullException: {0}", e);
}
catch (SocketException e)
{
Console.WriteLine("SocketException: {0}", e);
}
}
Only the lamp with the IP 192.168.178.83 turns itself on.
It even gets weirder. When just swapping the IP in the both con1/2 functions, so that the lamp with the IP 192.168.178.84 is called first, only it reacts and the lamp with the IP 192.168.178.83 does nothing.
I tried several different methods of sending TCP messages from C# (using async, multi-threading, have a dedicated class for each TcpClient ...)
Anyone got an idea what I can try to get this to work?
Wireshark log during NodeJS script
Wireshark log during C# script
Disclaimer: I do not know much about TCP connections and how they should look in Wireshark, but just seen from the software side of things this really seams nonsensical.

In the C # log you have RST because there you are using using which automatically closes the TCP connection. In Node.js, you would probably have to trigger the connection to close manually.
It's best to test your application using a TCP terminal (https://www.hw-group.com/software/hercules-setup-utility). Use it to create a local TCP server with listening port 55443.
Example settings
Now in your C # application refer both to IP 127.0.0.1 and change the transmitted content to distinguish between frames. For example, for the second lamp, set "id": 2.
This way you will see if your application is establishing a connection correctly and if the transferred data is complete.

Receiving messages and sending messages over NetworkStream to a Client

I have a question about sending and receiving messages from a TCPListener server to a client over a Network Stream in C#. Right now, my TCPListener instance can receive one message from a client and write it to the server console, and it can accept one input string and send it back to the client. But I wish to improve the function to accept multiple consecutive messages from a client and send multiple consecutive responses back to the client. Does anyone happen to have any pointers if the ReadAsync and WriteAsync functions of a NetworkStream can handle receiving multiple consecutive messages or sending multiple consecutive messages and if there is a better method to achieve this? Also, since the Console.ReadLine function will block in the case the server never receives any user input from ReadLine (and no Enter key is pushed), is there a way to test if there's optional user input from the keyboard? That way I could try to execute the send message commands only if the server received some kind of console input from the user, and could continue receiving client messages otherwise.
public static async Task getMessage(TcpListener server)
{
byte[] bytes = new byte[256];
using (var theStream = await server.AcceptTcpClientAsync())
{
using (var tcpStream = theStream.GetStream())
{
await tcpStream.ReadAsync(bytes, 0, bytes.Length);
var msg = Encoding.UTF8.GetString(bytes);
Console.WriteLine(msg);
var payload = Console.ReadLine();
var bytes2 = Encoding.UTF8.GetBytes(payload);
await tcpStream.WriteAsync(bytes2, 0, bytes2.Length);
}
}
}

It is interesting to see a server implementation dedicated to only one client!
TCP is a two-way communication protocol so yes, you can, of course, send what you want and when you want and receive anything at the same time.
I tried to put comments in the source to let it explain itself.
But the critical point is declaring the TcpClient and NetworkStream instances as static variables so that the main thread (the thread which reads from the console and sends the payload to the client) can access them
Hope this helps.
public static async Task getMessage(TcpListener server)
{
byte[] bytes = new byte[256];
using (theStream = await server.AcceptTcpClientAsync())
{
using (tcpStream = theStream.GetStream())
{
// We are using an infinite loop which ends when zero bytes have been received.
// Receiving zero bytes means the transmission is over (client disconnected)
while (await tcpStream.ReadAsync(bytes, 0, bytes.Length) > 0)
{
var msg = Encoding.UTF8.GetString(bytes);
Console.WriteLine(msg);
}
Console.WriteLine("Client has disconnected");
}
}
}
/// <summary>
/// Transmists the payload to the client
/// </summary>
/// <param name="payload">The payload to transmit to the client</param>
/// <returns></returns>
static async Task Transmit(string payload)
{
var bytes2 = Encoding.UTF8.GetBytes(payload);
await tcpStream.WriteAsync(bytes2, 0, bytes2.Length);
}
// We are declaring the TcpClient and NetworkStream as static variables
// to be able to access them from all the threads.
private static TcpClient theStream;
public static NetworkStream tcpStream;
static void Main(string[] args)
{
TcpListener server = new TcpListener(IPAddress.Loopback, 9876);
server.Start();
// Start the task getMessage() to accept a client and receive
Task.Run(() => getMessage(server));
string payload;
while ((payload = Console.ReadLine()) != "exit")
{
// Check if the client has connected.
if (tcpStream != null)
{
// Check if they are still connected
if (theStream.Client.Connected)
{
Task.Run(() => Transmit(payload));
}
else
{
Console.WriteLine("the client connection is lost");
break;
}
}
else
{
Console.WriteLine("The client has not connected yet.");
}
}
Console.WriteLine("Stopping the server");
server.Stop();
}

Async await TCP Server for concurrent connection with database calls

I have to write asynchronous TCP Server on to which multiple GPS Devices will be connecting simultaneously (Count :- 1000 Approx) and will push some data of size less than 1 Kb on server, In the response server will send them simple message containing byte received count, The same procedure will happen every 5 Min.
The data received at server is in CSV Format and contains many decimal values, server suppose to process this data and insert the same into database table
After doing lots of Google I decided to go with C#4.5 async and await methods,
This is the first time I am implementing the TCP Server, I believe this is not really the efficient and professional code so any small of small inputs for the same are greatly appreciated. My sample code is as below
// Server starts from Here
public async void Start()
{
IPAddress ipAddre = IPAddress.Parse("192.168.2.5");
TcpListener listener = new TcpListener(ipAddre, _listeningPort);
listener.Start();
while (true)
{
try
{
var tcpClient = await listener.AcceptTcpClientAsync();
HandleConnectionAsync(tcpClient);
}
catch (Exception exp)
{
}
}
}
// Handle the incoming connection and call process data
private async void HandleConnectionAsync(TcpClient tcpClient)
{
try
{
using (var networkStream = tcpClient.GetStream())
using (var reader = new StreamReader(networkStream, Encoding.Default))
using (var writer = new StreamWriter(networkStream))
{
networkStream.ReadTimeout = 5000;
networkStream.WriteTimeout = 5000;
char[] resp = new char[1024];
while (true)
{
var dataFromServer = await reader.ReadAsync(resp, 0, resp.Length);
string dataFromServer1 = new string(resp);
string status = await ProcessDataReceived(dataFromServer1);
if (status.Length != 0)
await writer.WriteAsync(status);
}
}
}
catch (Exception exp){}
}
//Process Data Function
private async Task<string> ProcessDataReceived(string dataFromServer)
{
List<string> values = dataFromServer.Split(',').ToList();
// Do some calculation and rearrange the data
// Create the datatable and insert the data into datatable
using (SqlBulkCopy bulkcopy = new SqlBulkCopy(_dbConn))
{
bulkcopy.WriteToServer(table);
}
return “status”;
}
At present, I have tested it with single GPS Devices and its working for some 10-15 min. than simply crashed and I am very much doubt full about its in the way it will work when there are multiple concurrent connection.
I just want to Make sure whether my basic approach as show in code is in right direction? Am I processing data in correct way or should I suppose to make the use of queue or some other data structure for processing?
Any inputs are greatly appreciable.

The lack of information means that I'm unable to tell whether this is your actual problem, but as far as problems go, this one's a biggy.
So you're not checking if the other end finished. This is indicated by a return value of 0 when calling ReadAsync.
The result value can be less than the number of bytes requested if the number of bytes currently available is less than the requested number, or it can be 0 (zero) if the end of the stream has been reached.
When this condition is detected, you need to get out of the loop, otherwise bad stuff will happen...
while(true)
{
//....
var dataFromServer = await reader.ReadAsync(resp, 0, resp.Length); //bad name!
if(dataFromServer == 0)
{
break;
}
//....
}
As a rule, when you're doing network programming, you need to trap every possible exception and understand what that exception means. Looking at failure in terms of "oh... it crashed" won't get you very far at all. Network stuff fails all the time and you have to understand why it's failing by reading all the diagnostic information you have to hand.

Why is my message divided into multiple frames?

When i send a request to my server or a reply to my client, the message i send is always divided into multiple parts.
So i need to call Receive multiple times to get to the last part/frame.
Why does this happen.. is there a better way of sending and receiving an xml encoded string?
This is the code of my client:
private void SendRequestAsyncTaskStart<T>(object contextObj, T request)
{
ZmqContext context = (ZmqContext)contextObj;
ZmqSocket requestSocket = CreateServerSocket(context);
SerializeAndSendRequest(request, requestSocket);
}
private ZmqSocket CreateServerSocket(ZmqContext context)
{
var client = context.CreateSocket(SocketType.REQ);
client.Connect(_requestReplyEndpoint);
client.Linger = TimeSpan.Zero;
client.ReceiveReady += PollInReplyHandler;
return client;
}
public static string Serialize(this object obj)
{
string result;
using (var memoryStream = new MemoryStream())
{
using (var reader = new StreamReader(memoryStream))
{
var serializer = new DataContractSerializer(obj.GetType());
serializer.WriteObject(memoryStream, obj);
memoryStream.Position = 0;
result = reader.ReadToEnd();
}
}
return result;
}
This is the code of my server:
private void ListenForRequestsThreadStart(object contextObj)
{
ZmqContext context = (ZmqContext)contextObj;
using (
ZmqSocket frontend = context.CreateSocket(SocketType.REP),
backend = context.CreateSocket(SocketType.DEALER))
{
string bindAddress = string.Format("tcp://*:{0}", _listenForRequetsPort);
frontend.Bind(bindAddress);
backend.Bind("inproc://backend");
frontend.ReceiveReady += HandleRequestReceived;
// polling
}
}
private void HandleRequestReceived(object sender, SocketEventArgs e)
{
string message;
bool hasNext;
do
{
message = socket.Receive(Encoding.ASCII);
hasNext = socket.ReceiveMore;
} while (hasNext);
// after calling Receive 3 times i get my actual message
}

Since you're sending via a socket you're at the mercy of the network. First, the network will have broken your message down in multiple packates each of which is received separately by your listener. Every now and then, the underlying socket on the listening machine will say to itself 'Got some incoming, but there's more to come. Wait a bit'. After a while it'll say, 'Oh well, give what I've got' and keep waiting'.
That's what's happening. In WCF, the WCF implementation gets its data via sockets which do exactly the same thing. But WCF waits till the whole message arrives before giving it to your waiting code. That's one of the advantages of using a Framework like WCF. It protects you from the metal.

Any message sent over TCP may be divided into several packets depending on its size. That's why you should never assume to get a message in one go, but read until you're sure you've received everything.