Develop Reference

UDP packets from a field device will not through azure infrastructure to my service

A field device (Loxone miniserver) is sending measurements in UDP packets to the public adress of an azure VM and to my local machine, both on port 1234(changed for integrity). On my local machine in the same network as the field device I implemented a test receiver in C# which gets the packets correctly.
On an already running application on an azure VM I added the same receiver code as in the local machine and got no UDP packets.
What I already did on the azure management platform:
- Allowed the incoming port 1234 in the network configuration
- Switched off the firewall
For testing I tried following:
- Implemented a UDP sender on the application in the azure VM which sends on broadcast-IP to port 1234 -> this works!
- Added a packet monitor to the network-interface -> here I see both packages, the internal broadcast and the field device
//Receiver-Code reduced to relevant parts
class Program{
public static void Main(string[] args){
program.Run();
}
public void Run(){
var sync = new LoxoneSyncProcess();
while(true){
sync.readUDP();
Thread.sleep(10000);
}
}
}
class LoxoneSyncProcess{
public LoxoneSyncProcess(){
private UDPSocket c = new UDPSocket();
c.Client(1234);
}
public void readUDP(){
String buffer = c.getDatBuf();
Console.WriteLine(buffer);
//process buffer
}
}
public class UDPSocket{
private Socket _socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
private const int bufSize = 8 * 1024;
private State state = new State();
private EndPoint epFrom = new IPEndPoint(IPAddress.Any, 0);
private AsyncCallback recv = null;
private String datBuf = "";
public class State
{
public byte[] buffer = new byte[bufSize];
}
public String getDatBuf()
{
String temp = datBuf;
datBuf = "";
return temp;
}
public void Client(int port)
{
_socket.Bind(new IPEndPoint(IPAddress.Any, port));
Receive();
}
private void Receive()
{
_socket.BeginReceiveFrom(state.buffer, 0, bufSize, SocketFlags.None, ref epFrom, recv = (ar) =>
{
State so = (State)ar.AsyncState;
int bytes = _socket.EndReceiveFrom(ar, ref epFrom);
_socket.BeginReceiveFrom(so.buffer, 0, bufSize, SocketFlags.None, ref epFrom, recv, so);
datBuf += Encoding.ASCII.GetString(so.buffer, 0, bytes);
//Console.WriteLine("RECV: {0}: {1}, {2}", epFrom.ToString(), bytes, Encoding.ASCII.GetString(so.buffer, 0, bytes));
}, state);
}
}
The code is working, I receive self sent UDP packets to port 1234 on Broadcast-IP, but not from the field device.
Are there any tables like NAT-configurations needed? Or portforewarding? Does azure support public UDP endpoints?

UDP public Endpoint is supported in Azure. Just wanted to check the size of the UDP payload that you use. If the UDP payload is > 1500 bytes then it will not work.
This is by design at the moment as we do not support fragments to public VIP endpoints right now in the vswitch. The recommendation is for customers to keep UDP datagrams to under 1500. There’s really nothing short term we could do to allow fragments through without code changes – they’re fundamentally not supported in the NAT

One thing to note is the mentioning of fragmentation according to Microsoft.
Granted the information is about TCP/IP but MTU obviously affects UDP traffic as well.
Azure and VM MTU
The default MTU for Azure VMs is 1,500 bytes. The Azure Virtual Network stack >will attempt to fragment a packet at 1,400 bytes.
Note that the Virtual Network stack isn't inherently inefficient because it >fragments packets at 1,400 bytes even though VMs have an MTU of 1,500. A >large percentage of network packets are much smaller than 1,400 or 1,500 >bytes.
Source https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-tcpip-performance-tuning

python / C# UDP hole punch

I have tried implementing UDP hole punch from this theory:
A rendezvous server (S) is reachable on UDP by client (A) and client (B).
Clients A and B send a registration package each to S. S reads public IP and outer port and stores it.
S sends A-info (endPointA) to B and B-info (endPointB) to A.
With this information A can send to B's endPoint and B can send to A's endPoint.
server S is python 3 and clients A and B are C#
So far the theory. Here is what works:
S can send as many messages to A and B as it likes on their public endPoint.
A and B can send to each other on innerIP:localPort if they are behind the same router ONLY!
Here is what does not work:
A cannot send to B nor B to A with their public endPointA / endPointB
even if I create a new socket on the server S, it cannot get a message through to these endPoints
This last limitation makes me think there is some "lock" on routers that only allow the original socket on S to send to the clients(?)
So my question is: what now?
Is there some socket option, socket flag, socket anything that needs to be set to fix this?
I have even tried making a script on the server S that sends to the clients public IP on all ports from 32000-65535 with no messages coming through.
Code for server:
def ServerUdpStart(host, port):
"""Start up the server"""
global myServerUdp
success = True
try :
myServerUdp = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
except socket.error as msg:
print('UDP Failed to create socket. Error Code:', str(msg[0]), 'Message', msg[1])
success = False
# Bind socket to local host and port
try:
myServerUdp.bind( (host, port) )
except socket.error as msg:
print('UDP bind failed. Error Code : ', str(msg[0]), 'Message', msg[1])
success = False
return success
def ServerUdpReceiveData():
"""Incoming UDP data (runs in a thread)"""
global myServerUdp
if myServerUdp != None:
d = myServerUdp.recvfrom(1024)
data = d[0]
addr = d[1] # source tuple (ip, port) to store for clients
if data:
incomingUdpMessage(addr, data) # handle incoming data
Code for a client:
public class SocketServerUdpHandler{
// ports
private int _port = 0;
private int _localAssignedPort = 0;
// udp client
private UdpClient _clientUdp = null;
private IPEndPoint recvEndP = new IPEndPoint( IPAddress.Any, 0);
//********** START **********
public bool StartUdp(int po){
// port set by user
_port = po;
// client
bool created = CreateClient();
// start it up
if(created){
FindLocalAssignedPort();
StartListen();
}
return created;
}
//********** LISTENER **********
private bool CreateClient(){
// state
bool success = true;
try{
// udp client
_clientUdp = new UdpClient();
}catch(Exception e){
success = false;
}
return success;
}
private void StartListen(){
// listen
_clientUdp.BeginReceive(new AsyncCallback(ReceiveCallback), null);
}
private void ReceiveCallback(IAsyncResult ar){
// read incoming
byte[] receiveBytes = _clientUdp.EndReceive(ar, ref recvEndP);
// handle bytes ...
// listen
StartListen();
}
private void FindLocalAssignedPort(){
IPEndPoint tmpEndP = new IPEndPoint( IPAddress.Parse("127.0.0.1"), 10001 );
byte[] msgByte = new byte[] {(byte) 9};
_clientUdp.Client.SendTo(msgByte, tmpEndP);
_localAssignedPort = ((IPEndPoint) _clientUdp.Client.LocalEndPoint).Port;
}
//********** SEND TO **********
public void writeSocketToEndPoint(IPEndPoint ipEndP, byte[] bytes){
_clientUdp.Client.SendTo(bytes, ipEndP);
}
}

Not multicast packets sent over network returns

Good day all.
My software uses a multicast group to send out messages, however the application acts as a client and server, thus it sends out udp packets and also receives them.
This is tested with text containing 2 000 char's.
I create a queue containing byte[1024] arrays containing the data which needs to be sent.
Problem:
The problem lies at receiving these multicast packets, out of 5 packets sent, I mostly receive only 1, regularly 2, etc, and handful of times I received all of them.
Sending and receiving results are as follows:
packets sent: 1, 2, 3, 4, 5 are sent
packets returned: 1, 2, 3, 4, F (F is last packet in the sequence, i.e. closing packet)
12345
134F
12345
1F
12345
134F
12345
14F
12345
1F
12345
1F
12345
1
12345
1
12345
1
12345
1234F
12345
1F
12345
1
12345
1
12345
1
12345
14F
12345
1F
12345
1F
12345
1
12345
1
12345
1
12345
1F
12345
1
Sending a multicast packet as follows:
Socket _listener_socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
foreach (IPAddress localIP in Dns.GetHostAddresses(Dns.GetHostName()).Where(i => i.AddressFamily == AddressFamily.InterNetwork))
{
//handle image and files
_listener_socket.SetSocketOption(SocketOptionLevel.IP, SocketOptionName.AddMembership, new MulticastOption(_MultiIP, localIP));
_listener_socket.SetSocketOption(SocketOptionLevel.IP, SocketOptionName.MulticastTimeToLive, 1);
_listener_socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
_listener_socket.MulticastLoopback = true;
_listener_socket.Connect(new IPEndPoint(_MultiIP, _PORT));
while (count > 0)
{
count--;
byte[] temp = (byte[])(MSGS_TO_SEND.Dequeue());
_listener_socket.Send(temp, _BYTE_BUFFER_SIZE, SocketFlags.None);
MSGS_TO_SEND.Enqueue(temp);
}
}
_listener_socket.Close();
One thread is for receiving packets and add them to a Queue for processing, and a second thread is for processing. Reason for 2 threads was possibly too much processing in the receiving thread was causing this "packet loss", but unfortunately it did not solve the problem
Receive Thread:
//initialise multicast group and bind to interface
Socket _sender_socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
IPEndPoint ipep = new IPEndPoint(IPAddress.Any, _PORT);
_sender_socket.Bind(ipep);
IPAddress localip = _MultiIP;
_sender_socket.SetSocketOption(SocketOptionLevel.IP, SocketOptionName.AddMembership, new MulticastOption(localip, IPAddress.Any));
Q_RECIEVE = new Queue<char[]>();
while (_sender_socket.IsBound && !bStop)
{
byte[] b = new byte[_BYTE_BUFFER_SIZE];
_sender_socket.Receive(b);
char[] chars = new char[_BYTE_BUFFER_SIZE];
System.Buffer.BlockCopy(b, 0, chars, 0, b.Length);
Q_RECIEVE.Enqueue(chars);
}

There are reliable UDP protocols and libraries for them. I've used UDT (http://udt.sourceforge.net/) before and it worked fine. I'm not aware of a C# wrapper for it, but there might be one or another reliable UDP C# implementation.
Other than that, here are some other suggestions:
Multicasting is mostly problematic on the switch(es). Try to change them, if you can. Sometimes it can be a problem on the routers, too, so, you may try to change them, too, if possible.
Try sending less packets (or less often). Packet length should not be that much of a problem if you're below 1500 bytee, but, you can also try to decrease the length.
To be sure that you're not missing packets in your code for some reason, use a packet sniffer and see if the packets actually arrived at your destination computer. WireShark has a nice enough GUI, but there are several GUI and CLI based ones out there.

Where is data sent by UDP stored?

I had never used UDP before, so I gave it a go. To see what would happen, I had the 'server' send data every half a second, and the client receive data every 3 seconds. So even though the server is sending data much faster than the client can receive, the client still receives it all neatly one by one.
Can anyone explain why/how this happens? Where is the data buffered exactly?
Send
class CSimpleSend
{
CSomeObjServer obj = new CSomeObjServer();
public CSimpleSend()
{
obj.changedVar = varUpdated;
obj.threadedChangeSomeVar();
}
private void varUpdated(int var)
{
string send = var.ToString();
byte[] packetData = System.Text.UTF8Encoding.UTF8.GetBytes(send);
string ip = "127.0.0.1";
int port = 11000;
IPEndPoint ep = new IPEndPoint(IPAddress.Parse(ip), port);
Socket client = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
client.SendTo(packetData, ep);
Console.WriteLine("Sent Message: " + send);
Thread.Sleep(100);
}
}
All CSomeObjServer does is increment an integer by one every half second
Receive
class CSimpleReceive
{
CSomeObjClient obj = new CSomeObjClient();
public Action<string> showMessage;
Int32 port = 11000;
UdpClient udpClient;
public CSimpleReceive()
{
udpClient = new UdpClient(port);
showMessage = Console.WriteLine;
Thread t = new Thread(() => ReceiveMessage());
t.Start();
}
private void ReceiveMessage()
{
while (true)
{
//Thread.Sleep(1000);
IPEndPoint remoteIPEndPoint = new IPEndPoint(IPAddress.Any, port);
byte[] content = udpClient.Receive(ref remoteIPEndPoint);
if (content.Length > 0)
{
string message = Encoding.UTF8.GetString(content);
if (showMessage != null)
showMessage("Recv:" + message);
int var_out = -1;
bool succ = Int32.TryParse(message, out var_out);
if (succ)
{
obj.alterSomeVar(var_out);
Console.WriteLine("Altered var to :" + var_out);
}
}
Thread.Sleep(3000);
}
}
}
CSomeObjClient stores the variable and has one function (alterSomeVar) to update it
Ouput:
Sent Message: 1
Recv:1
Altered var to :1
Sent Message: 2
Sent Message: 3
Sent Message: 4
Sent Message: 5
Recv:2
Altered var to :2
Sent Message: 6
Sent Message: 7
Sent Message: 8
Sent Message: 9
Sent Message: 10
Recv:3
Altered var to :3

The operating system kernel maintains separate send and receive buffers for each UDP and TCP socket. If you google SO_SNDBUF and SO_RCVBUF you'll find lots of information about them.
When you send data, it is copied from your application space into the send buffer. From there it is copied to the network interface card, and then onto the wire. The receive side is the reverse: NIC to receive buffer, where it waits until you read it. Additionally copies and buffering can also occur, depending on the OS.
It is critical to note that the sizes of these buffers can vary radically. Some systems might default to as little as 4 kilobytes, while others give you 2 megabytes. You can find the current size using getsockopt() with SO_SNDBUF or SO_RCVBUF and likewise set it using setsockopt(). But many systems limit the size of the buffer, sometimes to arbitrarily small amounts. This is typically a kernel value like net.core.wmem_max or net.core.rmem_max, but the exact reference will vary by system.
Also note that setsockopt() can fail even if you request an amount less than the supposed limit. So to actually get a desired size, you need to repeatedly call setsockopt() using decreasing amounts until it finally succeeds.
The following page is a Tech Note from my company which touches on this topic a little bit and provides references for some common systems: http://www.dataexpedition.com/support/notes/tn0024.html

It looks to me like the UdpClient-Class provides a buffer for received data. Try using a socket directly. You might also want to set that sockets ReceiveBufferSize to zero, even though I believe it is only used for TCP connections.

Socket ReceiveAll

I am trying to capture ip packets in c#.
Everything is working fine, except that i only get outgoing packets.
My Code:
using (Socket sock = new Socket(AddressFamily.InterNetwork, SocketType.Raw, ProtocolType.IP))
{
sock.Bind(new IPEndPoint(MYADDRESS, 0));
sock.SetSocketOption(SocketOptionLevel.IP, SocketOptionName.HeaderIncluded, true);
sock.IOControl(IOControlCode.ReceiveAll, BitConverter.GetBytes(1), null);
while (true)
{
byte[] buffer = new byte[sock.ReceiveBufferSize];
int count = sock.Receive(buffer);
// ...
}
}
The problem is definitely my pc! But maybe there is a workaround ...

It is likely that that the Windows firewall is blocking incoming packets. Add your program to the firewall exclusions.

What about a different approach like using WinPcap for .Net with SharpPcap (more info)
It provides an API for capturing, injecting, analyzing and building packets using any .NET language such as C# and VB.NET
....sounds more promising

I believe the problem is that you are binding to the loopback IP, assuming that 'LOCALHOST' in your code implies 127.0.0.1. Try binding to the IP address of the interface you want to capture the packets for.
I took your code an did a quick test, and definately I see data flowing in both directions, using Windows 7. NB I am running this as Administrator, not sure how well it works otherwise.
using (Socket sock = new Socket(AddressFamily.InterNetwork, SocketType.Raw, ProtocolType.IP))
{
sock.Bind(new IPEndPoint(IPAddress.Parse("192.168.0.121"), 0));
sock.SetSocketOption(SocketOptionLevel.IP, SocketOptionName.HeaderIncluded, true);
sock.IOControl(IOControlCode.ReceiveAll, BitConverter.GetBytes(1), null);
while (true)
{
byte[] buffer = new byte[sock.ReceiveBufferSize];
int count = sock.Receive(buffer);
IpHeader hdr = IpHeader.FromPacket(buffer, count);
if ((ProtocolType)hdr.Protocol == ProtocolType.Tcp)
{
Console.WriteLine("{0} : {1} -> {2}", (ProtocolType)hdr.Protocol, new IPAddress(hdr.SrcAddr).ToString(), new IPAddress(hdr.DestAddr).ToString());
}
}
}
IpHeader is from a library I wrote years ago, I used that to quickly decode the packets to ensure I was seeing data in both directions.
Here is a quick capture from the code above to verify (AA.BB.CC.DD is my public IP)
Tcp : 83.221.14.72 -> AA.BB.CC.DD
Tcp : AA.BB.CC.DD -> 83.221.14.72
Tcp : 83.221.14.72 -> AA.BB.CC.DD
Tcp : 83.221.14.72 -> AA.BB.CC.DD
Tcp : AA.BB.CC.DD -> 83.221.14.72
Tcp : 83.221.14.72 -> AA.BB.CC.DD
Tcp : 83.221.14.72 -> AA.BB.CC.DD
Tcp : AA.BB.CC.DD -> 83.221.14.72
Tcp : AA.BB.CC.DD -> 83.221.14.72
Tcp : AA.BB.CC.DD -> 83.221.14.72
Tcp : 83.221.14.72 -> AA.BB.CC.DD
Tcp : 83.221.14.72 -> AA.BB.CC.DD
Tcp : AA.BB.CC.DD -> 83.221.14.72

In my case I had to allow the the vshost.exe in the windows firewall

I think the problem is in the IOControl call. To be honest I don't really understand the documentation provided by MSDN about this function but there is an example in codeproject (here) about this topic, and my guess is that by passing null in the last parameter you are not getting the incoming packets.
Try this instead:
byte[] byTrue = new byte[4]{1, 0, 0, 0};
byte[] byOut = new byte[4];
sock.IOControl(IOControlCode.ReceiveAll, byTrue, byOut);