I receive UDP data from localhost. This data needs to be used and kept track of. There could be >250k packets sent over the course of about 5 minutes. Currently, I create the IPEndPoint, and Receive the data. I then get the byte[] data into a string. Then I split the string into an array because the data is comma separated. From this array, I instantiate an object with the correct values from the array.
The incoming data has 4 properties and looks like this: ProductId=1284,Quantity=48623,Time=08:45:12.034,Date=2019-09-09
Currently I am able to Console.WriteLine the incoming packets, and cast each packet to an object (Product). Currently I do not store the data anywhere.
I need to monitor Quantity levels, and send alerts if the Quantity goes above or below certain thresholds. Secondly, I need to send another alert if the change in Quantity exceeds thresholds. This I will implement with if statements later and do not require assistance with at the moment. This alerts do need to be sent as UDP packets are received.
My questions are:
1) Should I implement this Asynchronously, or on a different thread?
2) Should I be sending the UDP Data directly to a Queue of type byte[], and then Dequeue each byte[] into an object? or keep instantiating new objects (or updating the quantity for each product) on the fly as the UDP data is received? Does Queue reduce the risk of dropped packets?
3) What is the best way to store each packet? To a List (This would keep everything in memory), a database (A bit beyond my current needs), or append each object to a text file (easy to write to, and can eventually build a database from)?
4) Is 250k packets over 5 minutes, and creating objects for all those packets considered a lot for a computer to handle?
Please advise if I can clarify anything. I apologize if any of my terminology is incorrect, I will learn and fix it. The coding I should be able to figure out on my own, it is the design and proper implementation that I am asking for guidance on. Thank you.
Below is code for setting up the UDP connection, and the class. I will be able to figure out the monitoring/messaging Quantity information after the UDP has been setup properly. I do have a Product Class not shown, it has several properties including ProductId, Quantity, Initial Quantity, Change in Quantity, Time and Date.
class Program
{
static void Main(string[] args)
{
int port = 22486;
var client = new UdpClient();
IPEndPoint localEp = new IPEndPoint(IPAddress.Any, port);
client.Client.Bind(localEp);
while (true)
{
try
{
byte[] data = client.Receive(ref localEp);
string text = Encoding.UTF8.GetString(data);
string[] message = text.Split(',', '=');
Product product = new Product();
product.ProductId = message[1];
product.Quantity = message[3];
}
catch (Exception err)
{
Console.WriteLine(err.ToString());
}
}
}
}
If you need to do other things while listening on that port, make it a thread.
queue it up in a byte array... a buffer of sorts.
Are azure blobs an option for you? you can use the storage emulator and if you change your mind about storing a file in blob, you can switch to table storage (this is like a nosql approach where you provide a partition name.)
No
Related
Okay from my knowledge UDP works like this:
You have data you want to send, you say to the UDP client, hey send this data.
The UDP client then says, sure why not, and sends the data to the selected IP and Port.
If it get´s through or in the right order is another story, it have sent the data, you didn´t ask for anything else.
Now from this perspective, it´s pretty much impossible to send data and assemble it.
for example, i have a 1mb image, and i send it.
So i send divide it in 60kb files (or something to fit the packages), and send them one by one from first to last.
So in theory, if all get´s added, the image should be exactly the same.
But, that theory breaks as there is no law that tells the packages if it can arrive faster or slower than another, so it may only be possible if you make some kind of wait timer, and hope for the best that the arrive in the order they are sent.
Anyway, what i want to understand is, why does this work:
void Sending(object sender, NAudio.Wave.WaveInEventArgs e)
{
if (connect == true && MuteMic.Checked == false)
{
udpSend.Send(e.Buffer, e.BytesRecorded, otherPartyIP.Address.ToString(), 1500);
}
}
Recieving:
while (connect == true)
{
byte[] byteData = udpReceive.Receive(ref remoteEP);
waveProvider.AddSamples(byteData, 0, byteData.Length);
}
So this is basically, it sends the audio buffer through udp.
The receiving par just adds the udp data received in a buffer and plays it.
Now, this works.
And i wonder.. why?
How can this work, how come the data is sent in the right order and added so it appears as a constant audio stream?
Cause if i would to this with an image, i would probably get all the data.
But they would be in a random order probably, and i can only solve that by marking packages and stuff like that. And then there is simply no reason for it, and TCP takes over.
So if someone can please explain this, i just don´t get it.
Here is a code example that is when sending an image, and well, it works. But it seems to work better when the entire byte array isn´t sent, meanign some part of the image is corrupted (not sure why, probably something to do with how the size of the byte array are).
Send:
using (var udpcap = new UdpClient(0))
{
udpcap.Client.SendBufferSize *= 16;
bsize = ms.Length;
var buff = new byte[7000];
int c = 0;
int size = 7000;
for (int i = 0; i < ms.Length; i += size)
{
c = Math.Min(size, (int)ms.Length - i);
Array.Copy(ms.GetBuffer(), i, buff, 0, c);
udpcap.Send(buff, c, adress.Address.ToString(), 1700);
}
Receive:
using (var udpcap = new UdpClient(1700))
{
udpcap.Client.SendBufferSize *= 16;
var databyte = new byte[1619200];
int i = 0;
for (int q = 0; q < 11; ++q)
{
byte[] data = udpcap.Receive(ref adress);
Array.Copy(data, 0, databyte, i, data.Length);
i += data.Length;
}
var newImage = Image.FromStream(new MemoryStream(databyte));
gmp.DrawImage(newImage,0,0);
}
You should be using TCP. You write: it´s pretty much impossible to send data and assemble it. for example, i have a 1mb image, and i send it. So i send divide it in 60kb files (or something to fit the packages), and send them one by one from first to last. ... But, that theory breaks as there is no law that tells the packages if it can arrive faster or slower than another, so it may only be possible if you make some kind of wait timer, and hope for the best that the arrive in the order they are sent. That's exactly what TCP does: ensure that all the pieces of a stream of data are received in the order they were sent, with no omissions, duplications, or modifications. If you really want to re-implement that yourself, you should be reading RFC 793 - it talks at length about how to build a reliable data stream atop an unreliable packet service.
But really, just use TCP.
You're missing a lot of helpful details from your question, but based on the level of understanding presented I'll attempt to answer at a similar level:
You're absolutely right, in general the UDP protocol doesn't guarantee order of delivery or even delivery at all. Your local host is going to send the packets (i.e. parts of your request message) in the order it receives them from the sending application, and from there its up to network components to decide how your request gets delivered. In local networks however (within a handful of hops of the original requester) there aren't really a lot of directions for the packets to go. As such they will likely just flow in line and never see a hiccup.
On the greater internet however, there is likely a wide variety of routing choices available to each router between your requesting host and the destination. Every router along the way can make a choice on which direction parts of your message take. Assuming all paths are equal (which they aren't) and guaranteed reliability of every network segment between the 2 hosts its likely to see similar results as within network (with added latency). Unfortunately neither of the posed conditions can be considered reliable (different paths on the internet perform differently depending on client and server, and no single path on the internet should ever be considered to be reliable (that's why it's a "net").
These are of course based on general observations from my own experience in network support and admin roles. Members of other StackExchange sites may be able to provide better feedback.
My Context
I have a TCP networking program that sends large objects that have been serialized and encoded into base64 over a connection. I wrote a client library and a server library, and they both use NetworkStream's Begin/EndReadandBegin/EndWrite. Here's the (very much simplified version of the) code I'm using:
For the server:
var Server = new TcpServer(/* network stuffs */);
Server.Connect();
Server.OnClientConnect += new ClientConnectEventHandler(Server_OnClientConnect);
void Server_OnClientConnect()
{
LargeObject obj = CalculateLotsOfBoringStuff();
Server.Write(obj.SerializeAndEncodeBase64());
}
Then the client:
var Client = new TcpClient(/* more network stuffs */);
Client.Connect();
Client.OnMessageFromServer += new MessageEventHandler(Client_OnMessageFromServer);
void Client_OnMessageFromServer(MessageEventArgs mea)
{
DoSomethingWithLargeObject(mea.Data.DecodeBase64AndDeserialize());
}
The client library has a callback method for NetworkStream.BeginRead which triggers the event OnMessageFromServer that passes the data as a string through MessageEventArgs.
My Problem
When receiving large amounts of data through BeginRead/EndRead, however, it appears to be fragmented over multiple messages. E.G. pretend this is a long message:
"This is a really long message except not because it's for explanatory purposes."
If that really were a long message, Client_OnMessageFromServer might be called... say three times with fragmented parts of the "long message":
"This is a really long messa"
"ge except not because it's for explanatory purpos"
"es."
Soooooooo.... takes deep breath
What would be the best way to have everything sent through one Begin/EndWrite to be received in one call to Client_OnMessageFromServer?
You can't. On TCP, how things arrive is not necessarily the same as how they were sent. It the job of your code to know what constitutes a complete message, and if necessary to buffer incoming data until you have a complete message (taking care not to discard the start of the next message I the process).
In text protocols, this usually means "spot the newline / nul-char". For binary, it usually means "read the length-header in the preamble the the message".
TCP is a stream protocol, and has no fixed message boundaries. This means you can receive part of a message or the end of one and the beginning of another.
There are two ways to solve this:
Alter your protocol to add end-of-message markers. This way you continuously receive until you find the special marker. This can however lead that you have a buffer containing the end of one message and the beginning of another which is why I recommend the next way.
Alter protocol to first send the length of the message. Then you will know exactly how long the message is, and can count down while receiving so you won't read the beginning of the next message.
To start I am coding in C#. I am writing data of varying sizes to a device through a socket. After writing the data I want to read from the socket because the device will write back an error code/completion message once it has finished processing all of the data. Currently I have something like this:
byte[] resultErrorCode = new byte[1];
resultErrorCode[0] = 255;
while (resultErrorCode[0] == 255)
{
try
{
ReadFromSocket(ref resultErrorCode);
}
catch (Exception)
{
}
}
Console.WriteLine(ErrorList[resultErrorCode[0] - 48]);
I use ReadFromSocket in other places, so I know that it is working correctly. What ends up happening is that the port I am connecting from (on my machine) changes to random ports. I think that this causes the firmware on the other side to have a bad connection. So when I write data on the other side, it tries to write data to the original port that I connected through, but after trying to read several times, the connection port changes on my side.
How can I read from the socket continuously until I receive a completion command? If I know that something is wrong with the loop because for my smallest test file it takes 1 min and 13 seconds pretty consistently. I have tested the code by removing the loop and putting the code to sleep for 1 min and 15 seconds. When it resumes, it successfully reads the completion command that I am expecting. Does anyone have any advice?
What you should have is a separate thread which will act like a driver of your external hardware. This thread will receive all data, parse it and transmit the appropriate messages to the rest of your application. This portion of code will give you an idea of how receive and parse data from your hardware.
public void ContinuousReceive(){
byte[] buffer = new byte[1024];
bool terminationCodeReceived = false;
while(!terminationCodeReceived){
try{
if(server.Receive(buffer)>0){
// We got something
// Parse the received data and check if the termination code
// is received or not
}
}catch (SocketException e){
Console.WriteLine("Oops! Something bad happened:" + e.Message);
}
}
}
Notes:
If you want to open a specific port on your machine (some external hardware are configured to talk to a predefined port) then you should specify that when you create your socket
Never close your socket until you want to stop your application or the external hardware API requires that. Keeping your socket open will resolve the random port change
using Thread.Sleep when dealing with external hardware is not a good idea. When possible, you should either use events (in case of RS232 connections) or blocking calls on separate threads as it is the case in the code above.
i have a C# Winform that is accepting UDP Packets from several devices a single port.
The devices sent UDP packets to me at a set interval and i want to implement a way to know when a device has stopped sending packets.
I use a single UDPClient and using the Receive function. When data is received, i pass the RemoteIPEndPoint back to my mainform to update values.
What would be the best way to this?
Thanks for the help!
This is a little abstract questions and there could be many solutions to that. One simple and quick solution would be to create a HashTable where,
HashKey = IP of Remote device
Value = Timestamp when last packet received from that device.
Now whenever you receive a packet you just update the hashtable like
if (hashTable contains "RemoteEndPoint")
{
hashTable [ remoteEndPoint ] = now() // current Time.
}
else
{
// if you want to add/register new device to your list, do it here
}
Beside that you can just run a Timer with some interval that traverse the HashTable and check if some TimeStamp is less then (currentTime - Your Set Interval), then you can say that you haven't received the data from that end point.
I'm sending a large amount of data in one go between a client and server written C#. It works fine when I run the client and server on my local machine but when I put the server on a remote computer on the internet it seems to drop data.
I send 20000 strings using the socket.Send() method and receive them using a loop which does socket.Receive(). Each string is delimited by unique characters which I use to count the number received (this is the protocol if you like). The protocol is proven, in that even with fragmented messages each string is correctly counted. On my local machine I get all 20000, over the internet I get anything between 17000-20000. It seems to be worse the slower connection that the remote computer has. To add to the confusion, turning on Wireshark seems to reduce the number of dropped messages.
First of all, what is causing this? Is it a TCP/IP issue or something wrong with my code?
Secondly, how can I get round this? Receiving all of the 20000 strings is vital.
Socket receiving code:
private static readonly Encoding encoding = new ASCIIEncoding();
///...
while (socket.Connected)
{
byte[] recvBuffer = new byte[1024];
int bytesRead = 0;
try
{
bytesRead = socket.Receive(recvBuffer);
}
catch (SocketException e)
{
if (! socket.Connected)
{
return;
}
}
string input = encoding.GetString(recvBuffer, 0, bytesRead);
CountStringsIn(input);
}
Socket sending code:
private static readonly Encoding encoding = new ASCIIEncoding();
//...
socket.Send(encoding.GetBytes(string));
If you're dropping packets, you'll see a delay in transmission since it has to re-transmit the dropped packets. This could be very significant although there's a TCP option called selective acknowledgement which, if supported by both sides, it will trigger a resend of only those packets which were dropped and not every packet since the dropped one. There's no way to control that in your code. By default, you can always assume that every packet is delivered in order for TCP and if there's some reason that it can't deliver every packet in order, the connection will drop, either by a timeout or by one end of the connetion sending a RST packet.
What you're seeing is most likely the result of Nagle's algorithm. What it does is instead of sending each bit of data as you post it, it sends one byte and then waits for an ack from the other side. While it's waiting, it aggregates all the other data that you want to send and combines it into one big packet and then sends it. Since the max size for TCP is 65k, it can combine quite a bit of data into one packet, although it's extremely unlikely that this will occur, particularly since winsock's default buffer size is about 10k or so (I forget the exact amount). Additionally, if the max window size of the receiver is less than 65k, it will only send as much as the last advertised window size of the receiver. The window size also affects Nagle's algorithm as well in terms of how much data it can aggregate prior to sending because it can't send more than the window size.
The reason you see this is because on the internet, unlike your network, that first ack takes more time to return so Naggle's algorithm aggregates more of your data into a single packet. Locally, the return is effectively instantaneous so it's able to send your data as quickly as you can post it to the socket. You can disable Naggle's algorithm on the client side by using SetSockOpt (winsock) or Socket.SetSocketOption (.Net) but I highly recommend that you DO NOT disable Naggling on the socket unless you are 100% sure you know what you're doing. It's there for a very good reason.
Well there's one thing wrong with your code to start with, if you're counting the number of calls to Receive which complete: you appear to be assuming that you'll see as many Receive calls finish as you made Send calls.
TCP is a stream-based protocol - you shouldn't be worrying about individual packets or reads; you should be concerned with reading the data, expecting that sometimes you won't get a whole message in one packet and sometimes you may get more than one message in a single read. (One read may not correspond to one packet, too.)
You should either prefix each method with its length before sending, or have a delimited between messages.
It's definitely not TCP's fault. TCP guarantees in-order, exactly-once delivery.
Which strings are "missing"? I'd wager it's the last ones; try flushing from the sending end.
Moreover, your "protocol" here (I'm taking about the application-layer protocol you're inventing) is lacking: you should consider sending the # of objects and/or their length so the receiver knows when he's actually done receiving them.
How long are each of the strings? If they aren't exactly 1024 bytes, they'll be merged by the remote TCP/IP stack into one big stream, which you read big blocks of in your Receive call.
For example, using three Send calls to send "A", "B", and "C" will most likely come to your remote client as "ABC" (as either the remote stack or your own stack will buffer the bytes until they are read). If you need each string to come without it being merged with other strings, look into adding in a "protocol" with an identifier to show the start and end of each string, or alternatively configure the socket to avoid buffering and combining packets.