I have to write asynchronous TCP Server on to which multiple GPS Devices will be connecting simultaneously (Count :- 1000 Approx) and will push some data of size less than 1 Kb on server, In the response server will send them simple message containing byte received count, The same procedure will happen every 5 Min.
The data received at server is in CSV Format and contains many decimal values, server suppose to process this data and insert the same into database table
After doing lots of Google I decided to go with C#4.5 async and await methods,
This is the first time I am implementing the TCP Server, I believe this is not really the efficient and professional code so any small of small inputs for the same are greatly appreciated. My sample code is as below
// Server starts from Here
public async void Start()
{
IPAddress ipAddre = IPAddress.Parse("192.168.2.5");
TcpListener listener = new TcpListener(ipAddre, _listeningPort);
listener.Start();
while (true)
{
try
{
var tcpClient = await listener.AcceptTcpClientAsync();
HandleConnectionAsync(tcpClient);
}
catch (Exception exp)
{
}
}
}
// Handle the incoming connection and call process data
private async void HandleConnectionAsync(TcpClient tcpClient)
{
try
{
using (var networkStream = tcpClient.GetStream())
using (var reader = new StreamReader(networkStream, Encoding.Default))
using (var writer = new StreamWriter(networkStream))
{
networkStream.ReadTimeout = 5000;
networkStream.WriteTimeout = 5000;
char[] resp = new char[1024];
while (true)
{
var dataFromServer = await reader.ReadAsync(resp, 0, resp.Length);
string dataFromServer1 = new string(resp);
string status = await ProcessDataReceived(dataFromServer1);
if (status.Length != 0)
await writer.WriteAsync(status);
}
}
}
catch (Exception exp){}
}
//Process Data Function
private async Task<string> ProcessDataReceived(string dataFromServer)
{
List<string> values = dataFromServer.Split(',').ToList();
// Do some calculation and rearrange the data
// Create the datatable and insert the data into datatable
using (SqlBulkCopy bulkcopy = new SqlBulkCopy(_dbConn))
{
bulkcopy.WriteToServer(table);
}
return “status”;
}
At present, I have tested it with single GPS Devices and its working for some 10-15 min. than simply crashed and I am very much doubt full about its in the way it will work when there are multiple concurrent connection.
I just want to Make sure whether my basic approach as show in code is in right direction? Am I processing data in correct way or should I suppose to make the use of queue or some other data structure for processing?
Any inputs are greatly appreciable.
The lack of information means that I'm unable to tell whether this is your actual problem, but as far as problems go, this one's a biggy.
So you're not checking if the other end finished. This is indicated by a return value of 0 when calling ReadAsync.
The result value can be less than the number of bytes requested if the number of bytes currently available is less than the requested number, or it can be 0 (zero) if the end of the stream has been reached.
When this condition is detected, you need to get out of the loop, otherwise bad stuff will happen...
while(true)
{
//....
var dataFromServer = await reader.ReadAsync(resp, 0, resp.Length); //bad name!
if(dataFromServer == 0)
{
break;
}
//....
}
As a rule, when you're doing network programming, you need to trap every possible exception and understand what that exception means. Looking at failure in terms of "oh... it crashed" won't get you very far at all. Network stuff fails all the time and you have to understand why it's failing by reading all the diagnostic information you have to hand.
Related
I'm learning C# and I've got a TCP server that works asynchronously and supports multiple connected clients that I'm having trouble keeping a consistent flow to a method processData(String) . I think I've managed to trace the issue down to the byte buffer where if I receive data from multiple clients (also if a single client sends the same data multiple times in quick succession).
I've tried using an isWorking bool to prevent processData() from being accessed and that threw a JSONReader Exception because I think the buffer was sending the incomplete string from processing. At this point I tried sending a EOF at the end of a string and having the processData() accessed when the check was true. However, this presented another problem where the buffer would contain half of one string and the beginning of another - at this point I posted on here thinking there may be a better approach to ensure properly formatted Strings were sent to the processData() method.
I also tried splitting the response into an array using the } as a pointer but this didn't work resulting in the same half of one string and the beginning of another issue.
string data = content.Replace("[EOF]", "");
string[] TooManyDataObjects = data.Split('}');
string firstObject = TooManyDataObjects[0] += "}";
Here's the code I use to create a listen task and the handleDevice() method when a device connects.
private async Task Listen()
{
try
{
while (true)
{
Token.ThrowIfCancellationRequested();
TcpClient client = await server.AcceptTcpClientAsync();
client.NoDelay = true;
Console.WriteLine("Connected!");
await Task.Run(async () => await HandleDevice(client), Token);
}
}
catch (SocketException e)
{
Console.WriteLine("Exception: {0}", e);
}
}
private async Task HandleDevice(TcpClient client)
{
string data = null;
Byte[] bytes = new Byte[8196];
int i;
try
{
using (stream = client.GetStream())
{
while ((i = await stream.ReadAsync(bytes, 0, bytes.Length, Token)) != 0)
{
Token.ThrowIfCancellationRequested();
string hex = BitConverter.ToString(bytes);
data += Encoding.UTF8.GetString(bytes, 0, i);
//Console.WriteLine(data);
if (data.IndexOf("[EOF]") >= 0)
{
processData(data);
data = "";
Array.Clear(bytes, 0, bytes.Length);
}
}
}
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Console.WriteLine("Exception: {0}", e.ToString());
}
finally
{
client.Close();
}
}
Question: How can I get properly formatted string JSON responses received from clients to processData() in a queued or the received processing order?
I'm learning C# and I've got a TCP server that works asynchronously
Sorry, but you're trying to learn three fairly complex topics simultaneously? I strongly recommend you learn C#, sockets, and asynchrony one at a time. C# and asynchrony are fairly common, so I'd recommend learning those first; socket development is much more rare, so I'd learn that only if necessary.
the buffer would contain half of one string and the beginning of another
This is a common problem when starting out with socket programming. My first response to everyone running into this is to drop the sockets completely and use something much higher-level like ASP.NET Core (self-hosted if necessary).
But if you need sockets for some reason, then the proper solution for this socket problem (the first of several usually encountered) is message framing. I discuss message framing on my blog, as part of a series on socket programming in .NET. I also have a video series I did earlier this year on asynchronous socket programming that would be well worth watching if you really need to learn socket programming (again, the vast majority of developers never need to learn that).
This is take II, i posted the other week and my question was put on hold, i adjusted my text but could not get a review, and the system closed the original post.
Logic
Server Side: Read only - Server Opens pipe then at regular interval checks if there is content (i.e. not at end of stream) and reads info. This check has to be poll based as only during the poll is there a valid context to pass on the data..
Client Side: Write-only - Open pipe, write to pipe, close (client.exe called many times, has short life span, code below is test code), e.g. some other script will "call client.exe with info"
Can this work flow be handled in pipes ? e.g. snippet of client code shown only the first client message is seen by the "server"
If pipes can do this after coding tips as most examples are for client - servers having similar life cycles.
Code snippets
for (int i = 0; i < 10; i++)
{
//Client - simulate exe starting and ending
var client = new NamedPipeClientStream(".", "PipesOfPiece", PipeDirection.Out, PipeOptions.WriteThrough);
client.Connect();
StreamWriter writer = new StreamWriter(client);
Console.WriteLine("Client about to send message");
writer.WriteLine("Called from client i = {0}", i);
writer.Close();
client.Close();
Thread.Sleep(5000);
}
// server snippet
var server = new NamedPipeServerStream("PipesOfPiece", PipeDirection.In);
server.WaitForConnection(); <= can this we optional with code below
StreamReader reader = new StreamReader(server);
while (true)
{
// simulate start of poll code
if (server.IsConnected)
{
if (!reader.EndOfStream)
{
var line = reader.ReadToEnd();
Console.WriteLine("Server: {0}", line);
}
} // End of poll code
Thread.Sleep(1000);
}
// server snippet
var server = new NamedPipeServerStream("PipesOfPiece", PipeDirection.In);
server.WaitForConnection(); <= can this we optional with code below
StreamReader reader = new StreamReader(server);
while (true)
{
// simulate start of poll code
if (server.IsConnected)
{
if (!reader.EndOfStream)
{
var line = reader.ReadToEnd();
Console.WriteLine("Server: {0}", line);
}
} // End of poll code
Thread.Sleep(1000);
}
So I am rusty on my pipes, I am hoping that a pipe can be opened, written to then read, and the waitforconnect() is there for the cases where you want this and is optional. I suppose it all triggers around who owns the pipe, i.e. if the server opens a pipe and is waiting for someone to write for it, why does it need to wait for a connect ? (I am hoping the server is the owner so when it ends, the pipe disappears)
Without a good, minimal, complete code example that reliably reproduces whatever specific problem you are having, it is impossible to provide specific advice as to how to fix that problem. However, I can at least try to answer some of your questions about how named pipes can be used, and provide a code example to illustrate some of the concepts.
First, some rules and observations:
A pipe instance can be used for only one connection. Note that pipes inherit Stream, and streams have a very specific paradigm: you open one, read to the end, and then you're done with the stream. Some streams, like FileStream, are seekable but even there you are only ever dealing with a single resource (i.e. the original file…you can't reconnect a FileStream to a different file), and network streams aren't even seekable.
A pipe must be connected before you perform I/O on it.
You may have multiple instances of pipes with the same name (if you initialize them correctly…by default, you may only have one pipe of any given name).
Clients trying to connect to a named pipe will wait until such a pipe exists. It does not need to exist at the time the client initiates the connection.
Only one client can connect to any given instance of a pipe. Any given instance of a server pipe can only ever handle a single client during its entire lifetime (see the very first point above).
So, what about your questions?
Can this work flow be handled in pipes ?
If I understand the work-flow correctly, yes. But you need to be careful to implement it correctly.
As I understand it, you want for your server to only attempt to read from clients periodically. At the same time, you want for a client to be able to write to a pipe at any time. This can be done, but it won't be straightforward.
Note per the above that you cannot open a single server pipe, and then let multiple clients periodically connect and disconnect from that pipe. Once the first client has connected, the pipe is no longer usable. It's a stream, and that first client's disconnection causes the stream to reach its end. It's done.
Note also that while a client can attempt to connect to a pipe that doesn't exist yet, it will wait until it can. So if you want your clients to not have to wait until the polling interval has expired, you'll need to maintain a server pipe available to connect to at all times.
But you've already said that you won't be able to handle data read from the server pipe at arbitrary points in time, but rather only during your polling interval.
Because pipes don't inherently support this specific scenario, IMHO the right way to implement it is to separate the behaviors into two different components. Maintain a simple pipe server that opens a pipe, waits for a client to connect, reads whatever the client has sent, closes the pipe, and then starts over.
Then have an intermediary class that can act as the go-between for the server I/O and whatever component you have that ultimately receives the data. This intermediary will retain a copy of the data after it's been received (the pipe code will deliver it to the intermediary as soon as it's received, regardless of the polling interval); the data will then later be retrieved by the polling component on its next polling interval (i.e. when the "context" as you put it is in fact available to which to deliver the data).
I am hoping that a pipe can be opened, written to then read, and the waitforconnect() is there for the cases where you want this and is optional
Unfortunately, your hope doesn't match the reality. Pipes can be bidirectional; i.e. "written to then read". But WaitForConnect() is not optional. The server must wait for a connection before attempting to read from the pipe, and for that pipe instance it will only ever be able to receive data from a single client.
I am hoping the server is the owner so when it ends, the pipe disappears
The server process is the one that actually creates the pipe. So yes, in that sense it is the owner. And yes, when the server process is terminated, any pipes it's created are destroyed.
Below, please find a simple code example that illustrates the use of multiple and concurrent servers and clients. You can adjust the numbers of each with the declared constants at the top of the example.
When running it, note that if more clients are active than servers, the additional clients will simply wait until a server pipe is available to connect to. Once one is, they will connect and proceed normally. If there are at least as many server pipe instances as there are clients trying to connect, all of the clients are serviced concurrently.
// NOTE: as a sample program, contrary to normal and correct
// programming practices error-handling has been omitted, and
// non-awaited async methods have been declared as void.
class Program
{
private const string _kserverName = "TestSO33093954NamedPipeClients";
private const int _kmaxServerCount = 3;
private const int _kmaxClientCount = 3;
static void Main(string[] args)
{
StartServers(_kmaxServerCount);
StartClients(_kmaxClientCount);
Console.WriteLine("Clients are being started. Press return to exit program.");
Console.ReadLine();
}
private static async void StartClients(int clientCount)
{
for (int i = 0; i < clientCount; i++)
{
RunClient(i);
await Task.Delay(300);
}
}
private static async void RunClient(int instance)
{
NamedPipeClientStream client = new NamedPipeClientStream(
".", _kserverName, PipeDirection.InOut, PipeOptions.Asynchronous);
client.Connect();
ReadClient(client);
using (StreamWriter writer = new StreamWriter(client))
{
writer.AutoFlush = true;
for (int i = 0; i < 5; i++)
{
string text =
string.Format("Instance #{0}, iteration #{1}", instance, i);
Console.WriteLine("Client send: " + text);
await writer.WriteLineAsync(text);
await Task.Delay(1000);
}
client.WaitForPipeDrain();
}
}
private static async void ReadClient(Stream stream)
{
using (TextReader reader = new StreamReader(stream))
{
string line;
while ((line = await reader.ReadLineAsync()) != null)
{
Console.WriteLine("Client recv: " + line);
}
}
}
private static void StartServers(int maxServerInstances)
{
for (int i = 0; i < maxServerInstances; i++)
{
RunServer(maxServerInstances);
}
}
private static async void RunServer(int maxServerInstances)
{
while (true)
{
using (NamedPipeServerStream server = new NamedPipeServerStream(
_kserverName, PipeDirection.InOut, maxServerInstances,
PipeTransmissionMode.Byte, PipeOptions.Asynchronous))
{
await server.WaitForConnectionAsync();
byte[] buffer = new byte[1024];
int bytesRead;
Decoder decoder = Encoding.UTF8.GetDecoder();
while ((bytesRead =
await server.ReadAsync(buffer, 0, buffer.Length)) > 0)
{
int cch = decoder.GetCharCount(buffer, 0, bytesRead);
char[] rgch = new char[cch];
decoder.GetChars(buffer, 0, bytesRead, rgch, 0);
Console.Write("Server recv: " + new string(rgch));
await server.WriteAsync(buffer, 0, bytesRead);
}
}
}
}
}
static class PipeExtensions
{
// As I am not running with .NET 4.6 yet, I need this little helper extension
// to wrap the APM-based asynchronous connection-waiting with the await-friendly
// Task-based syntax. Anyone using .NET 4.6 will have this in the framework already
public static Task WaitForConnectionAsync(this NamedPipeServerStream server)
{
return Task.Factory.FromAsync(
server.BeginWaitForConnection, server.EndWaitForConnection, null);
}
}
I have a small game server I'm making that will have dozens of connections sending player data constantly. While I've finally accomplished some basics and now have data sending/receiving, I now face a problem of flooding the server and the client with too much data. I've tried to throttle it back but even then I am hitting 90-100% cpu simply because of receiving and processing the data received running up the CPU.
The method below is a bare version of receiving data from the server. The server sends a List of data to be received by the player, then it goes through that list. I've thought perhaps instead just using a dictionary with a key based on type rather than for looping but I don't think that will significantly improve it, the problem is that it is processing data non-stop because player positions are constantly being updated, sent to the server, then send to other players.
The code below shows receive for the client, the server receive looks very similar. How might I begin to overcome this issue? Please be nice, I am still new to network programming.
private void Receive(System.Object client)
{
MemoryStream memStream = null;
TcpClient thisClient = (TcpClient)client;
List<System.Object> objects = new List<System.Object>();
while (thisClient.Connected && playerConnected == true)
{
try
{
do
{
//when receiving data, first comes length then comes the data
byte[] buffer = GetStreamByteBuffer(netStream, 4); //blocks while waiting for data
int msgLenth = BitConverter.ToInt32(buffer, 0);
if (msgLenth <= 0)
{
playerConnected = false;
thisClient.Close();
break;
}
if (msgLenth > 0)
{
buffer = GetStreamByteBuffer(netStream, msgLenth);
memStream = new MemoryStream(buffer);
}
} while (netStream.DataAvailable);
if (memStream != null)
{
BinaryFormatter formatter = new BinaryFormatter();
memStream.Position = 0;
objects = new List<System.Object>((List<System.Object>)formatter.Deserialize(memStream));
}
}
catch (Exception ex)
{
Console.WriteLine("Exception: " + ex.ToString());
if (thisClient.Connected == false)
{
playerConnected = false;
netStream.Close();
thisClient.Close();
break;
}
}
try
{
if (objects != null)
{
for (int i = 0; i < objects.Count; i++)
{
if(objects[i] != null)
{
if (objects[i].GetType() == typeof(GameObject))
{
GameObject p = (GameObject)objects[i];
GameObject item;
if (mapGameObjects.TryGetValue(p.objectID, out item))
{
mapGameObjects[p.objectID] = p;;
}
else
{
mapGameObjects.Add(p.objectID, p);
}
}
}
}
}
}
catch (Exception ex)
{
Console.WriteLine("Exception " + ex.ToString());
if (thisClient.Connected == false)
{
playerConnected = false;
netStream.Close();
break;
}
}
}
Console.WriteLine("Receive thread closed for client.");
}
public static byte[] GetStreamByteBuffer(NetworkStream stream, int n)
{
byte[] buffer = new byte[n];
int bytesRead = 0;
int chunk = 0;
while (bytesRead < n)
{
chunk = stream.Read(buffer, (int)bytesRead, buffer.Length - (int)bytesRead);
if (chunk == 0)
{
break;
}
bytesRead += chunk;
}
return buffer;
}
Based on the code shown, I can't say why the CPU utilization is high. The loop will wait for data, and the wait should not consume CPU. That said, it still polls the connection in checking the DataAvailable property, which is inefficient and can cause you to ignore received data (in the implementation shown...that's not an inherent problem with DataAvailable).
I'll go one further than the other answer and state that you should simply rewrite the code. Polling the socket is just no way to handle network I/O. This would be true in any scenario, but it is especially problematic if you are trying to write a game server, because you're going to use up a lot of your CPU bandwidth needlessly, taking it away from game logic.
The two biggest changes you should make here are:
Don't use the DataAvailable property. Ever. Instead, use one of the asynchronous APIs for dealing with network I/O. My favorite approach with the latest .NET is to wrap the Socket in a NetworkStream (or get the NetworkStream from a TcpClient as you do in your code) and then use the Stream.ReadAsync() along with async and await. But the older asynchronous APIs for Sockets work well also.
Separate your network I/O code from the game logic code. The Receive() method you show here has both the I/O and the actual processing of the data relative to the game state in the same method. This two pieces of functionality really belong in two separate classes. Keep both classes, and especially the interface between them, very simple and the code will be a lot easier to write and to maintain.
If you decide to ignore all of the above, you should at least be aware that your GetStreamByteBuffer() method has a bug in it: if you reach the end of the stream before reading as many bytes were requested, you still return a buffer as large as was requested, with no way for the caller to know the buffer is incomplete.
And finally, IMHO you should be more careful about how you shutdown and close the connection. Read about "graceful closure" for the TCP protocol. It's important that each end signal that they are done sending, and that each end receive the other end's signal, before either end actually closes the connection. This will allow the underlying networking protocol to release resources as efficiently and as quickly as possible. Note that TcpClient exposes the socket as the Client property, which you can use to call Shutdown().
Polling is rarely a good approach to communication, unless you're programming 16-bit microcontrollers (and even then, probably not the best solution).
What you need to do is to switch to a producer-consumer pattern, where your input port (a serial port, an input file, or a TCP socket) will act as a producer filling a FIFO buffer (a queue of bytes), and some other part of your program will be able to asynchronously consume the enqueued data.
In C#, there are several ways to do it: you can simply write a couple of methods using a ConcurrentQueue<byte>, or a BlockingCollection, or you can try a library like the TPL Dataflow Library which IMO doesn't add too much value over existing structures in .NET 4. Prior to .NET 4, you would simply use a Queue<byte>, a lock, and a AutoResetEvent to do the same job.
So the general idea is:
When your input port fires a "data received" event, enqueue all received data into the FIFO buffer and set a synchronization event to notify the consumer,
In your consumer thread, wait for the synchonization event. When the signal is received, check if there is enough data in the queue. If yes, process it, if not, continue waiting for the next signal.
For robustness, use an additional watchdog timer (or simply "time since last received data") to be able to fail on timeout.
You want to use the Task-based Asynchronous Pattern. Probably making liberal use of the async function modifier and the await keyword.
You'd be best replacing GetStreamByteBuffer with a direct call to ReadAsync.
For instance you could asynchronously read from a stream like this.
private static async Task<T> ReadAsync<T>(
Stream source,
CancellationToken token)
{
int requestLength;
{
var initialBuffer = new byte[sizeof(int)];
var readCount = await source.ReadAsync(
initialBuffer,
0,
sizeof(int),
token);
if (readCount != sizeof(int))
{
throw new InvalidOperationException(
"Not enough bytes in stream to read request length.");
}
requestLength = BitConvertor.ToInt32(initialBuffer, 0);
}
var requestBuffer = new byte[requestLength];
var bytesRead = await source.ReadAsync(
requestBuffer,
0,
requestLength,
token);
if (bytesRead != requestLength)
{
throw new InvalidDataException(
string.Format(
"Not enough bytes in stream to match request length." +
" Expected:{0}, Actual:{1}",
requestLength,
bytesRead));
}
var serializer = new BinaryFormatter();
using (var requestData = new MemoryStream(requestBuffer))
{
return (T)serializer.Deserialize(requestData);
}
}
Like your code this reads an int from the stream to get the length, then reads that number of bytes and uses the BinaryFormatter to deserialize the data to the specified generic type.
Using this generic function you can simplify your logic,
private Task Receive(
TcpClient thisClient,
CancellationToken token)
{
IList<object> objects;
while (thisClient.Connected && playerConnected == true)
{
try
{
objects = ReadAsync<List<object>>(netStream, token);
}
catch (Exception ex)
{
Console.WriteLine("Exception: " + ex.ToString());
if (thisClient.Connected == false)
{
playerConnected = false;
netStream.Close();
thisClient.Close();
break;
}
}
try
{
foreach (var p in objects.OfType<GameObject>())
{
if (p != null)
{
mapGameObjects[p.objectID] = p;
}
}
}
catch (Exception ex)
{
Console.WriteLine("Exception " + ex.ToString());
if (thisClient.Connected == false)
{
playerConnected = false;
netStream.Close();
break;
}
}
}
Console.WriteLine("Receive thread closed for client.");
}
You need to put a Thread.Sleep(10) in your while loop. This is also a very fragile way to receive tcp data because it assumes the other side has sent all data before you call this receive. If the other side has only sent half of the data this method fails. This can be countered by either sending fixed sized packages or sending the length of a package first.
Your player position update is similar to the framebuffer update in the VNC protocol where the client request a screen frame & server responds to it with the updated screen data. But there is one exception, VNC server doesn't blindly send the new screen it only sends the changes if there is one. So you need to change the logic from sending all the requested list of objects to only to the objects which are changed after the last sent. Also in addition to it, you should send entire object only once after that send only the changed properties, this will greatly reduce the size of data sent & processed both at clients & server.
I have an application that receives data from GPRS clients in the field on a TCP connection. From time to time the GPRS client devices loses connection and buffers the data, when the connection is restored all the buffered data is send to the TCP connection, cauing my application to throw a System.OutOfMemoryException.
I presume this is because the data received is bigger than my buffer size (which is set to int.MaxValue).
How do I prevent my application to run out of memory?
How do I make sure that I do not lose any of the incomming data?
Below is the code used to listen for, and handle incomming data
public void Listen(string ip, int port)
{
_logger.Debug("All.Tms.SensorDataServer : SensorDataListener : Listen");
try
{
var listener = new TcpListener(IPAddress.Parse(ip), port);
listener.Start();
while (true)
{
var client = listener.AcceptTcpClient();
client.SendBufferSize = int.MaxValue;
var thread = new Thread(() => ReadAndHandleIncommingData(client));
thread.IsBackground = true;
thread.Start();
}
}
catch (Exception ex)
{
_logger.Error("TMS.Sensor.SensorDataServer : SensorDataListener : Error : ", ex);
}
}
and
private void ReadAndHandleIncommingData(TcpClient connection)
{
try
{
var stream = connection.GetStream();
var data = new byte[connection.ReceiveBufferSize];
var bytesRead = stream.Read(data, 0, System.Convert.ToInt32(connection.ReceiveBufferSize));
var sensorDataMapper = new SensorDataMapperProvider().Get(data);
if (sensorDataMapper != null)
{
_sensorDataHandler.Handle(sensorDataMapper.Map(data));
}
}
catch (Exception ex)
{
_logger.Error("TMS.Sensor.SensorDataServer : SensorDataListener : ReadAndHandleIncommingData : Error : ", ex);
}
finally
{
try
{
connection.Close();
}
catch(Exception ex)
{
_logger.Error("All.Tms.SensorDataServer : SensorDataListener : ReadAndHandleIncommingData : Error : ", ex);
}
}
}
About buffers
OutOfMemoryExceptions are thrown when there is no sequential memory left.
In your case, that means connection.ReceiveBufferSize is too big to keep in memory in one piece. Not because the data received is bigger than your buffer size.
You can use a smaller, fixed buffer to get the received bytes, append it somewhere and use the same buffer to receive the rest of the data, until you have it all.
One thing to look out for is the collection you use to store the received data. You can't use List<byte> for example because it stores its elements in a single array under the hood which makes no difference than getting everything in one go - like you do now.
You can see MemoryTributary, a stream implementation meant to replace MemoryStream. You can copy your stream to this one and keep it as a stream. This page also contains a lot of information that can help you understand the causes of the OutOfMemoryExceptions.
In addition, I wrote a buffer manager to provide fixed sized buffers a while ago. You can find that in Code Review, here.
About threads
Creating a thread for every single connection is brutal. They cost you 1 MB each so you should either use ThreadPool or better, IOCP (via asynchronous methods of the Socket class).
You may want to look into these for common pitfalls and best practices about socket programming:
Stephen Cleary (the blog) | TCP/IP .NET Sockets FAQ
Stack Overflow | How to write a scalable Tcp/Ip based server
Code Project | C# SocketAsyncEventArgs High Performance Socket Code
Use a fixed receive buffer, and parse data incrementally
I need to connect a server (with ip and port) and create a read-loop that will get messages from the server as XML. sometimes there are no messages from the server.
I tried to create a connection (works fine) and read messages, I get the first message from the server and when I'm trying to read another one - it get stuck. I think that maybe there are no messages right now but I need that the loop will continue until there will be messages... it doesn't even go to "catch" or "finally", just do nothing..
public class Connection
{
public Connection()
{
Socket server = null;
try
{
string p = string.Empty;
using (var client = new TcpClient(myIPAddress, myPort))
using (var stream = client.GetStream())
using (var reader = new StreamReader(stream))
{
while (p != null)
{
try
{
p = reader.ReadLine();
}
catch (Exception e)
{
//
}
}
}
}
catch (Exception e)
{
//
}
finally {
server.Close();
}
}
}
The loop is continuing, waiting for data. The issue here seems to be simply that ReadLine() is a blocking call. You mention that there might not be a message yet; well, ReadLine() is going to block until one of two conditions is met:
it can successfully read some data, terminated by a newline (or EOF, i.e. a message without a newline followed by socket closure) - in which case it returns the line of data
no more data is received and the stream is closed, in which case it returns null
So basically, ReadLine() is going to wait until either a message comes in, or the socket is closed. That is simply the behaviour of ReadLine(). If that is problematic, you could work closer to the socket, and check NetworkStream.DataAvailable but: note that only tells you if some data is currently available; it doesn't mean "this is an entire message", nor can it be used to tell if more messages will arrive. The main use of DataAvailable is to decide between sync and async access. Plus if you work close to the socket you'll have to do all your own buffering and encoding/decoding.
It looks to me like ReadLine() is working successfully. The only thing I would do here is re-phrase it a bit:
string line;
while((line = reader.ReadLine()) != null) {
// line is meaningful; do something
}
One last thought: xml is not always trivially split into messages simply on a "per-line" basis. You might want to consider some other form of framing, but that may well mean working closer to the socket, rather than a StreamReader.
You have to wait till data arrives at the stream, you could try using follwing,
if(reader.EndOfStream)
continue;