I've written some code which is a mini simple imitation of messenger program.
In the program; when the user signs out, instance of my LogOutCommand class is prepared by client program, serialized, and sent to server. When the server receives the LogOutCommand, it deserializes and invokes Execute method of the class, which performs db operations, etc.
The problem is that, sometimes Server can deserialize very well, but sometimes fails.
As far as I understand, server sometimes starts deserialization before associated bytes are sent totally and accurately.
How can I make the server start deserialization in a way that it waits for all associated bytes are completed being sent?
Or do you think there is another problem?
Here is the code:
// Server listens to socket
private void ReadData(object obj)
{
Socket client = (Socket)obj;
while (true)
{
if (client.Available > 0)
{
byte[] buffer = new byte[client.Available];
client.Receive(buffer);
ServerCommandBase cmd = CommandReader.ReadSrvCommand(buffer);
cmd.Execute(context);
}
}
}
//CommandReader class
public class CommandReader
{
public static ServerCommandBase ReadSrvCommand(byte[] buffer)
{
return (ServerCommandBase)SerializationUtility.SerializationHelper.Deserialize(buffer);
}
public static ClientCommandBase ReadCliCommand(byte[] buffer)
{
return (ClientCommandBase)SerializationUtility.SerializationHelper.Deserialize(buffer);
}
}
// Serialization / Deserialization class
public class SerializationHelper
{
public static byte[] Serialize(object obj)
{
BinaryFormatter formatter = new BinaryFormatter();
MemoryStream stream = new MemoryStream();
try
{
formatter.Serialize(stream, obj);
}
catch (Exception)
{
MessageBox.Show("Serialize Edilemiyor");
}
stream.Position = 0;
return stream.ToArray();
}
public static object Deserialize(byte[] byteArr)
{
BinaryFormatter formatter = new BinaryFormatter();
MemoryStream ms = new MemoryStream(byteArr);
ms.Position = 0;
object retObj = null;
try
{
retObj = formatter.Deserialize(ms);
}
catch (Exception)
{
MessageBox.Show("Cannot Be Deserialized!");
}
return retObj;
}
}
The problem as I see it is that you are attempting to deserialize data as soon as it comes down the pipe. This may or may not be possible due to how the packets are sent across the network.
You should send a header with your message that instructs how long (in bytes) the data will be. You then take in the bytes until that number has been hit and deserialize then and only then.
Your header should be of a specific format and have a specific length so that it is easy to take it off at the beginning.
First of all you shouldn't use BinaryFormatter. It's much better to use e.g. XmlSerializer.
Secondly, please include exception details that you get.
Related
So I'm currently learning about networking in terms of the relationship between a server and it's clients. And as far as I understand, the server starts listening for incoming connections and once a connection has been established, packets will be sent back and forth.
Here is my take on it, I like to visualize it in a way of how a game would do it.
So you have your server running and then you connect with your client, with the connection comes a Login packet, the server would then parse this packet and continue the login process and the client would eventually login to the game world.
Packets are structured in a certain way to easily keep track of what type of packet it is by using an OpCode, the length of the packet and the actual payload.
So something like this.. [OpCode(2bytes), Length(4bytes), Payload(xBytes)]
And the server would then expect this format.
Issue
What if there was a malicious user who were to send a packet that doesn't match that packet format? I would have to dump that entire packet that the malicious user sent, but I don't know how big it is so I don't know how much to dump, leaving me at a spot prone for server crashes.
Question
How do I dump/ignore a packet that doesn't have a valid OpCode.. A Valid OpCode is an OpCode that exists in the OpCode enum
enum OpCode : short
{
Connect = 0
}
The way I'm doing it now is that I start the server by initializing the packets
Packets = new Dictionary<OpCode, IPacket>();
Packets.Add(OpCode.Connect, new ConnectPacket());
and calling a method named Run
public void Run()
{
_listener = new TcpListener(IPAddress.Any, Constants.PORT);
_listener.Start(10);
_isRunning = true;
Console.WriteLine("Server has started..");
while (_isRunning)
{
_clients.Add(new Connection(_listener.AcceptTcpClient(), Packets));
}
}
When a TcpClient has been accepted, it enters the constructor
private TcpClient _socket;
private Dictionary<OpCode, IPacket> _packets;
public NetReader Reader;
public Connection(TcpClient client, Dictionary<OpCode, IPacket> packets)
{
_buffer = new byte[bufferSize];
_socket = client;
_packets = packets;
Reader = new NetReader(_socket.GetStream());
Console.WriteLine($"Client connected: {_socket.Client.RemoteEndPoint}");
ProcessPackets();
}
And as you can see I call the ProcessPackets function which checks if the first two bytes evaluates to a valid OpCode, if so, process the rest of the data that follows, if not I guess I want to dump the data and read the next packet.
Same goes for if the malicious packet somehow gets the OpCode right, if that's the case then dump the packet because the rest of the packet will probably be corrupt.
public void ProcessPackets()
{
Task.Run(() =>
{
IPacket packet;
while (true)
{
OpCode = Reader.ReadInt16();
if (_packets.TryGetValue((OpCode)OpCode, out packet))
{
_packets[(OpCode)OpCode].Process(this);
}
else
{
//Dump the packet
}
}
});
}
And here is an example of what a Process function could look like, but it doesn't matter that much since I want to solve dumping the packet if the OpCode isn't valid first.
public void Process(Connection connection)
{
var length = connection.Reader.ReadInt32();
var payload = connection.Reader.ReadBytes(length);
Console.WriteLine($"Received and processed Connection Packet");
}
NetReader is just a class that derives from BinaryReader
class NetReader : BinaryReader
{
private NetworkStream _ns;
private byte[] _tempBuffer;
public NetReader(NetworkStream ns) : base(ns)
{
_ns = ns;
_tempBuffer = new byte[8];
}
public override byte ReadByte()
{
byte val = (byte)_ns.ReadByte();
return val;
}
public override short ReadInt16()
{
int val = _ns.Read(_tempBuffer, 0, 2);
return BitConverter.ToInt16(_tempBuffer, 0)
}
public override int ReadInt32()
{
int val = _ns.Read(_tempBuffer, 0, 4);
return BitConverter.ToInt32(_tempBuffer, 0)
}
public override byte[] ReadBytes(int count)
{
byte[] bytes = new byte[count];
_ns.Read(bytes, 0, count);
return bytes;
}
}
I'm currently trying to setup a server that accepts multiple clients and can receive and respond to messages.
The client and server use a common library at the core, which contains a Request class that gets serialized and sent from client to server and similar in reverse.
The server listens asyncronously to clients on each of their sockets and attempts to take the data received and deserialize the data into the Request class.
Data is sent via a NetworkStream using a BinaryFormatter to send directly on the socket. The received data is then parsed using a Network Stream on the other end.
I've tried using a MemoryStream to store the data to a buffer and then deserialize it as shown below, however this hasn't worked. Directly deserializing the NetworkStream didn't work either.
Searching around I haven't found much information that has worked for my use case.
This is the active code after the sockets are successfully connected:
On the request class, sending from the client:
public void SendData(Socket socket)
{
IFormatter formatter = new BinaryFormatter();
Stream stream = new NetworkStream(socket, false);
formatter.Serialize(stream, this);
stream.Close();
}
Server Code receiving this data:
public void Receive(Socket socket)
{
try
{
ReceiveState state = new ReceiveState(socket);
state.Stream.BeginRead(state.Buffer, 0, ReceiveState.BUFFER_SIZE, new AsyncCallback(DataReceived), state);
}
catch (Exception e)
{
Logger.LogError(e.ToString());
}
}
private void DataReceived(IAsyncResult ar)
{
ReceiveState state = (ReceiveState)ar.AsyncState;
int bytesRead = state.Stream.EndRead(ar);
//Resolve Message
try
{
IFormatter formatter = new BinaryFormatter();
MemoryStream memoryStream = new MemoryStream(state.Buffer, 0, bytesRead);
Request request = (Request)formatter.Deserialize(memoryStream);
Logger.Log("Request Received Successfully");
ResolveRequest(request, state.Socket);
}
catch (Exception e)
{
Logger.LogError(e.ToString());
}
//Resume listening
Receive(state.Socket);
}
public class ReceiveState
{
public byte[] Buffer;
public const int BUFFER_SIZE = 1024;
public Socket Socket;
public NetworkStream Stream;
public ReceiveState(Socket socket)
{
Buffer = new byte[BUFFER_SIZE];
Socket = socket;
Stream = new NetworkStream(Socket, false);
}
}
Currently, when BeginRead() is called on the NetworkStream I get a single byte of data, then the remaining data when the next BeginRead() is called.
e.g. The Serialized data should be: 00-01-00-00-00-FF-FF-FF-FF-01-...
I receive: 00 followed by 01-00-00-00-FF-FF-FF-FF-01-... which fails to deserialize.
I take it that the issue is that the DataReceived() method is called as soon as any data appears, which is the single byte taken, then the remainder arrives before listening is resumed.
Is there a way to make sure each message is received in full before deserializing? I'd like to be able to deserialize the object as soon as the last byte is received.
TCP is a stream protocol, not a packet protocol. That means you are only guaranteed to get the same bytes in the same order (or a network failure); you are not guaranteed to get them in the same chunk configurations. So: you need to implement your own framing protocol. A frame is how you partition messages. For binary messages, a simple framing protocol might be "length = 4 bytes little-endian int32, followed by {length} bytes of payload", in which case the correct decode is to buffer until you have 4 bytes, decode the length, buffer {length} bytes, then decode the payload. YOU NEED TO WRITE the code that buffers the correct amounts, and at every point you need to deal with over-reading, back-buffers, etc. It is a complex topic. Frankly, a lot of the nuances are solved by using the "pipelines" API (I have a multi-part discussion on that API here).
However, additional guidance:
never ever use BinaryFormatter, especially for scenarios like this; it will hurt you, and it is not a good fit for most use-cases (it also isn't a particularly good serializer); my recommendation would be something like protobuf (perhaps protobuf-net), but I'm arguably biased
network code is subtle and complex, and RPC is largely a "solved" problem; consider trying tools like gRPC instead of rolling it yourself; this can be very easy
The Problem
The other day I had an "amazing" idea for my C# Application. Instead of using one port for each data type (text, image, file), I would like to use a single port for all three data types. I am very close to implementing this.
So far I've got a TcpListener listening on port 23722. When a TcpClient connects I start writing incoming data to a file using StreamWriter.Write(datareceived); and data comes in a specific pattern (each line represents an image, a text message or a file):
dnett{This is a sample text message.) dneti{NLtqmuvtGBdDY546 ... NLtqmuvtGBdDY546} dnetf{Example.exe,NLtqmuvtGBdDY546 ... NLtqmuvtGBdDY546}
As you can see from above images and files are converted to Base64 before sending. To convert them back I would use byte[] tmp = Convert.FromBase64String(string), but the problem is that I can't read the file line by line while the data is still incoming (being written with StreamWriter). The other problem is, that I don't know which data has already been processed (example: which files had already been written to the filesystem).
What I need is:
a solution for reading/writing file at the same time
knowing which data has already been processed
OR
another way of doing this (different approach)
Thanks
By the way I am only 15 yrs old and English is not my first language, so I am sorry for potentially stupid question and for mistakes in the above question asked.
Usually, you would first send the size of the "message" and then the actual "message" (from the server). You could extend this to first send the size of your message type, then send your message type, then send the size of the actual message and then send that message.
To read back, you'd do something like this
using System;
using System.IO;
using System.Net.Sockets;
using System.Runtime.Serialization.Formatters.Binary;
using System.Threading;
using Newtonsoft.Json;
using Unity3DRemoteRendererClient.Helpers;
namespace Unity3DRemoteRendererClient.Communications.TCPCommunication {
public sealed partial class TCPClientManager {
public sealed class Receiver {
private NetworkStream _stream;
private Thread _thread;
//Subscribe to this event if you want an event ever time data arrives.
public event EventHandler < UnityToClientMessage > DataReceivedEvent;
private static ManualResetEvent ShutDownEvent = new ManualResetEvent(false);
public void Start(NetworkStream stream) {
_stream = stream;
_thread = new Thread(Run);
_thread.Start();
}
private void Run() {
try {
// ShutdownEvent is a ManualResetEvent signaled by
// Client when its time to close the socket.
while (!ShutDownEvent.WaitOne(0)) {
try {
if (!_stream.DataAvailable) continue;
//Read the first 4 bytes which represent the size of the message, and convert from byte array to int32
var sizeinfo = new byte[4];
_stream.Read(sizeinfo, 0, 4);
var messageSize = BitConverter.ToInt32(sizeinfo, 0);
//create a new buffer for the data to be read
var buffer = new byte[messageSize];
var read = 0;
//Continue reading from the stream until we have read all bytes #messageSize
while (read != messageSize) {
read += _stream.Read(buffer, read, buffer.Length - read);
}
//I use flatbuffers, so you should deserialize yourself.
var message = new UnityToClientMessage().FlatDeserialize(buffer);
//raise data received event
OnDataReceived(message);
} catch (IOException ex) {
// Handle the exception...
throw;
}
}
} catch (Exception ex) {
// Handle the exception...
throw;
} finally {
_stream.Close();
}
}
private void OnDataReceived(UnityToClientMessage e) {
EventHandler < UnityToClientMessage > handler = DataReceivedEvent;
if (handler != null) {
handler(this, e);
}
}
public void ShutDown() {
ShutDownEvent.Set();
}
}
}
}
Just modify it to first read the size of the message type message and then to read the actual message. You can do a switch statement once you have the message describing the type of message coming, and process it accordingly.
i'm beginner too. But i think that you can use an ArrayList to converse all lines until the all lines are readed and sended. So when you are ready and the file is all readed yoy can save it . If you are asking how to know recognize which kind of files is sended first send a file you can send a caracter or a string that allow the other program that wait file to recognize whic file is going to be revived. Excuse me for my English. I hope that I could be helpful for you.
When i send a request to my server or a reply to my client, the message i send is always divided into multiple parts.
So i need to call Receive multiple times to get to the last part/frame.
Why does this happen.. is there a better way of sending and receiving an xml encoded string?
This is the code of my client:
private void SendRequestAsyncTaskStart<T>(object contextObj, T request)
{
ZmqContext context = (ZmqContext)contextObj;
ZmqSocket requestSocket = CreateServerSocket(context);
SerializeAndSendRequest(request, requestSocket);
}
private ZmqSocket CreateServerSocket(ZmqContext context)
{
var client = context.CreateSocket(SocketType.REQ);
client.Connect(_requestReplyEndpoint);
client.Linger = TimeSpan.Zero;
client.ReceiveReady += PollInReplyHandler;
return client;
}
public static string Serialize(this object obj)
{
string result;
using (var memoryStream = new MemoryStream())
{
using (var reader = new StreamReader(memoryStream))
{
var serializer = new DataContractSerializer(obj.GetType());
serializer.WriteObject(memoryStream, obj);
memoryStream.Position = 0;
result = reader.ReadToEnd();
}
}
return result;
}
This is the code of my server:
private void ListenForRequestsThreadStart(object contextObj)
{
ZmqContext context = (ZmqContext)contextObj;
using (
ZmqSocket frontend = context.CreateSocket(SocketType.REP),
backend = context.CreateSocket(SocketType.DEALER))
{
string bindAddress = string.Format("tcp://*:{0}", _listenForRequetsPort);
frontend.Bind(bindAddress);
backend.Bind("inproc://backend");
frontend.ReceiveReady += HandleRequestReceived;
// polling
}
}
private void HandleRequestReceived(object sender, SocketEventArgs e)
{
string message;
bool hasNext;
do
{
message = socket.Receive(Encoding.ASCII);
hasNext = socket.ReceiveMore;
} while (hasNext);
// after calling Receive 3 times i get my actual message
}
Since you're sending via a socket you're at the mercy of the network. First, the network will have broken your message down in multiple packates each of which is received separately by your listener. Every now and then, the underlying socket on the listening machine will say to itself 'Got some incoming, but there's more to come. Wait a bit'. After a while it'll say, 'Oh well, give what I've got' and keep waiting'.
That's what's happening. In WCF, the WCF implementation gets its data via sockets which do exactly the same thing. But WCF waits till the whole message arrives before giving it to your waiting code. That's one of the advantages of using a Framework like WCF. It protects you from the metal.
Any message sent over TCP may be divided into several packets depending on its size. That's why you should never assume to get a message in one go, but read until you're sure you've received everything.
I'm having a hard time getting something done.
I am building an mailclient using asp.net MVC 4.
I am at the point that i have to download an image related to a message ( NOT an attachment ) to the client browser.
Now i have this setup :
Client browser -> Controller/backend -> Mail server
to clarify : i have a client request containing the content ID of the image, the right mailbox, message etc. With that information i can download the image from the Mail server and upload it to the client.
Now comes the hard part: I want to do this asynchronous. I want to be able to download a chunk of 512 KB from the mailserver, decode that part, and send it to the client.. fetch - decode - send.. So long that the browser got all the data of the image.
I just dont want to first download ALL the data first to the server and then create a new memorystream with all that data and return that as a fileresult. I am just too afraid of getting too large files in my memory and block other processes etc.
I am planning to use this method too of uploading real attachments ( which could be 100's of MBs ). So i am gonna need that method later on.
Now i just have no idea how to achieve this, because i have a connection to the mail server, and i have a connection to the client. and i have to pass data to a new stream or something to get this done..
Can someone please help me?
Edit: to clarify: no i cannot refer to the file on the mail server. i HAVE to download the file to the server through sockets.
Edit2: could http chuncked be the solution? If yes, could you give me a small example?
You just need to copy data from one stream (the tcp connection to the mail server) to another (the http connection to the browser), right? If you want to scale, you'll need to use non-blocking IO as described in this article. So you'll want to call the code in that article from an IHttpAsyncHandler implementation. You'll end up with something like this:
class MyHandler : IHttpAsyncHandler
{
public IAsyncResult BeginProcessRequest(HttpContext context, AsyncCallback cb, object extraData)
{
Stream src = null; // remote data source
Stream dst = context.Response.OutputStream;
// set content type, etc
var res = new MyResult();
AsynchCopy(src, dst, () =>
{
((ManualResetEvent)res.AsyncWaitHandle).Set();
cb(res);
src.Close();
dst.Flush();
});
return res;
}
public void EndProcessRequest(IAsyncResult result)
{
}
public bool IsReusable
{
get { return true; }
}
public void ProcessRequest(HttpContext context)
{
throw new NotImplementedException();
}
class MyResult : IAsyncResult
{
public MyResult()
{
AsyncWaitHandle = new ManualResetEvent(false);
}
public object AsyncState
{
get { return null; }
}
public WaitHandle AsyncWaitHandle
{
get;
private set;
}
public bool CompletedSynchronously
{
get { return false; }
}
public bool IsCompleted
{
get { return AsyncWaitHandle.WaitOne(0); }
}
}
public static void AsynchCopy(Stream src, Stream dst, Action done)
{
byte[] buffer = new byte[2560];
AsyncCallback readCallback = null, writeCallback = null;
readCallback = (readResult) =>
{
int read = src.EndRead(readResult);
if (read > 0)
{
dst.BeginWrite(buffer, 0, read, writeCallback, null);
}
else
{
done();
}
};
writeCallback = (writeResult) =>
{
dst.EndWrite(writeResult);
src.BeginRead(buffer, 0, buffer.Length, readCallback, null);
};
src.BeginRead(buffer, 0, buffer.Length, readCallback, null);
}
}
The above code is untested and doesn't contain error handling, but it should get you started.