Me and my partner working on a project where my partner is developing in Pascal and using Indy10 socket for networking and I'm developing under C# in an UWP project. Therefore for networking I have to use streamsocket namespace.
The problem is, that my partner uses ReadStream method with -1 and false parameters, that mean that the length of the receiving stream is indetermined. Therefore, I have to send a length prefix calculated (in this case I use BitConverter.GetBytes method) from the data in my stream (for example using DataWriter class) continued with the data itself.
But Indy10 socket don't receives nothing. It looks like it stucks on waiting for the packet or I don't know.
Does anybody hit that problem? Please help, we've been looking after the solution for about two weeks. I had read all the similar questions here in StackOverflow, a blog about the use of framing, delimitering. Searched for solution in Indy Socket source code itself.
Please help!
UPDATE 1
C# code. In this case I am sending from a WPF application. The logic, how streams are used in UWP are almost the same, just using StreamSocket and DataWriter.
TcpClient client = new TcpClient(hostTextBox.Text, int.Parse(portTextBox.Text));
Byte[] byteMessage = Encoding.Unicode.GetBytes(messageTextBox.Text);
Byte[] lenghtPrefix = BitConverter.GetBytes(byteMessage.Length);
Byte[] concatedData = new byte[lenghtPrefix.Length + byteMessage.Length];
lenghtPrefix.CopyTo(concatedData, 0);
byteMessage.CopyTo(concatedData, lenghtPrefix.Length);
NetworkStream stream = client.GetStream();
StreamWriter writer = new StreamWriter(stream);
writer.Write(concatedData);
writer.Flush();
stream.Close();
client.Close();
But we have tried this solution too:
Byte[] byteMessage = Encoding.Unicode.GetBytes(messageTextBox.Text);
Byte[] lenghtPrefix = BitConverter.GetBytes(byteMessage.Length);
Socket socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
socket.Connect(hostTextBox.Text, int.Parse(portTextBox.Text));
socket.Send(lenghtPrefix, SocketFlags.None);
socket.SendBufferSize = byteMessage.Length;
socket.Send(byteMessage, SocketFlags.None);
socket.Close();
Pascal code on the server side, waiting for stream. It is crucial to receive in this way, because the software's other modules (also written in Pascal using Indy10) communicating with the server in this way.
function Receive(const AContext: TIdContext): string;
var
Stream: TMemoryStream;
begin
try
Result := '';
Stream := TMemoryStream.Create;
if AContext.Connection.Connected then
AContext.Connection.IOHandler.ReadStream(Stream, -1, False);
finally
Result := UnPackStream(Stream);
Stream.Free;
end;
end;
Finally, here is the solution. We worked on that all day. We used Synapse socket as a reference. In the end here is the code under UWP, and we tested it in other desktop projects (WPF, Winforms) too (with TcpClient and NetworkStream).
BitConverter and prefixing the hole stream with a number didn't worked, because Indy waits for the length in binary format, therefore the byte array have to contain values representing the binary length. I hope it will help others having similar problem!
Windows.Networking.Sockets.StreamSocket socket = new Windows.Networking.Sockets.StreamSocket();
Windows.Networking.HostName serverHost = new Windows.Networking.HostName(hostTextBox.Text);
string serverPort = portTextBox.Text;
await socket.ConnectAsync(serverHost, serverPort);
//Write data to the server.
DataWriter writer = new DataWriter(socket.OutputStream);
byte[] data = Encoding.Unicode.GetBytes(messageTextBox.Text);
byte[] size = new byte[4];
ushort x = (ushort)(data.Length >> 16);
ushort y = (ushort)(data.Length);
size[0] = (byte)(x / 256);
size[1] = (byte)(x % 256);
size[2] = (byte)(y / 256);
size[3] = (byte)(y % 256);
writer.WriteBytes(size);
writer.WriteBytes(data);
await writer.StoreAsync();
await writer.FlushAsync();
writer.DetachStream();
//Read response.
var bytes = new byte[4];
DataReader reader = new DataReader(socket.InputStream);
await reader.LoadAsync(4);
reader.ReadBytes(bytes);
int length = bytes[0] * 256 * 256 * 256 + bytes[1] * 256 * 256 + bytes[2] * 256 + bytes[3];
byte[] dat = new byte[length];
var count = await reader.LoadAsync((uint)length);
reader.ReadBytes(dat);
responseTextBlock.Text = Encoding.Unicode.GetString(dat);
reader.DetachStream();
socket.Dispose();
When calling TIdIOHandler.ReadStream() in Indy, if AByteCount is -1 and AReadUntilDisconnect is False, ReadStream() will read the first 4 bytes (Int32) or 8 bytes (Int64) (depending on the value of the TIdIOHandler.LargeStream property) and interpret them as an integer in network byte order (big-endian), then read the number of bytes specified by that integer (if greater than 0).
The main problem with your first C# example is that you are using a StreamWriter, which is designed for sending text, not binary data. If you try to write a Byte[] array with a StreamWriter, it will not be sent as-is, it will be formatted to a textual representation that is not compatible with Indy. You don't need the StreamWriter, you can use the NetworkStream as-is, it has a Write() method for Byte[] data. Or use a BinaryWriter.
Another problem with both C# examples is that you are using BitConverter without taking endian into account. Its GetBytes(Int32) method creates an array of bytes in the same endian as the calling machine's architecture. Windows is primarily a little-endian environment, but ReadStream() is expecting the integer bytes in big-endian.
Try something more like this:
TcpClient client = new TcpClient(hostTextBox.Text, int.Parse(portTextBox.Text));
Byte[] byteMessage = Encoding.Unicode.GetBytes(messageTextBox.Text);
Byte[] lengthPrefix = BitConverter.GetBytes(byteMessage.Length);
if (BitConverter.IsLittleEndian)
Array.Reverse(lengthPrefix);
NetworkStream stream = client.GetStream();
stream.Write(lengthPrefix, 0, lengthPrefix.Length);
stream.Write(byteMessage, 0, byteMessage.Length);
stream.Close();
client.Close();
Or this:
TcpClient client = new TcpClient(hostTextBox.Text, int.Parse(portTextBox.Text));
Byte[] byteMessage = Encoding.Unicode.GetBytes(messageTextBox.Text);
Byte[] lengthPrefix = BitConverter.GetBytes(byteMessage.Length);
if (BitConverter.IsLittleEndian)
Array.Reverse(lengthPrefix);
NetworkStream stream = client.GetStream();
BinaryWriter writer = new BinaryWriter(stream);
// BinaryWriter.Write(Int32) writes in little-endian only,
// so you still need to use Write(Byte[]) for the length value...
writer.Write(lengthPrefix);
writer.Write(byteMessage)
writer.Flush();
stream.Close();
client.Close();
Related
I am getting data from client and saving it to the local drive on local host .I have checked it for a file of 221MB but a test for file of 1Gb gives the following exception:
An unhandled exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll
Following is the code at server side where exception stems out.
UPDATED
Server:
public void Thread()
{
TcpListener tcpListener = new TcpListener(ipaddr, port);
tcpListener.Start();
MessageBox.Show("Listening on port" + port);
TcpClient client=new TcpClient();
int bufferSize = 1024;
NetworkStream netStream;
int bytesRead = 0;
int allBytesRead = 0;
// Start listening
tcpListener.Start();
// Accept client
client = tcpListener.AcceptTcpClient();
netStream = client.GetStream();
// Read length of incoming data to reserver buffer for it
byte[] length = new byte[4];
bytesRead = netStream.Read(length, 0, 4);
int dataLength = BitConverter.ToInt32(length,0);
// Read the data
int bytesLeft = dataLength;
byte[] data = new byte[dataLength];
while (bytesLeft > 0)
{
int nextPacketSize = (bytesLeft > bufferSize) ? bufferSize : bytesLeft;
bytesRead = netStream.Read(data, allBytesRead, nextPacketSize);
allBytesRead += bytesRead;
bytesLeft -= bytesRead;
}
// Save to desktop
File.WriteAllBytes(#"D:\LALA\Miscellaneous\" + shortFileName, data);
// Clean up
netStream.Close();
client.Close();
}
I am getting the file size first from client side followed by data.
1).Should i increase the buffer size or any other technique ?
2). File.WriteAllBytes() and File.ReadAllBytes() seems blocking and freezes the PC.Is there any async method for it to help provide the progress of file recieved at server side.
You don't need to read the whole thing to memory before writing it to disc. Just copy straight from the network stream to a FileStream:
byte[] length = new byte[4];
// TODO: Validate that bytesRead is 4 after this... it's unlikely but *possible*
// that you might not read the whole length in one go.
bytesRead = netStream.Read(length, 0, 4);
int bytesLeft = BitConverter.ToInt32(length,0);
using (var output = File.Create(#"D:\Javed\Miscellaneous\" + shortFileName))
{
netStream.CopyTo(output, bytesLeft);
}
Note that instead of calling netStream.Close() explicitly, you should use a using statement:
using (Stream netStream = ...)
{
// Read from it
}
That way the stream will be closed even if an exception is thrown.
The CLR has a per-object limit a bit short of 2GB. However that's the theory, in practice how much memory you can allocate depends on how much memory the framework allows you to allocate. I wouldn't expect it to allow you to allocate 1 GB data table. You should allocate smaller table, and write the data in chunks into disk file.
The "out of memory" exception happens because you are trying to place the entire file into memory before dumping it on disk. This is suboptimal, because you don't need the entire file in memory in order to write into the file: you can read it block-by-block in reasonably-sized increments, and write it out as you go.
Starting with .NET 4.0 you can use Stream.CopyTo method to accomplish this in a few lines of code:
// Read and ignore the initial four bytes of length from the stream
byte[] ignore = new byte[4];
int bytesRead = 0;
do {
// This should complete in a single call, but the API requires you
// to do it in a loop.
bytesRead += netStream.Read(ignore, bytesRead, 4-bytesRead);
} while (bytesRead != 4);
// Copy the rest of the stream to a file
using (var fs = new FileStream(#"D:\Javed\Miscellaneous\" + shortFileName, FileMode.Create)) {
netStream.CopyTo(fs);
}
netStream.Close();
Starting with .NET 4.5 you can use CopyToAsync, too, which would give you a way to do reading and writing asynchronously.
Note the code that drops the initial four bytes from the stream. This is done to avoid writing the length of the stream along with the "payload" bytes. If you have control over the network protocol, you could change the sending side to stop prefixing the stream with its length, and remove the code that reads and ignores it on the receiving side.
I'm sending a message over a socket.
On the client side i'm assembling the message using StringBuilder
StringBuilder sb = new StringBuilder(message);
sb.Insert(0, (char)11);
sb.Append((char)28);
sb.Append((char)13);
Sending it from client to server
Byte[] data = new Byte[1024];
data = Encoding.ASCII.GetBytes(message.ToString());
NetworkStream stream = client.GetStream();
stream.Write(data, 0, data.Length);
Server Side
StringBuilder message = new StringBuilder(Encoding.ASCII.GetString(bytesReceived, 0, bytesReceived.Length));
I then want to check to see if my message is contained within the correct container but for some reason the last 2 characters equal 0 in the check instead of the correct 28 and 13.
if (((int)messsage[message.Length - 2] == 28) && ((int)message[message.Length - 1] == 13))
Thanks in advance for any help
Added Data that was asked for
byte[] bytes = new byte[1024];
NetworkStream stream = tcpClient.GetStream();
stream.Read(bytes, 0, bytes.Length);
Stream.Read will read up to bytes.Length bytes, it's return value will tell you how many bytes it actually read.
If that is not enough, then you will need to call Stream.Read() again.
Also, bytes.Length will always return the length of the array, not the number of bytes read.
Looking at your sending code, you probably want to read as much as you can from the stream, append what was read to the string builder, then check to see if the last 2 characters are 28 and 13, and if they are then you have your complete message.
i have a simplistic file server\client application ive written in c#. but i commonly run into the problem that my stream writes two different reads into a single buffer. i have a synchronized stream, still isnt helping. any suggestions? thanks!
System.Threading.Thread.Sleep(25);
receive_fspos = new byte[30];
int bytesread = stream_1.Read(receive_fspos, 0, receive_fspos.Length);//this is where it gets combined
if (bytesread == 0)
{
finished = true;
System.Threading.Thread.Sleep(25);
}
string string_1 = utf.GetString(receive_fspos).TrimEnd(new char[] { (char)0 });
int fsposition = (int)Convert.ToInt64(string_1);
bytestosend = fsposition;
filestream.Position = fsposition;
byte[] buffer_1 = new byte[bufsize];
int bytesreadfromfs = filestream.Read(buffer_1, 0, buffer_1.Length);
stream_1.Write(buffer_1, 0, buffer_1.Length);
Console.Write("\rSent " + fsposition + " / " + length + " bytes");
finished = true;
I would not recommend writing your own stream method if you do not fully understand it.
The problem that you are having is because the incoming data is a stream of bytes that does not give you a way of knowing how many bytes in length that the message is.
In the code below you are stating that you would like to read "receive_fspos.Length" bytes of the stream. Since "receive_fspos.Length" is 30, the amount of bytes that will be read will be anywhere from 0 to 30.
If there is only 15 bytes that have been received by the connection. It will give you 15 bytes. If the message was 20 bytes long. Then the message is now split up into different segments.
If the first message was 4 bytes and the second message is 12 bytes. Now you have 2 messages and a set of 16 blank bytes at the end. Even worse those 16 "blank" bytes could be the beginning of a third message coming in to the stream.
If the message is 50 bytes long. Then you will only receive half of the message. Now you would need to add the bytes that were read to a seperate buffer. Read from the stream again. Then repeat this until you have determined that you have read the exact amount of bytes that are needed to complete the entire message. Then concat all of the read bytes back to a single byte[].
receive_fspos = new byte[30];
int bytesread = stream_1.Read(receive_fspos, 0, receive_fspos.Length);//this is where it gets combined
Instead of rolling your own loop please use the BCL methods. It sounded like you are using strings so this would be the preferred method.. I would suggest the following.
using(NetworkStream networkStream = tcpClient.GetStream())
using(StreamReader streamReader = new StreamReader(networkStream))
using(StreamWriter streamWriter = new StreamWriter(networkStream))
{
networkStream.ReadTimeout = timeout; //Set a timeout to stop the stream from reading indefinately
//To receive a string
string incomingString = stream.ReadLine();
//To send a string
stream.WriteLine(messageToSend);
stream.Flush();
}
Your answer clarified that you are trying to send a file. For this I would recommend sending an array of bytes[]. Using this method you can send anything that can be serialized. This includes a file. Please note that the size of the file is limited since it must be kept in memory. To write a larger file you would want to save the data in chunks as it is being streamed in.
//Please note that if the file size is large enough. It may be preferred to use a stream instead of holding the entire file in memory.
byte[] fileAsBytes = File.ReadAllBytes(fileName);
using(NetworkStream networkStream = tcpClient.GetStream())
using(BinaryReader binaryReader = new BinaryReader(networkStream))
using(BinaryWriter binaryWriter = new BinaryWriter(networkStream))
{
networkStream.ReadTimeout = timeout; //Set a timeout to stop the stream from reading indefinately
//To receive a byte array
int incomingBytesLength = BinaryReader.ReadInt32(); //The header is 4 bytes that lets us know how large the incoming byte[] is.
byte[] incomingBytes = BinaryReader.ReadBytes(incomingBytesLength);
//To send a byte array
BinaryWriter.Write(fileAsBytes.Length); //Send a header of 4 bytes that lets the listener know how large the incoming byte[] is.
BinaryWriter.Write(fileAsBytes);
}
got it working, code > 30000 chars :\
it is a little messy but hey, it's functional.
server : https://www.dropbox.com/s/2wyccxpjbja10z3/Program.cs?m
client : https://www.dropbox.com/s/yp78nx4ubacsz6f/Program.cs?m
I have a client server situation, where the client sends the data (a movie for example) to the server, the server saves that data to the HDD.
It sends the data by a fixed array of bytes. After the bytes are sent, the server asks if there is more, if yes, send more and so on. Every thing is going well, all the data gets across.
But when I try to play the movie, it cant be played and if I look to the file length of each movie (client and server) the server movie is bigger then the client movie.also when I look at the command screen at the end of the sending/receiving data there is more then a 100% of the bytes that are across.
The only thing I can think of that can be wrong is the fact that my server reads in the stream till the fixed buffer array is full and therefor has at the end more bytes then the client. However if that is the problem how can I solve this?
I've just added the 2methods of sending, because the tcp connection works, any help is welcome.
Client
public void SendData(NetworkStream nws, StreamReader sr, StreamWriter sw)
{
using (FileStream reader = new FileStream(this.path, FileMode.Open, FileAccess.Read))
{
byte[] buffer = new byte[1024];
int currentBlockSize = 0;
while ((currentBlockSize = reader.Read(buffer, 0, buffer.Length)) > 0)
{
sw.WriteLine(true.ToString());
sw.Flush();
string wait = sr.ReadLine();
nws.Write(buffer, 0, buffer.Length);
nws.Flush();
label1.Text = sr.ReadLine();
}
sw.WriteLine(false.ToString());
sw.Flush();
}
}
Server
private void GetMovieData(NetworkStream nws, StreamReader sr, StreamWriter sw, Film filmInfo)
{
Console.WriteLine("Adding Movie: {0}", filmInfo.Titel);
double persentage = 0;
string thePath = this.Path + #"\films\" + filmInfo.Titel + #"\";
Directory.CreateDirectory(thePath);
thePath += filmInfo.Titel + filmInfo.Extentie;
try
{
byte[] buffer = new byte[1024]; //1Kb buffer
long fileLength = filmInfo.TotalBytes;
long totalBytes = 0;
using (FileStream writer = new FileStream(thePath, FileMode.CreateNew, FileAccess.Write))
{
int currentBlockSize = 0;
bool more;
sw.WriteLine("DATA");
sw.Flush();
more = Convert.ToBoolean(sr.ReadLine());
while (more)
{
sw.WriteLine("SEND");
sw.Flush();
currentBlockSize = nws.Read(buffer, 0, buffer.Length);
totalBytes += currentBlockSize;
writer.Write(buffer, 0, currentBlockSize);
persentage = (double)totalBytes * 100.0 / fileLength;
Console.WriteLine(persentage.ToString());
sw.WriteLine("MORE");
sw.Flush();
string test = sr.ReadLine();
more = Convert.ToBoolean(test);
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
There is a reason why Read() returns the number of bytes read: it's possible it will return less than the size of the buffer. Because of this, you should do something like nws.Write(buffer, 0, currentBlockSize); in SendData(). But this will break your protocol, because the blocks won't have the size anymore.
But I find it hard to believe your code actually behaves the way you describe. That's because Read() in GetMovieData() also may not fill the whole buffer. Also, StreamReader is allowed to keep some data in an internal buffer, which would mean you could read some completely bogus data.
I think code like this, where you're combining Streams and StreamReaders/StreamWriters is a really bad idea. It would be hard to make it actually correct. What you should do instead is to make your protocol completely byte-based (not character-based), even if those bytes are ASCII-encoded "SEND".
Let me give it a try, but don't shoot me if it doesn't work.
I see that you have a buffer size of 1024, regardless of how many bytes there are left in the file that you send. Say you have a file of 2900 bytes, which would require to send 3 times, the last you send there will only be 852 bytes left to send. Yet, you create a buffer of 1024 and send over 1024 bytes. This means that your server receives 852 bytes of real data, and 172 zero-filled bytes. Even though, all those 172 bytes are save to the movie file on the server.
I guess there's an easy fix: When you write the data to the server, use the currentBlockSize as argument for the length. So in method SendData on the client, inside the while loop, change:
nws.Write(buffer, 0, buffer.Length);
to this:
nws.Write(buffer, 0, currentBlockSize);
Hey, I'm having an issue seperating packets using a custom binary protocol.
Currently the server side code looks like this.
public void HandleConnection(object state)
{
TcpClient client = threadListener.AcceptTcpClient();
NetworkStream stream = client.GetStream();
byte[] data = new byte[4096];
while (true)
{
int recvCount = stream.Read(data, 0, data.Length);
if (recvCount == 0) break;
LogManager.Debug(Utility.ToHexDump(data, 0, recvCount));
//processPacket(new MemoryStream(data, 0, recvCount));
}
LogManager.Debug("Client disconnected");
client.Close();
Dispose();
}
I've been watching the hex dumps of the packets, and sometimes the entire packet comes in one shot, let's say all 20 bytes. Other times it comes in fragmented, how do I need to buffer this data to be able to pass it to my processPacket() method correctly. I'm attempting to use a single byte opcode header only, should I add something like a (ushort)contentLength to the header aswell? I'm trying to make the protocol as lightweight as possible, and this system won't be sending very large packets(< 128 bytes).
The client side code I'm testing with is, as follows.
public void auth(string user, string password)
{
using (TcpClient client = new TcpClient())
{
client.Connect(IPAddress.Parse("127.0.0.1"), 9032);
NetworkStream networkStream = client.GetStream();
using (BinaryWriter writer = new BinaryWriter(networkStream))
{
writer.Write((byte)0); //opcode
writer.Write(user.ToUpper());
writer.Write(password.ToUpper());
writer.Write(SanitizationMgr.Verify()); //App hash
writer.Write(Program.Seed);
}
}
}
I'm not sure if that could be what's messing it up, and binary protocol doesn't seem to have much info on the web, especially where C# is involved. Any comment's would be helpful. =)
Solved with this, not sure if it's correct, but it seems to give my handlers just what they need.
public void HandleConnection(object state)
{
TcpClient client = threadListener.AcceptTcpClient();
NetworkStream stream = client.GetStream();
byte[] data = new byte[1024];
uint contentLength = 0;
var packet = new MemoryStream();
while (true)
{
int recvCount = stream.Read(data, 0, data.Length);
if (recvCount == 0) break;
if (contentLength == 0 && recvCount < headerSize)
{
LogManager.Error("Got incomplete header!");
Dispose();
}
if(contentLength == 0) //Get the payload length
contentLength = BitConverter.ToUInt16(data, 1);
packet.Write(data, (int) packet.Position, recvCount); //Buffer the data we got into our MemStream
if (packet.Length < contentLength + headerSize) //if it's not enough, continue trying to read
continue;
//We have a full packet, pass it on
//LogManager.Debug(Utility.ToHexDump(packet));
processPacket(packet);
//reset for next packet
contentLength = 0;
packet = new MemoryStream();
}
LogManager.Debug("Client disconnected");
client.Close();
Dispose();
}
You should just treat it as a stream. Don't rely on any particular chunking behaviour.
Is the amount of data you need always the same? If not, you should change the protocol (if you can) to prefix the logical "chunk" of data with the length in bytes.
In this case you're using BinaryWriter on one side, so attaching a BinaryReader to the NetworkStream returned by TcpClient.GetStream() would seem like the easiest approach. If you really want to capture all the data for a chunk at a time though, you should go back to my idea of prefixing the data with its length. Then just loop round until you've got all the data.
(Make sure you've got enough data to read the length though! If your length prefix is 4 bytes, you don't want to read 2 bytes and miss the next 2...)