Develop Reference

C# TCPClient-Server doesn't always send the full data

I'm currently working on a TCPClient and Server. Lately I added an encryption for the messages, and had no trouble. Once I started noticing that I'm getting a weird error like this:
But It's totally random, and no idea why. It happens at larger messages, but as I said, not always.
Checking the byte[] length at the server side says 1920 (Sometimes it says 1920 on the client too, and thats the point when i dont have error)
On client it says a lot lesser.
I actually think that sometimes the client doesn't receive the full byte that It should, this is how I do It:
Client:
byte[] bb = new byte[12288];
int k = stm.Read(bb, 0, 12288);
string message = Encoding.UTF8.GetString(bb, 0, k);
MessageBox.Show(message.Length.ToString()); // This sometimes says 1460, and 1920
message = Encrypter.DecryptData(message); // Error here If the length is not 1920
Server:
bmsg = Encrypter.EncryptData(((int)Codes.XYZEnum) + "=" + data);
Logger.Log(bmsg.Length.ToString()); // Original msg, always says 1920
s.Send(asen.GetBytes(bmsg));
s.Close();
What could be the problem? Should I try async sending?
SOLUTION:
Server code, took me a little while to make it cool:
System.Net.Sockets.Socket s = myList.AcceptSocket(); // Accept the connection
Stream stream = new NetworkStream(s); // Create the stream object
byte[] leng = new byte[4]; // We will put the length of the upcoming message in a 4 length array
int k2 = s.Receive(leng); // Receive the upcoming message length
if (BitConverter.IsLittleEndian)
{
Array.Reverse(leng);
}
int upcominglength = (BitConverter.ToInt32(leng, 0)); // Convert it to int
byte[] b = ByteReader(upcominglength, stream); // Create the space for the bigger message, read all bytes until the received length!
string message = Encoding.UTF8.GetString(b, 0, b.Length); // Convert it to string!
internal byte[] ByteReader(int length, Stream stream)
{
byte[] data = new byte[length];
using (MemoryStream ms = new MemoryStream())
{
int numBytesRead;
int numBytesReadsofar = 0;
while (true)
{
numBytesRead = stream.Read(data, 0, data.Length);
numBytesReadsofar += numBytesRead;
ms.Write(data, 0, numBytesRead);
if (numBytesReadsofar == length)
{
break;
}
}
return ms.ToArray();
}
}
Client code, and it is working nicely!:
var result = tcpclnt.BeginConnect(User.IP, User.Port, null, null);
var success = result.AsyncWaitHandle.WaitOne(TimeSpan.FromSeconds(3)); // Connect with timeout
if (!success)
{
return "Failed to connect!";
}
Stream stm = tcpclnt.GetStream(); // get the stream
UTF8Encoding asen = new UTF8Encoding();
byte[] ba = asen.GetBytes(msg); // get the bytes of the message we are sending
byte[] intBytes = BitConverter.GetBytes(ba.Length); // Get the length of that in bytes
if (BitConverter.IsLittleEndian)
{
Array.Reverse(intBytes);
}
stm.Write(intBytes, 0, intBytes.Length); // Write the length in the stream!
stm.Flush(); // Clear the buffer!
stm.Write(ba, 0, ba.Length); // Write the message we are sending!
// If we have answers....
byte[] bb = new byte[10000];
int k = stm.Read(bb, 0, 10000);
string mmessage = Encoding.UTF8.GetString(bb, 0, k);
// If we have answers....
tcpclnt.Close(); // Close the socket

Because only 8Kb can to send by once packet. if you have large data you need use cycle.

C# - Is there a limit to the size of an httpWebRequest stream?

I am trying to build an application that downloads a small binary file (20-25 KB) from a custom webserver using httpwebrequests.
This is the server-side code:
Stream UpdateRequest = context.Request.InputStream;
byte[] UpdateContent = new byte[context.Request.ContentLength64];
UpdateRequest.Read(UpdateContent, 0, UpdateContent.Length);
String remoteVersion = "";
for (int i = 0;i < UpdateContent.Length;i++) { //check if update is necessary
remoteVersion += (char)UpdateContent[i];
}
byte[] UpdateRequestResponse;
if (remoteVersion == remotePluginVersion) {
UpdateRequestResponse = new byte[1];
UpdateRequestResponse[0] = 0; //respond with a single byte set to 0 if no update is required
} else {
FileInfo info = new FileInfo(Path.Combine(Directory.GetCurrentDirectory(), "remote logs", "PointAwarder.dll"));
UpdateRequestResponse = File.ReadAllBytes(Path.Combine(Directory.GetCurrentDirectory(), "remote logs", "PointAwarder.dll"));
//respond with the updated file otherwise
}
//this byte is past the threshold and will not be the same in the version the client recieves
Console.WriteLine("5000th byte: " + UpdateRequestResponse[5000]);
//send the response
context.Response.ContentLength64 = UpdateRequestResponse.Length;
context.Response.OutputStream.Write(UpdateRequestResponse, 0, UpdateRequestResponse.Length);
context.Response.Close();
After this the array UpdateRequestResponse contains the entire file and has been sent to the client.
The client runs this code:
//create the request
WebRequest request = WebRequest.Create(url + "pluginUpdate");
request.Method = "POST";
//create a byte array of the current version
byte[] requestContentTemp = version.ToByteArray();
int count = 0;
for (int i = 0; i < requestContentTemp.Length; i++) {
if (requestContentTemp[i] != 0) {
count++;
}
}
byte[] requestContent = new byte[count];
for (int i = 0, j = 0; i < requestContentTemp.Length; i++) {
if (requestContentTemp[i] != 0) {
requestContent[j] = requestContentTemp[i];
j++;
}
}
//send the current version
request.ContentLength = requestContent.Length;
Stream dataStream = request.GetRequestStream();
dataStream.Write(requestContent, 0, requestContent.Length);
dataStream.Close();
//get and read the response
WebResponse response = request.GetResponse();
Stream responseStream = response.GetResponseStream();
byte[] responseBytes = new byte[response.ContentLength];
responseStream.Read(responseBytes, 0, (int)response.ContentLength);
responseStream.Close();
response.Close();
//if the response containd a single 0 we are up-to-date, otherwise write the content of the response to file
if (responseBytes[0] != 0 || response.ContentLength > 1) {
BinaryWriter writer = new BinaryWriter(File.Open(Path.Combine(Directory.GetCurrentDirectory(), "ServerPlugins", "PointAwarder.dll"), FileMode.Create));
writer.BaseStream.Write(responseBytes, 0, responseBytes.Length);
writer.Close();
TShockAPI.Commands.HandleCommand(TSPlayer.Server, "/reload");
}
The byte array responseBytes on the client should be identical to the array UpdateRequestResponse on the server, but it isn't. after about 4000 bytes every byte after that is set to 0 rather than what it should be (responseBytes[3985] is the last non-zero byte).
Does this happen because httpWebRequest has a size limit? I can't see any bug in my code that could be causing it and the same code works in other instances where I only have to pass around short sequences of data (less than 100 bytes).
The MSDN pages don't mention any size limit like this.

It's not that it has any artificial limit, this is a byproduct of the Streaming nature of what you're attempting to do. I have a feeling the following line is the offender:
responseStream.Read(responseBytes, 0, (int)response.ContentLength);
I've had this issue in the past (with TCP streams), it doesn't read all of the contents of the array, because they haven't all been sent over the wire yet. This is what I would try instead.
for (int i = 0; i < response.ContentLength; i++)
{
responseBytes[i] = responseStream.ReadByte();
}
That way, it will make sure to read all the way until the end of the stream.
EDIT
usr's BinaryReader based solution is much more efficient. Here is the relevant solution:
BinaryReader binReader = new BinaryReader(responseStream);
const int bufferSize = 4096;
byte[] responseBytes;
using (MemoryStream ms = new MemoryStream())
{
byte[] buffer = new byte[bufferSize];
int count;
while ((count = binReader.Read(buffer, 0, buffer.Length)) != 0)
ms.Write(buffer, 0, count);
responseBytes = ms.ToArray();
}

You are assuming that Read is reading as many bytes as you request. But the requested count is just an upper limit. You must tolerate reading small chunks.
You can use var bytes = new BinaryReader(myStream).ReadBytes(count); to read an exact number. Don't call ReadByte too often because that is very CPU intensive.
The best solution would be to step away from the fairly manual HttpWebRequest and use HttpClient or WebClient. All of this is automated for you and you get back a byte[].

Sending and receiving images via a socket

I have a C# desktop app. It connects to another PC on my network which is a UWP C# app.
I am trying to send an image or 2 to my listening socket and to test this I get the listening socket to send me the image back.
The trouble is that even though my server recieves all the bytes that were orginally sent the recieved image back to the client is not of the same size.
To make this even more weird is sometimes the returned bytes are correct and I get the whole image and when I attempt to send 2 images the 1st one is OK and the 2nd one is not.
Then it will/can revert back to no images being sent back correctly.
I think is maybe to do with the async/await parts bit I am not sure how.
This is my server code:
using (IInputStream input = args.Socket.InputStream)
{
byte[] data = new byte[BufferSize];
IBuffer buffer = data.AsBuffer();
uint dataRead = BufferSize;
while (dataRead == BufferSize)
{
await input.ReadAsync(buffer, BufferSize, InputStreamOptions.Partial);
requestInBytes.AddRange(data.Take((int) buffer.Length));
dataRead = buffer.Length;
}
}
var ct = requestInBytes.Count;
I then trip out the header info:
int counter = 0;
counter = requestCommand[0].Length;
counter = counter + requestCommand[1].Length;
counter = counter + requestCommand[2].Length;
counter = counter + requestCommand[3].Length;
counter = counter + requestCommand[4].Length;
counter = counter + requestCommand[5].Length;
counter = counter + 6;
Now I extract the image:
var imgBody = new byte[totalBytes.Length- counter];
System.Buffer.BlockCopy(totalBytes, counter, imgBody, 0, imgBody.Length);
byteArray = imgBody;
And send just the image back:
using (IOutputStream output = args.Socket.OutputStream)
{
using (Stream response = output.AsStreamForWrite())
{
MemoryStream stream = new MemoryStream(byteArray);
await response.WriteAsync(byteArray, 0, byteArray.Length);
await response.FlushAsync();
}
}
This is my client code:
StringBuilder sb = new StringBuilder();
foreach (var gallery in Shared.CurrentJobGallery)
{
try
{
sb.Clear();
sb.Append(GeneralTags.ACTION_ADD);
sb.Append(Shared.DLE);
sb.Append("GALLERY");
sb.Append(Shared.DLE);
sb.Append(Shared.CurrentClientId);
sb.Append(Shared.DLE);
sb.Append(gallery.Title);
sb.Append(Shared.DLE);
sb.Append(gallery.Description);
sb.Append(Shared.DLE);
sb.Append(jobRef);
sb.Append(Shared.DLE);
byte[] galleryHdr = Encoding.UTF8.GetBytes(sb.ToString());
byte[] byteArray = new byte[galleryHdr.Length + gallery.ImageData.Length];
Buffer.BlockCopy(galleryHdr, 0, byteArray, 0, galleryHdr.Length);
Buffer.BlockCopy(gallery.ImageData, 0, byteArray, galleryHdr.Length, gallery.ImageData.Length);
List<byte> requestInBytes2 = new List<byte>();
System.Diagnostics.Debug.WriteLine("SENT: " + gallery.ImageData.Length.ToString());
using (TcpClient clientSocket = new TcpClient())
{
await clientSocket.ConnectAsync(GeneralTags.RASPBERRY_PI_IP_ADDRESS, GeneralTags.RASPBERRY_PI_PORT);
using (NetworkStream serverStream = clientSocket.GetStream())
{
List<byte> requestInBytes = new List<byte>();
serverStream.Write(byteArray, 0, byteArray.Length);
serverStream.Flush();
int i;
Byte[] bytes = new Byte[1024];
do
{
i = serverStream.Read(bytes, 0, bytes.Length);
byte[] receivedBuffer = new byte[i];
Array.Copy(bytes, receivedBuffer, i);
requestInBytes2.AddRange(receivedBuffer);
} while (serverStream.DataAvailable);
}
}
using (MemoryStream ms = new MemoryStream())
{
System.Diagnostics.Debug.WriteLine("BACK: " + requestInBytes2.Count.ToString());
ms.Write(requestInBytes2.ToArray(), 0, requestInBytes2.ToArray().Length);
Shared.ViewImage(Image.FromStream(ms, true));
}
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine(ex.ToString());
}
}

Your problem is that TCP sockets are based around streams, not packets. It's true that "on the wire" everything is a packet, but when you're using TCP, you have no control over how the data is split up into packets or is reassembled into a stream.
In particular, this line of code is incorrect:
await input.ReadAsync(buffer, BufferSize, InputStreamOptions.Partial);
According to the docs, you must use the buffer returned from ReadAsync. Also note that this buffer may be a partial image, and it's up to your code to detect that situation, read more if necessary, and append those blocks together. Also, the buffer may contain part of one image and part of the next image; it's also up to your code to detect that and handle it correctly.
For this reason, most TCP applications use some form of message framing (described in more detail on my blog). Note that getting this right is surprisingly hard.
I strongly recommend that you use SignalR instead of raw TCP sockets. SignalR handles message framing for you, and it is capable of self-hosting (i.e., it does not require ASP.NET).

How to get all data from NetworkStream

I am trying to read all data present in the buffer of the Machine connected through TCP/IP but i don't know why i am not getting all data ,some data is getting Missed.
Here is the code that i am using ..
using (NetworkStream stream = client.GetStream())
{
byte[] data = new byte[1024];
int numBytesRead = stream.Read(data, 0, data.Length);
if (numBytesRead > 0)
{
string str= Encoding.ASCII.GetString(data, 0, numBytesRead);
}
}
Please tell me what i am missing to get all the data from the machine.
Thanks in advance..

The problem with your code is that you will not get all the data if the data size is bigger than the buffer size (1024 bytes in your case) so you have to Read the stream inside the loop. Then you can Write all the data inside a MemoryStream until the end of the NetworkStream.
string str;
using (NetworkStream stream = client.GetStream())
{
byte[] data = new byte[1024];
using (MemoryStream ms = new MemoryStream())
{
int numBytesRead ;
while ((numBytesRead = stream.Read(data, 0, data.Length)) > 0)
{
ms.Write(data, 0, numBytesRead);
}
str = Encoding.ASCII.GetString(ms.ToArray(), 0, (int)ms.Length);
}
}

This example from MSDN: NetworkStream.DataAvailable shows how you can use that property to do so:
// Examples for CanRead, Read, and DataAvailable.
// Check to see if this NetworkStream is readable.
if(myNetworkStream.CanRead)
{
byte[] myReadBuffer = new byte[1024];
StringBuilder myCompleteMessage = new StringBuilder();
int numberOfBytesRead = 0;
// Incoming message may be larger than the buffer size.
do{
numberOfBytesRead = myNetworkStream.Read(myReadBuffer, 0, myReadBuffer.Length);
myCompleteMessage.AppendFormat("{0}", Encoding.ASCII.GetString(myReadBuffer, 0, numberOfBytesRead));
}
while(myNetworkStream.DataAvailable);
// Print out the received message to the console.
Console.WriteLine("You received the following message : " +
myCompleteMessage);
}
else
{
Console.WriteLine("Sorry. You cannot read from this NetworkStream.");
}

Try this:
private string GetResponse(NetworkStream stream)
{
byte[] data = new byte[1024];
using (MemoryStream memoryStream = new MemoryStream())
{
do
{
stream.Read(data, 0, data.Length);
memoryStream.Write(data, 0, data.Length);
} while (stream.DataAvailable);
return Encoding.ASCII.GetString(memoryStream.ToArray(), 0, (int)memoryStream.Length);
}
}

Try this code:
using (NetworkStream stream = client.GetStream())
{
while (!stream.DataAvailable)
{
Thread.Sleep(20);
}
if (stream.DataAvailable && stream.CanRead)
{
Byte[] data = new Byte[1024];
List<byte> allData = new List<byte>();
do
{
int numBytesRead = stream.Read(data,0,data.Length);
if (numBytesRead == data.Length)
{
allData.AddRange(data);
}
else if (numBytesRead > 0)
{
allData.AddRange(data.Take(numBytesRead));
}
} while (stream.DataAvailable);
}
}
Hope this helps, it should prevent that you miss any data sended to you.

The synchronous method sometimes does not display the request body. Using the asynchronous method stably displays request body.
string request = default(string);
StringBuilder sb = new StringBuilder();
byte[] buffer = new byte[client.ReceiveBufferSize];
int bytesCount;
if (client.GetStream().CanRead)
{
do
{
bytesCount = client.GetStream().ReadAsync(buffer, 0, buffer.Length).Result;
sb.Append(Encoding.UTF8.GetString(buffer, 0, bytesCount));
}
while(client.GetStream().DataAvailable);
request = sb.ToString();
}

TCP itself does not have any ways to define "end of data" condition. This is responsibility of application level portocol.
For instance see HTTP request description:
A client request (consisting in this case of the request line and only one header field) is followed by a blank line, so that the request ends with a double newline
So, for request end of data is determined by two newline sequences. And for response:
Content-Type specifies the Internet media type of the data conveyed by the HTTP message, while Content-Length indicates its length in bytes.
The response content size is specified in header before data.
So, it's up to you how to encode amount of data transferred at once - it can be just first 2 or 4 bytes in the beginning of the data holding total size to read or more complex ways if needed.

for my scenario, the message itself was telling the length of subsequent message. here is the code
int lengthOfMessage=1024;
string message = "";
using (MemoryStream ms = new MemoryStream())
{
int numBytesRead;
while ((numBytesRead = memStream.Read(MessageBytes, 0, lengthOfMessage)) > 0)
{
lengthOfMessage = lengthOfMessage - numBytesRead;
ms.Write(MessageBytes, 0, numBytesRead);
}
message = Encoding.ASCII.GetString(ms.ToArray(), 0, (int)ms.Length);
}

#George Chondrompilas answer is correct but instead of writing it by yourself you can use CopyTo function which does the same :
https://stackoverflow.com/a/65188160/4120180

TCPClient. How do I receive big messages?

I have the following code:
private string Connect()
{
string responseData;
try
{
TcpClient client = new TcpClient(ServerIp, Port);
client.ReceiveBufferSize = Int32.MaxValue;
Byte[] data = Encoding.GetEncoding(1251).GetBytes(ReadyQuery);
NetworkStream stream = client.GetStream();
// send data
stream.Write(data, 0, data.Length);
// buffer
data = new Byte[65536];
Int32 bytes = stream.Read(data, 0, data.Length);
responseData = Encoding.GetEncoding(1251).GetString(data, 0, bytes);
// close all
stream.Close();
client.Close();
return responseData;
}
I have problem with a big message. The receive message size is 22K chars. I get only part of message.
How can I receive big messages?
PS. In the debugger bytes equal 4096.

You call stream.Read in a loop until you read the entire message. If you know the message size in advance it's relatively easy:
int messageSize = 22000;
int readSoFar = 0;
byte [] msg = new byte[messageSize];
while(readSoFar < messageSize)
{
var read = stream.Read(msg, readSoFar, msg.Length - readSoFar);
readSoFar += read;
if(read==0)
break; // connection was broken
}
If the message size is part of the message (say, encoded in the first 4 bytes), you should read those first and then do as I suggested.