When i am try to read from SslStream function Read() is never end if i am not set connection timeout but if i do i've got timeout exception. This guy have the same problem http://msdn.microsoft.com/en-us/library/system.net.security.sslstream.read.aspx. I don't what to do here the code
public byte[] ReadBytes()
{
this.bufferGlobal.Clear();
byte[] buffer = new byte[this.bufferSize];
int recv = this.stream.Read(buffer, 0, buffer.Length);
while (recv != 0)
{
addBytes(buffer, ref bufferGlobal, recv);
recv = this.stream.Read(buffer, 0, buffer.Length);
}
return (byte[])this.bufferGlobal.ToArray(typeof(byte));
}
Thx in advance.
UPD:
i think i find the answer. I can set read timeout on SslStream equal one, thats value does not make sense for socket alive (its mean you can download huge files and don't worry about SslStream he wouldn't close connection or interupt recieve data). I just testing this solution but seems works fine. Thx everybody.
You are reading from a network stream, which means you will not encounter the end of stream until the other side closes its half of the connection. It isn't enough for it to stop sending data. So, make your other program close the connection after it has sent all data it intends to.
The stream won't end until the connection is closed. This is the nature of all streams of an unknown length.
You either need to know the length ahead of time or you need to keep reading until the connection is closed. A common way is to transmit the byte length of the stream as a 64bit int. So the first 8 bytes of your stream is read into an int64, then the rest is the data.
Typically one reads in a stream one "buffer" at a time.
pseudo code
while(!Stream.end)
{
i = Stream.Read(buffer, buffer.length)
DestStream.Write(buffer, i)
}
bit late but I am using msdn code with following modification
if (sb.ToString().IndexOf("a OK") != -1 || sb.ToString().IndexOf("a BAD") != -1 || sb.ToString().IndexOf("a NO") != -1)
{
break;
}
please use this working patch if it works for you:
int bytesRead = 0;
string chunkString = "";
do
{
byte[] responseBuffer = new byte[CHUNKSIZE];
bytesRead = sslStream.Read(responseBuffer, 0, responseBuffer.Length);
ApplicationHelper.ApplicationLogger.WriteInfoLog("Bytes Received " + bytesRead);
if (bytesRead > 0)
{
chunkString = Encoding.UTF8.GetString(responseBuffer, 0, bytesRead);
responseXML += chunkString;
}
}
while (!chunkString.EndsWith(EOF));
Related
I have an app that uses a TCP socket to exchange byte arrays which in most cases contain JSON string data. What I'm experiencing is that, for larger messages and less than ideal network conditions, use of NetworkStream.DataAvailable does NOT seem to be a reliable way to detect an end of message. It seems that in some cases DataAvailable is set to false even when only part of the message has been transmitted by peer (which is using TcpClient.GetStream().Write(data, 0, data.Length). This results in incomplete data being passed back to the app, which in the case of a JSON message, means deserialization fails.
I've tried two implementations which exhibit the same issue:
Implementation 1:
byte[] Data;
byte[] buffer = new byte[2048];
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = ClientStream.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
BytesRead += read;
if (!ClientStream.DataAvailable) break;
}
Data = ms.ToArray();
}
Implementation 2:
byte[] Data;
using (MemoryStream ms = new MemoryStream())
{
byte[] buffer = new byte[2048];
do
{
int read = ClientStream.Read(buffer, 0, buffer.Length);
if (read > 0)
{
ms.Write(buffer, 0, read);
BytesRead += read;
}
}
while (ClientStream.DataAvailable);
Data = ms.ToArray();
}
It seems one solution that works really well but is completely sub-optimal is to add a Thread.Sleep in case NetworkStream.DataAvailable is false (while inside the loop) to allow data to be delivered. However this severely limits overall IOPS which I would like to avoid, i.e.
Implementation 3 (works, but suboptimal)
byte[] Data;
using (MemoryStream ms = new MemoryStream())
{
byte[] buffer = new byte[2048];
do
{
int read = ClientStream.Read(buffer, 0, buffer.Length);
if (read > 0)
{
ms.Write(buffer, 0, read);
BytesRead += read;
}
if (!ClientStream.DataAvailable) System.Threading.Thread.Sleep(250);
}
while (ClientStream.DataAvailable);
Data = ms.ToArray();
}
I'd really like to find a way to remain in the loop until all of the data is delivered. As I mentioned, I'm doing a simple write operation on the client from zero to data length, so I'm not thinking there is an issue there.
Has anyone had any experience like this before and a recommendation?
It seems .DataAvailable is indeed reliable and that, since the data arrives over the network potentially at a rate slower than data is read from the stream, .DataAvailable can flip-flop between the start and end of what my application thinks is a message.
I'm answering and closing this as I believe the only solutions to this are:
1) add an over-arching receive timeout value, and perform a thread.sleep in the read loop, and expiring the operation once the receive timeout is reached
2) implement some mechanism of indicating the data payload size - either explicitly or by creating a system of metadata headers - to indicate how much data should be read, and exiting after that much data has been read or the operation has timed out
These two are the best I could come up with and seem to be validated by the likes of other TCP-based protocols like HTTP and generally any other RPC out there.
Hopefully this saves someone some time.
I am getting data from client and saving it to the local drive on local host .I have checked it for a file of 221MB but a test for file of 1Gb gives the following exception:
An unhandled exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll
Following is the code at server side where exception stems out.
UPDATED
Server:
public void Thread()
{
TcpListener tcpListener = new TcpListener(ipaddr, port);
tcpListener.Start();
MessageBox.Show("Listening on port" + port);
TcpClient client=new TcpClient();
int bufferSize = 1024;
NetworkStream netStream;
int bytesRead = 0;
int allBytesRead = 0;
// Start listening
tcpListener.Start();
// Accept client
client = tcpListener.AcceptTcpClient();
netStream = client.GetStream();
// Read length of incoming data to reserver buffer for it
byte[] length = new byte[4];
bytesRead = netStream.Read(length, 0, 4);
int dataLength = BitConverter.ToInt32(length,0);
// Read the data
int bytesLeft = dataLength;
byte[] data = new byte[dataLength];
while (bytesLeft > 0)
{
int nextPacketSize = (bytesLeft > bufferSize) ? bufferSize : bytesLeft;
bytesRead = netStream.Read(data, allBytesRead, nextPacketSize);
allBytesRead += bytesRead;
bytesLeft -= bytesRead;
}
// Save to desktop
File.WriteAllBytes(#"D:\LALA\Miscellaneous\" + shortFileName, data);
// Clean up
netStream.Close();
client.Close();
}
I am getting the file size first from client side followed by data.
1).Should i increase the buffer size or any other technique ?
2). File.WriteAllBytes() and File.ReadAllBytes() seems blocking and freezes the PC.Is there any async method for it to help provide the progress of file recieved at server side.
You don't need to read the whole thing to memory before writing it to disc. Just copy straight from the network stream to a FileStream:
byte[] length = new byte[4];
// TODO: Validate that bytesRead is 4 after this... it's unlikely but *possible*
// that you might not read the whole length in one go.
bytesRead = netStream.Read(length, 0, 4);
int bytesLeft = BitConverter.ToInt32(length,0);
using (var output = File.Create(#"D:\Javed\Miscellaneous\" + shortFileName))
{
netStream.CopyTo(output, bytesLeft);
}
Note that instead of calling netStream.Close() explicitly, you should use a using statement:
using (Stream netStream = ...)
{
// Read from it
}
That way the stream will be closed even if an exception is thrown.
The CLR has a per-object limit a bit short of 2GB. However that's the theory, in practice how much memory you can allocate depends on how much memory the framework allows you to allocate. I wouldn't expect it to allow you to allocate 1 GB data table. You should allocate smaller table, and write the data in chunks into disk file.
The "out of memory" exception happens because you are trying to place the entire file into memory before dumping it on disk. This is suboptimal, because you don't need the entire file in memory in order to write into the file: you can read it block-by-block in reasonably-sized increments, and write it out as you go.
Starting with .NET 4.0 you can use Stream.CopyTo method to accomplish this in a few lines of code:
// Read and ignore the initial four bytes of length from the stream
byte[] ignore = new byte[4];
int bytesRead = 0;
do {
// This should complete in a single call, but the API requires you
// to do it in a loop.
bytesRead += netStream.Read(ignore, bytesRead, 4-bytesRead);
} while (bytesRead != 4);
// Copy the rest of the stream to a file
using (var fs = new FileStream(#"D:\Javed\Miscellaneous\" + shortFileName, FileMode.Create)) {
netStream.CopyTo(fs);
}
netStream.Close();
Starting with .NET 4.5 you can use CopyToAsync, too, which would give you a way to do reading and writing asynchronously.
Note the code that drops the initial four bytes from the stream. This is done to avoid writing the length of the stream along with the "payload" bytes. If you have control over the network protocol, you could change the sending side to stop prefixing the stream with its length, and remove the code that reads and ignores it on the receiving side.
I've got a client / server application that works pretty well, but it's missing one crucial piece of behavior to make it a bit more solid.
Right now, it's far from "strong" in terms of network capabilities. I'm trying to get it there, and research has lead me to believe that I need some sort of protocol in place to ensure that no data is lost during network transmissions.
I've heard of a few methods. One that I think will work best for our situations is to use a terminator, something like an <EOF> tag. My issue is that I'm not sure of the best way to implement this.
Here's a couple code snippets that I'll be modifying to include a terminator after figuring out the best solution.
Client:
TcpClient client = new TcpClient();
client.Connect(hostname, portNo);
using (var stream = client.GetStream())
{
//send request
stream.Write(data, 0, data.Length);
stream.Flush();
//read server response
if (stream.CanRead)
{
byte[] buffer = new byte[1024];
string response = "";
int bytesRead = 0;
do
{
bytesRead = stream.Read(buffer, 0, buffer.Length);
response += Encoding.ASCII.GetString(buffer, 0, bytesRead);
} //trying to replace 'DataAvailable', it doesn't work well
while (stream.DataAvailable);
}
}
Note that I'm trying to replace the stream.DataAvailable method of checking for more data in the stream. It's been causing problems.
Server:
var listener = new TcpListener(IPAddress.Any, portNo);
listener.Start();
var client = listener.AcceptTcpClient();
using (var stream = client.GetStream())
{
var ms = new System.IO.MemoryStream();
byte[] buffer = new byte[4096];
int bytesRead = 0;
do
{
bytesRead = stream.Read(buffer, 0, buffer.Length);
ms.Write(buffer, 0, bytesRead);
} //also trying to replace this 'stream.DataAvailable'
while (stream.DataAvailable);
ms.Position = 0;
string requestString = Encoding.UTF8.GetString(ms.ToArray());
ms.Position = 0;
/*
process request and create 'response'
*/
byte[] responseBytes = Encoding.UTF8.GetBytes(response);
stream.Write(responseBytes, 0, responseBytes.Length);
stream.Flush();
}
So, given these two code examples, how can I modify these to both include and check for some sort of data terminator that indicates it's safe to stop reading data?
You can rely on TCP transmitting all the data before the FIN. The problem with your code is that available() is not a valid test for end of stream. It is a test for data being available now.
So you are terminating your reading loop prematurely, and thus missing data at the receiver, and getting resets at the sender.
You should just block in the Read() method until you receive the EOS indication from the API, whatever that is in C#.
You don't need your own additional EOS indicator.
We ended up using an EOS (end of stream) indicator on both ends of our project. I won't post the full code example, but here's a small snippet of how it works:
stream.Write(data, 0, data.Length);
stream.WriteByte(Convert.ToByte(ConsoleKey.Escape));
stream.Flush();
On the receiving end of this stream, the loop reads data byte-by-byte. Once it receives the ConsoleKey.Escape byte, it terminates the loop. It works!
Was just wondering if anybody can look at the following code and explain to me why the length variable returns 0:
textToBeEncrypted = Encoding.ASCII.GetString(buffer);
txtEncryptedText = AESEncryption(textToBeEncrypted, key, true);
byte[] encText = Encoding.ASCII.GetBytes(txtEncryptedText);
NetworkStream stream = s.GetStream();
stream.Write(encText, 0, PACKET_SIZE);
s.ReceiveTimeout = Timeout;
int length = stream.Read(buffer, 0, PACKET_SIZE);
if (length == PACKET_SIZE)
{
string decText = Encoding.ASCII.GetString(encText);
txtDecryptedText = AESDecryption(decText, key, true);
buffer = Encoding.ASCII.GetBytes(txtDecryptedText);
retval = Decode();
}
After I've encoded everything using AES, I'm writing out 1366 bytes of data in encText (PACKET_SIZE is 1036). I get no complaints regarding the Send; the data is sent out happily. When it tries to read it back in, however, length is always 0, meaning I don't get to enter the decode statement bracket. Any ideas? (retval is a string, before anyone asks)
If the length is zero from this:
int length = stream.Read(buffer, 0, PACKET_SIZE);
it means the other machine has closed their outbound socket (your inbound socket), and no more data is ever going to be available.
You should also be very careful about this:
if (length == PACKET_SIZE)
{...}
There is absolutely no guarantee about what you read. What you should do here is buffer the data until you have an entire message (frame), and then process what you buffered. In particular, if the other end sends less than PACKET_SIZE bytes, your code is guaranteed to do nothing. Even if the other end sent exactly PACKET_SIZE bytes, it would be pretty unlikely to arrive in exactly a single chunk of PACKET_SIZE bytes.
For example:
int length;
MemoryStream ms = new MemoryStream();
while((length = stream.Read(buffer, 0, buffer.Length)) > 0) {
ms.Write(buffer, 0, length); // append what we just recieved
// now: could check `ms` to see if we have a "frame" here...
}
//...or you could just process the entire recieved data here
Hey, I'm having an issue seperating packets using a custom binary protocol.
Currently the server side code looks like this.
public void HandleConnection(object state)
{
TcpClient client = threadListener.AcceptTcpClient();
NetworkStream stream = client.GetStream();
byte[] data = new byte[4096];
while (true)
{
int recvCount = stream.Read(data, 0, data.Length);
if (recvCount == 0) break;
LogManager.Debug(Utility.ToHexDump(data, 0, recvCount));
//processPacket(new MemoryStream(data, 0, recvCount));
}
LogManager.Debug("Client disconnected");
client.Close();
Dispose();
}
I've been watching the hex dumps of the packets, and sometimes the entire packet comes in one shot, let's say all 20 bytes. Other times it comes in fragmented, how do I need to buffer this data to be able to pass it to my processPacket() method correctly. I'm attempting to use a single byte opcode header only, should I add something like a (ushort)contentLength to the header aswell? I'm trying to make the protocol as lightweight as possible, and this system won't be sending very large packets(< 128 bytes).
The client side code I'm testing with is, as follows.
public void auth(string user, string password)
{
using (TcpClient client = new TcpClient())
{
client.Connect(IPAddress.Parse("127.0.0.1"), 9032);
NetworkStream networkStream = client.GetStream();
using (BinaryWriter writer = new BinaryWriter(networkStream))
{
writer.Write((byte)0); //opcode
writer.Write(user.ToUpper());
writer.Write(password.ToUpper());
writer.Write(SanitizationMgr.Verify()); //App hash
writer.Write(Program.Seed);
}
}
}
I'm not sure if that could be what's messing it up, and binary protocol doesn't seem to have much info on the web, especially where C# is involved. Any comment's would be helpful. =)
Solved with this, not sure if it's correct, but it seems to give my handlers just what they need.
public void HandleConnection(object state)
{
TcpClient client = threadListener.AcceptTcpClient();
NetworkStream stream = client.GetStream();
byte[] data = new byte[1024];
uint contentLength = 0;
var packet = new MemoryStream();
while (true)
{
int recvCount = stream.Read(data, 0, data.Length);
if (recvCount == 0) break;
if (contentLength == 0 && recvCount < headerSize)
{
LogManager.Error("Got incomplete header!");
Dispose();
}
if(contentLength == 0) //Get the payload length
contentLength = BitConverter.ToUInt16(data, 1);
packet.Write(data, (int) packet.Position, recvCount); //Buffer the data we got into our MemStream
if (packet.Length < contentLength + headerSize) //if it's not enough, continue trying to read
continue;
//We have a full packet, pass it on
//LogManager.Debug(Utility.ToHexDump(packet));
processPacket(packet);
//reset for next packet
contentLength = 0;
packet = new MemoryStream();
}
LogManager.Debug("Client disconnected");
client.Close();
Dispose();
}
You should just treat it as a stream. Don't rely on any particular chunking behaviour.
Is the amount of data you need always the same? If not, you should change the protocol (if you can) to prefix the logical "chunk" of data with the length in bytes.
In this case you're using BinaryWriter on one side, so attaching a BinaryReader to the NetworkStream returned by TcpClient.GetStream() would seem like the easiest approach. If you really want to capture all the data for a chunk at a time though, you should go back to my idea of prefixing the data with its length. Then just loop round until you've got all the data.
(Make sure you've got enough data to read the length though! If your length prefix is 4 bytes, you don't want to read 2 bytes and miss the next 2...)