I'm using the following code to send a file over tcp.
If i send many times the same file consecutively to test if it is robust, i receive the first file correctly and the other messed up.
All messed up files have the same incorrect bytes and if i Sleep(a while) all files are transfered correctly. I noticed I must instantiate a new buffer while reading my file to get everything done right. But i don't get why.
I fear my solution to reinstantiate a buffer could be just hiding another major problem. Any suggestion?
using(var fileStream = new FileStream(file, FileMode.Open, FileAccess.Read))
{
using(var binaryReader = new BinaryReader(fileStream))
{
var _sendingBuffer = new byte[BUFFER_SIZE];
int length = (int)fileStream.Length;
int bytesRead = 0;
//Ensure we reached the end of the stream regardless of encoding
while (binaryReader.BaseStream.Position != binaryReader.BaseStream.Length)
{
bytesRead = binaryReader.Read( _sendingBuffer, 0, _sendingBuffer.Length);
_socket.BeginSend(_sendingBuffer, 0, bytesRead, SocketFlags.None, SendFileCallback, null);
//without this i received some messed up data
_sendingBuffer = new byte[BUFFER_SIZE];
}
}
}
BeginSend is an asynchronous operation. It will only be guaranteed to be started after you call it, it won't be finished immediatly. As long as the socket is sending the passed data, that data must not be mutated.
The end of the operation will be signaled through the AsyncCallback callback parameter.
Your problem is exactly that you mutated the transmit buffer while the transmit was still in progress. Creating a new array for each transmit call fixes this.
Other ways to fix the problem:
Use the blocking Socket.Send function which will block until the whole data was sent and the buffer can be reused. This will also make your error handling much easier, because the error will not show up through the AsyncCallback.
Make your complete program acting asynchronously, e.g. using C#5's async Task and async/await functionalities
Therefore:
First read one part of the file asynchronously.
When the async read finishes send it asynchronously through the socket
When this completes and there is more data to read go back to 1.
Related
This question already has an answer here:
TcpClient's NetworkStream reads incomplete data unless Thread.Sleep(1) is called
(1 answer)
Closed 1 year ago.
So I have read more on this online and haven't found any solution. I know the problem is that my ReadAsync is faster than the connection is sending data. But I don't like to use the Thread.Sleep(1) because maybe 1 is not enough, maybe there is a hiccup in the connection. Anything can happen.
This is my code, sometimes my dataBytes Length == 0. But if I debug and set a breakpoint, there is always data. When I set a Thread.Sleep of 500ms works.
So what is happening is that DataAvailable is sometimes false, while more data is coming.
using (var client = (TcpClient)c)
{
using (NetworkStream stream = client.GetStream())
{
using (MemoryStream memory = new MemoryStream())
{
do
{
byte[] b = new byte[256];
int read = await stream.ReadAsync(b, 0, b.Length);
await memory.WriteAsync(b, 0, read);
} while (stream.DataAvailable && stream.CanRead);
memory.Seek(0, SeekOrigin.Begin);
byte[] dataBytes = memory.ToArray();
}
}
}
To test this connection I use RestSharp to send an HTTP message to the code above.
My question is, how can I fix this in a way I am not depending on a Thread.Sleep.
DataAvailable doesn't tell you whether more data is coming; it only tells you whether data is available in the buffers right now, and can be useful in deciding whether to do a synchronous read vs an asynchronous read. It should not have any place in a while loop that determines the end of the read operation.
If you want to read a socket to the end of the data ever, you need to keep reading until Read returns a non-positive value (assuming you're not doing zero-length reads for async IO reasons, which you aren't). However, most socket work doesn't involve a scenario where you can just read from start to end in one big chunk, and you are instead required to implement "framing" (see the second section here), allowing you to detect individual messages (which are completely different to individual Read calls).
I have an exe which simulates a video stream. I connect to it and very occasionally read the expected data, but usually I get only the first 28 bytes and then 65508 bytes of zeros. Assume the video stream is working correctly.
TcpClient tcpClient = new TcpClient ();
int port = 13000;
myIP = IPAddress.Loopback.ToString();
tcpClient.Connect (myIP, port);
NetworkStream netStream = tcpClient.GetStream ();
byte[] bytes = new byte[tcpClient.ReceiveBufferSize];
netStream.Read (bytes, 0, (int)tcpClient.ReceiveBufferSize);
string dataString = Encoding.ASCII.GetString (bytes);
Console.WriteLine("\ndataString: "+dataString.Substring(0,1000));
Console.WriteLine("\nnumber of bytes read: "+bytes.Length);
tcpClient.Close ();
// Closing the tcpClient instance does not close the network stream.
netStream.Close();
How can I make it so that I get the expected output every time?
TCP represents a (bi-directional) stream of data. You're supposed to keep reading from it in a loop, and parsing the data as you need them. It doesn't have a concept of messages - ten writes on one side can result in a single read on the other side just as easily as one write on one side can result in ten reads on the other side.
The contract you have with TCP is as follows:
If there is data in the receive buffer, Read returns immediately, filling the buffer you provided with as much data as is available, up to the length of the buffer. The number of bytes read is the return value of Read.
If there is no data in the receive buffer, Read will block until there's at least a single byte of data. Then it follows as in the first case.
If the other sides shuts down the socket, Read will return zero.
So to get TCP working, you need a loop. How exactly you form the loop depends on what you're trying to do. If you're really working with data that is logically a stream (e.g. audio data), just keep reading as fast as you can and process whatever data you get as it comes in. If you need to send messages, you need to implement a message protocol. If you need a one-off message, you can just keep reading until Read returns zero.
Your case can be handled with the first approach - keep reading until the stream closes, and push the received data forward. Assuming the data is actually a UTF8 stream of text, the basic receiver would look something like this:
using (var client = new TcpClient())
{
tcpClient.Connect(myIP, port);
var stream = client.GetStream();
var buffer = new byte[4096]; // Adapt the size based on what you want to do with the data
var charBuffer = new char[4096];
var decoder = Encoding.UTF8.GetDecoder();
int bytesRead;
while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) != 0)
{
var expectedChars = decoder.GetCharCount(buffer, 0, bytesRead);
if (charBuffer.Length < expectedChars) charBuffer = new char[expectedChars];
var charCount = decoder.GetChars(buffer, 0, bytesRead, charBuffer, 0);
Console.Write(new string(charBuffer, 0, charCount));
}
client.Close();
}
Note that this does no error handling, so don't use it as is in any production code. You would probably also want to use Async methods if you're expecting more than a few simultaneous connections. It's just to illustrate the basic way one would handle a stream of data being received over TCP.
If you want some more insight into dealing with TCP, I have a few very simple examples at https://github.com/Luaancz/Networking. I haven't found any good tutorials or code samples for C#, so if this isn't enough, you'll probably have to dig deeper into the documentation around sockets, TCP and all that.
Or just use an existing networking library, rather than trying to write your own :) TCP is still very low level.
I am have a pretty weird issue right now with how i'm sending my data over sockets. My read buffer is getting all the data i'm sending through the socket, but for some reason the File.Write or File.WriteAsync methods i'm calling are not writing all the bytes, or just stopping(I think that's the issue) OR it could be something wrong with the sockets and how they're being read and written too.
So here is a snippet of my code, keep in mind, both of these are on the same file but are different threads.
byte[] Buffer = new Byte[1024];
int bytesRead;
while((bytesRead = ReceiverSocket.Receive(Buffer)) > 0)
{
totalBytesRead += bytesRead;
//myDownload.Write(Buffer, 0, bytesRead);
myDownload.WriteAsync(Buffer, 0, bytesRead);
}
The code above is my receive function for my download client. Keep in mind i'm getting the same results regardless of what write method I use.
And I am simply using this:
byte[] fileData = File.ReadAllBytes(fileToSend);
serverSocket.SendFile(fileToSend);
I send a file with filesize: 10560 and actually retrieve all the bytes over the receive loop, yet when I check my file after i'm done it only receives a portion like so:
TotalBytesRead
1024
2048
3072
...
...8192
9216
10240
10560
Which means it reads all the data, though in my program, it's still stuck inside the loop and when i close it to check the filesize, it's size is 8192, so it seems to me it didn't read the next 3 socket streams.
I'm at a loss for this, is there anything i'm doing wrong? Could this be a threading issue i'm not aware of?
I figured out the answer, I was locking on my
while((bytesRead = ReceiverSocket.Receive(Buffer)) > 0)
due to the socket always waiting on a receive, so the SendFile wouldn't necessarily send 0, so it was always stuck in the while loop, but one I closed the while, it finished writing to file.
I am writing an application that is interested in the status information from certain network devices. One of the devices provides the status information through Http and uses a multipart message; The first time you query the information it sends down the whole status and from then on whenever the status of the device changes a new multipart message is sent down the stream with just the changes.
I am using C# and am interested in using HttpClient or equivalent to open the stream, read all the information currently in the stream and then monitor this stream for when there is new information so that I can update the status information of the device accordingly in the application.
In essence the code I have looks something like this
using (var handler = new HttpClientHandler { Credentials = new NetworkCredential(username, password) })
{
using (var client = new HttpClient(handler))
{
var task = client.GetStreamAsync(uri);
task.Wait();
var stream = task.Result;
while(true)
{
byte[] bytes = ReadBytesFromStream(stream);
DoSomethingWithBytes(bytes);
}
}
The code in real life runs in a thread however and would need terminated correctly when told to.
The issue I am having is that when there is nothing in the stream the Read call on stream.ReadByte() blocks. If I put a ReadTimeout on the stream then when the Read call fails(i.e. when no new information is ready) then the CanRead property is set to false and I have to restart the process however in doing so recieve all the original status information again instead of only the elements that have changed.
Is there something that can be done to keep the stream alive until I tell it to terminate while being able to unblock on the read if no information is available? The reason I need to do this is since the application is multithreaded I need to terminate this code safely and the read is stopping the application from closing down.
Instead of using HttpClient I used an HttpWebRequest and set KeepAlive to true and AllowReadStreamBuffering properties to true. This keeps the stream alive and allows you to read bytes as of when they become available.
By keeping a reference to the network stream returned from GetResponseStream we can call Dispose on the NetworkStream which interrupts any reads that are currently taking place otherwise the read can block for as long as it needs to i.e. until it recieves data which solves the thread lifetime issues.
The right way to deal with the "I/O operation blocks my thread" problem is to use asynchronous I/O. .NET networking components offer a number of options here, but in your case you seem to be reading from a stream and even using (incorrectly) the GetStreamAsync() method, so the code can be cleaned up to handle both correctly and cleanly.
For example:
async Task ExecuteUriAsync(string username, string password, Uri uri)
{
using (var handler = new HttpClientHandler { Credentials = new NetworkCredential(username, password) })
{
using (var client = new HttpClient(handler))
{
Stream stream = await client.GetStreamAsync(uri);
byte[] buffer = new byte[10240];
while(true)
{
int byteCount = await stream.ReadAsync(buffer, 0, buffer.Length);
if (byteCount == 0)
{
// end-of-stream...must be done with the connection
return;
}
DoSomethingWithBytes(bytes, byteCount);
}
}
}
}
Your post is vague on what your previous ReadBytesFromStream() method did, and what DoSomethingWithBytes() does, but presumably you can figure out how to integrate that logic in the above.
If we create a HttpWebRequest and get the ResponseStream from its response, then whether the data will get downloaded completely at once or, when we call the ReadBytes of the stream , then only the data will get download from the network and then reads the content?
Code sample which i want to refer is mentioned below:
var webRequest = HttpWebRequest.Create('url of a big file approx 700MB') as HttpWebRequest;
var webResponse = webRequest.GetResponse();
using (BinaryReader ns = new BinaryReader(webResponse.GetResponseStream()))
{
Thread.Sleep(60000); //Sleep for 60seconds, hope 700MB file get downloaded in 60 seconds
//At this point whether the response is totally downloaded or will not get downloaded at all
var buffer = ns.ReadBytes(bufferToRead);
//Or, in the above statement ReadBytes function is responsible for downloading the content from the internet.
}
GetResponseStream opens and returns a Stream object. The stream object is sourced from the underlying Socket. This Socket is sent data by the network adapter asynchronously. The data just arrives and is buffered. GetResponseStream will block execution until the first data arrives.
ReadByte pulls the data up from the socket layer to c#. This method will block execution until there is a byte avaliable.
Closing the stream prematurely will end the asynchronous transfer (closes the Socket, the sender will be notified of this as their connection will fail) and discard (flush) any buffered data that you have not used yet.
var webRequest = HttpWebRequest.Create('url of a big file approx 700MB') as HttpWebRequest;
Okay, we're set up ready to go. It's a bit different if you PUT or POST a stream of your own, but the differences are analogous.
var webResponse = webRequest.GetResponse();
When GetResponse() returns, it will at the very least have read all of the HTTP headers. It may well have read the headers of a redirect, and done another request to the URI it was redirected to. It's also possible that it's actually hitting a cache (either directly or because the webserver setnt 304 Not Modified) but by default the details of that are hidden from you.
There will likely be some more bytes in the socket's buffer.
using (BinaryReader ns = new BinaryReader(webResponse.GetResponseStream()))
{
At this point, we've got a stream representing the network stream.
Let's remove the Thread.Sleep() it does nothing except add a risk of the connection timing out. Even assuming it doesn't timeout while waiting, the connection will have "backed off" from sending bytes since you weren't reading them, so the effect will be to slow things even more than you did by adding a deliberate slow-down.
var buffer = ns.ReadBytes(bufferToRead);
At this point, either bufferToRead bytes have been read to create a byte[] or else fewer than bufferToRead because the total size of the stream was less than that, in which case buffer contains the entire stream. This will take as long as it takes.
}
At this point, because a successful HTTP GET was performed, the underlying web-access layer may cache the response (probably not if it's very large - the default assumption is that very large requests don't get repeated a lot and don't benefit from caching).
Error conditions will raise exceptions if they occur, and in that case no caching will ever be done (there is no point caching a buggy response).
There is no need to sleep, or otherwise "wait" on it.
It's worth considering the following variant that works at just a slightly lower level by manipulating the stream directly rather than through a reader:
using(var stm = webResponse.GetResponseStream())
{
We're going to work on the stream directly;
byte[] buffer = new byte[4096];
do
{
int read = stm.Read(buffer, 0, 4096);
This will return up to 4096 bytes. It may read less, because it has a chunk of bytes already available and it returns that many immediately. It will only return 0 bytes if it is at the end of the stream, so this gives us a balance between waiting and not waiting - it promises to wait long enough to get at least one byte, but whether or not it waits until it gets all 4096 bytes is up to the stream to choose whether it is more efficient to wait that long or return fewer bytes;
DoSomething(buffer, 0, read);
We work with the bytes we got.
} while(read != 0);
Read() only gives us zero bytes, if it's at the end of the stream.
}
And again, when the stream is disposed, the response may or may not be cached.
As you can see, even at the lowest level .NET gives us access to when using HttpWebResponse, there's no need to add code to wait on anything, as that is always done for us.
You can use asynchronous access to the stream to avoid waiting, but then the asynchronous mechanism still means you get the result when it's available.
To answer your question about when streaming starts, GetResponseStream() will start receiving data from the server. However, at some point the network buffers will become full and the server will stop sending data if you don't read off the buffers. For a detailed description of the tcp buffers, etc see here.
So your sleep of 60000 will not be helping you much as the network buffers along the way will fill up and data will stop arriving until you read it off. It is better to read it off and write it in chunks as you go.
More info on the workings of ResponseStream here.
If you are wondering about what buffer size to use, see here.