I'm trying to stream a file over a socket, and write it to a file as I receive it (so that it can be opened in other applications). It mostly works except for near the end where (I'm guessing) it blocks because it's trying to read an entire 8 KB chunk when there isn't that much left to receive. I'm at a loss trying to figure out how to fix this. I know the total amount of bytes I'm supposed to receive, if that helps. Thanks!
using (var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)) {
socket.Connect(e.IPAddress, e.Port);
using (var fs = new FileStream(e.Filename, FileMode.OpenOrCreate, FileAccess.Write, FileShare.Read)) {
byte[] buffer = new byte[8192];
int read;
while ((read = socket.Receive(buffer)) > 0) {
fs.Write(buffer, 0, read);
}
}
}
From the MSDN...
If no data is available for reading, the Receive method will block until data is available, unless a time-out value was set by using Socket.ReceiveTimeout. If the time-out value was exceeded, the Receive call will throw a SocketException. If you are in non-blocking mode, and there is no data available in the in the protocol stack buffer, the Receive method will complete immediately and throw a SocketException. You can use the Available property to determine if data is available for reading. When Available is non-zero, retry the receive operation.
The Socket.Receive() method blocks because the connection is still open on the other end, even if there isn't any more data available. There are a couple ways to get around this, one which requres you to know the size of the file beforehand and stopping once you've read that many bytes Another would be using a non-blocking socket (by setting the Socket.Blocking property to false) and getting the data available from the Socket.Available property. The latter requires a little more finesse in writing your code since read operations won't halt your code anymore.
If you know the size of the data beforehand, you can compare that to how much you've read so far and stop once you've reached that size:
//If you know the size beforehand, set it as an int and compare
// it to what you have so far
int totalBytes = 9001;
using (var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))
{
socket.Connect(e.IPAddress, e.Port);
using (var fs = new FileStream(e.Filename, FileMode.OpenOrCreate, FileAccess.Write, FileShare.Read))
{
byte[] buffer = new byte[8192];
int read;
int bytesSoFar = 0; //Use this to keep track of how many bytes have been read
do
{
read = socket.Receive(buffer);
fs.Write(buffer, 0, read);
bytesSoFar += read;
} while (bytesSoFar < totalBytes);
}
}
Keep in mind that this second example isn't very robust and will not work when trying to send multiple files using the same socket. But for just a simple 1 file transfer, this should work fine.
Related
I have an app that uses a TCP socket to exchange byte arrays which in most cases contain JSON string data. What I'm experiencing is that, for larger messages and less than ideal network conditions, use of NetworkStream.DataAvailable does NOT seem to be a reliable way to detect an end of message. It seems that in some cases DataAvailable is set to false even when only part of the message has been transmitted by peer (which is using TcpClient.GetStream().Write(data, 0, data.Length). This results in incomplete data being passed back to the app, which in the case of a JSON message, means deserialization fails.
I've tried two implementations which exhibit the same issue:
Implementation 1:
byte[] Data;
byte[] buffer = new byte[2048];
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = ClientStream.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
BytesRead += read;
if (!ClientStream.DataAvailable) break;
}
Data = ms.ToArray();
}
Implementation 2:
byte[] Data;
using (MemoryStream ms = new MemoryStream())
{
byte[] buffer = new byte[2048];
do
{
int read = ClientStream.Read(buffer, 0, buffer.Length);
if (read > 0)
{
ms.Write(buffer, 0, read);
BytesRead += read;
}
}
while (ClientStream.DataAvailable);
Data = ms.ToArray();
}
It seems one solution that works really well but is completely sub-optimal is to add a Thread.Sleep in case NetworkStream.DataAvailable is false (while inside the loop) to allow data to be delivered. However this severely limits overall IOPS which I would like to avoid, i.e.
Implementation 3 (works, but suboptimal)
byte[] Data;
using (MemoryStream ms = new MemoryStream())
{
byte[] buffer = new byte[2048];
do
{
int read = ClientStream.Read(buffer, 0, buffer.Length);
if (read > 0)
{
ms.Write(buffer, 0, read);
BytesRead += read;
}
if (!ClientStream.DataAvailable) System.Threading.Thread.Sleep(250);
}
while (ClientStream.DataAvailable);
Data = ms.ToArray();
}
I'd really like to find a way to remain in the loop until all of the data is delivered. As I mentioned, I'm doing a simple write operation on the client from zero to data length, so I'm not thinking there is an issue there.
Has anyone had any experience like this before and a recommendation?
It seems .DataAvailable is indeed reliable and that, since the data arrives over the network potentially at a rate slower than data is read from the stream, .DataAvailable can flip-flop between the start and end of what my application thinks is a message.
I'm answering and closing this as I believe the only solutions to this are:
1) add an over-arching receive timeout value, and perform a thread.sleep in the read loop, and expiring the operation once the receive timeout is reached
2) implement some mechanism of indicating the data payload size - either explicitly or by creating a system of metadata headers - to indicate how much data should be read, and exiting after that much data has been read or the operation has timed out
These two are the best I could come up with and seem to be validated by the likes of other TCP-based protocols like HTTP and generally any other RPC out there.
Hopefully this saves someone some time.
I am getting data from client and saving it to the local drive on local host .I have checked it for a file of 221MB but a test for file of 1Gb gives the following exception:
An unhandled exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll
Following is the code at server side where exception stems out.
UPDATED
Server:
public void Thread()
{
TcpListener tcpListener = new TcpListener(ipaddr, port);
tcpListener.Start();
MessageBox.Show("Listening on port" + port);
TcpClient client=new TcpClient();
int bufferSize = 1024;
NetworkStream netStream;
int bytesRead = 0;
int allBytesRead = 0;
// Start listening
tcpListener.Start();
// Accept client
client = tcpListener.AcceptTcpClient();
netStream = client.GetStream();
// Read length of incoming data to reserver buffer for it
byte[] length = new byte[4];
bytesRead = netStream.Read(length, 0, 4);
int dataLength = BitConverter.ToInt32(length,0);
// Read the data
int bytesLeft = dataLength;
byte[] data = new byte[dataLength];
while (bytesLeft > 0)
{
int nextPacketSize = (bytesLeft > bufferSize) ? bufferSize : bytesLeft;
bytesRead = netStream.Read(data, allBytesRead, nextPacketSize);
allBytesRead += bytesRead;
bytesLeft -= bytesRead;
}
// Save to desktop
File.WriteAllBytes(#"D:\LALA\Miscellaneous\" + shortFileName, data);
// Clean up
netStream.Close();
client.Close();
}
I am getting the file size first from client side followed by data.
1).Should i increase the buffer size or any other technique ?
2). File.WriteAllBytes() and File.ReadAllBytes() seems blocking and freezes the PC.Is there any async method for it to help provide the progress of file recieved at server side.
You don't need to read the whole thing to memory before writing it to disc. Just copy straight from the network stream to a FileStream:
byte[] length = new byte[4];
// TODO: Validate that bytesRead is 4 after this... it's unlikely but *possible*
// that you might not read the whole length in one go.
bytesRead = netStream.Read(length, 0, 4);
int bytesLeft = BitConverter.ToInt32(length,0);
using (var output = File.Create(#"D:\Javed\Miscellaneous\" + shortFileName))
{
netStream.CopyTo(output, bytesLeft);
}
Note that instead of calling netStream.Close() explicitly, you should use a using statement:
using (Stream netStream = ...)
{
// Read from it
}
That way the stream will be closed even if an exception is thrown.
The CLR has a per-object limit a bit short of 2GB. However that's the theory, in practice how much memory you can allocate depends on how much memory the framework allows you to allocate. I wouldn't expect it to allow you to allocate 1 GB data table. You should allocate smaller table, and write the data in chunks into disk file.
The "out of memory" exception happens because you are trying to place the entire file into memory before dumping it on disk. This is suboptimal, because you don't need the entire file in memory in order to write into the file: you can read it block-by-block in reasonably-sized increments, and write it out as you go.
Starting with .NET 4.0 you can use Stream.CopyTo method to accomplish this in a few lines of code:
// Read and ignore the initial four bytes of length from the stream
byte[] ignore = new byte[4];
int bytesRead = 0;
do {
// This should complete in a single call, but the API requires you
// to do it in a loop.
bytesRead += netStream.Read(ignore, bytesRead, 4-bytesRead);
} while (bytesRead != 4);
// Copy the rest of the stream to a file
using (var fs = new FileStream(#"D:\Javed\Miscellaneous\" + shortFileName, FileMode.Create)) {
netStream.CopyTo(fs);
}
netStream.Close();
Starting with .NET 4.5 you can use CopyToAsync, too, which would give you a way to do reading and writing asynchronously.
Note the code that drops the initial four bytes from the stream. This is done to avoid writing the length of the stream along with the "payload" bytes. If you have control over the network protocol, you could change the sending side to stop prefixing the stream with its length, and remove the code that reads and ignores it on the receiving side.
i'm working with C#, i am connected to a server using one Async TCP socket, the purpose is that the server sends me 5 image files every second.
I took one tutorial about how to make the socket connection, so i think the problem is on the ReadCallBack function. Some jpgs arrive fine but other in a format that windows can't open and i can't understand why.
EDIT: After read different posts and look for more information, i changed completely the way to make that and finally is working fine, if somebody is in the same case, here is how i did it:
StateObject state = (StateObject)ar.AsyncState;
Socket handler = state.workSocket;
int cont = 0;
//i'm not sure why i was receiving a "\" character cathing file name, but this will control that
while (state.buffer[cont] != 0x0a && cont < StateObject.BufferSize)
cont++;
string inc_file_name = System.Text.Encoding.ASCII.GetString(state.buffer, 0, cont);
FileStream Fs = new FileStream(inc_file_name, FileMode.OpenOrCreate, FileAccess.Write, FileShare.ReadWrite);
int read_cont;
byte [] read_buffer = new byte [1024];
//till end of incoming file
while ((read_cont = handler.Receive(read_buffer)) > 0)
Fs.Write(read_buffer, 0, read_cont);
//save file
Fs.Close();
I hope it helps you.
Thanks for the comments and for your help.
I am developing a TCP file transfer client-server program. At the moment I am able to send text files and other file formats perfectly fine, such as .zip with all contents intact on the server end. However, when I transfer a .gif the end result is a gif with same size as the original but with only part of the image showing as if most of the bytes were lost or not written correctly on the server end.
The client sends a 1KB header packet with the name and size of the file to the server. The server then responds with OK if ready and then creates a fileBuffer as large as the file to be sent is.
Here is some code to demonstrate my problem:
// Serverside method snippet dealing with data being sent
while (true)
{
// Spin the data in
if (streams[0].DataAvailable)
{
streams[0].Read(fileBuffer, 0, fileBuffer.Length);
break;
}
}
// Finished receiving file, write from buffer to created file
FileStream fs = File.Open(LOCAL_FOLDER + fileName, FileMode.CreateNew, FileAccess.Write);
fs.Write(fileBuffer, 0, fileBuffer.Length);
fs.Close();
Print("File successfully received.");
// Clientside method snippet dealing with a file send
while(true)
{
con.Read(ackBuffer, 0, ackBuffer.Length);
// Wait for OK response to start sending
if (Encoding.ASCII.GetString(ackBuffer) == "OK")
{
// Convert file to bytes
FileStream fs = new FileStream(inPath, FileMode.Open, FileAccess.Read);
fileBuffer = new byte[fs.Length];
fs.Read(fileBuffer, 0, (int)fs.Length);
fs.Close();
con.Write(fileBuffer, 0, fileBuffer.Length);
con.Flush();
break;
}
}
I've tried a binary writer instead of just using the filestream with the same result.
Am I incorrect in believing successful file transfer to be as simple as conversion to bytes, transportation and then conversion back to filename/type?
All help/advice much appreciated.
Its not about your image .. It's about your code.
if your image bytes were lost or not written correctly that's mean your file transfer code is wrong and even the .zip file or any other file would be received .. It's gonna be correpted.
It's a huge mistake to set the byte buffer length to the file size. imagine that you're going to send a large a file about 1GB .. then it's gonna take 1GB of RAM .. for an Idle transfering you should loop over the file to send.
This's a way to send/receive files nicely with no size limitation.
Send File
using (FileStream fs = new FileStream(srcPath, FileMode.Open, FileAccess.Read))
{
long fileSize = fs.Length;
long sum = 0; //sum here is the total of sent bytes.
int count = 0;
data = new byte[1024]; //8Kb buffer .. you might use a smaller size also.
while (sum < fileSize)
{
count = fs.Read(data, 0, data.Length);
network.Write(data, 0, count);
sum += count;
}
network.Flush();
}
Receive File
long fileSize = // your file size that you are going to receive it.
using (FileStream fs = new FileStream(destPath, FileMode.Create, FileAccess.Write))
{
int count = 0;
long sum = 0; //sum here is the total of received bytes.
data = new byte[1024 * 8]; //8Kb buffer .. you might use a smaller size also.
while (sum < fileSize)
{
if (network.DataAvailable)
{
{
count = network.Read(data, 0, data.Length);
fs.Write(data, 0, count);
sum += count;
}
}
}
}
happy coding :)
When you write over TCP, the data can arrive in a number of packets. I think your early tests happened to fit into one packet, but this gif file is arriving in 2 or more. So when you call Read, you'll only get what's arrived so far - you'll need to check repeatedly until you've got as many bytes as the header told you to expect.
I found Beej's guide to network programming a big help when doing some work with TCP.
As others have pointed out, the data doesn't necessarily all arrive at once, and your code is overwriting the beginning of the buffer each time through the loop. The more robust way to write your reading loop is to read as many bytes as are available and increment a counter to keep track of how many bytes have been read so far so that you know where to put them in the buffer. Something like this works well:
int totalBytesRead = 0;
int bytesRead;
do
{
bytesRead = streams[0].Read(fileBuffer, totalBytesRead, fileBuffer.Length - totalBytesRead);
totalBytesRead += bytesRead;
} while (bytesRead != 0);
Stream.Read will return 0 when there's no data left to read.
Doing things this way will perform better than reading a byte at a time. It also gives you a way to ensure that you read the proper number of bytes. If totalBytesRead is not equal to the number of bytes you expected when the loop is finished, then something bad happened.
Thanks for your input Tvanfosson. I tinkered around with my code and managed to get it working. The synchronicity between my client and server was off. I took your advice though and replaced read with reading a byte one at a time.
Hey, I'm having an issue seperating packets using a custom binary protocol.
Currently the server side code looks like this.
public void HandleConnection(object state)
{
TcpClient client = threadListener.AcceptTcpClient();
NetworkStream stream = client.GetStream();
byte[] data = new byte[4096];
while (true)
{
int recvCount = stream.Read(data, 0, data.Length);
if (recvCount == 0) break;
LogManager.Debug(Utility.ToHexDump(data, 0, recvCount));
//processPacket(new MemoryStream(data, 0, recvCount));
}
LogManager.Debug("Client disconnected");
client.Close();
Dispose();
}
I've been watching the hex dumps of the packets, and sometimes the entire packet comes in one shot, let's say all 20 bytes. Other times it comes in fragmented, how do I need to buffer this data to be able to pass it to my processPacket() method correctly. I'm attempting to use a single byte opcode header only, should I add something like a (ushort)contentLength to the header aswell? I'm trying to make the protocol as lightweight as possible, and this system won't be sending very large packets(< 128 bytes).
The client side code I'm testing with is, as follows.
public void auth(string user, string password)
{
using (TcpClient client = new TcpClient())
{
client.Connect(IPAddress.Parse("127.0.0.1"), 9032);
NetworkStream networkStream = client.GetStream();
using (BinaryWriter writer = new BinaryWriter(networkStream))
{
writer.Write((byte)0); //opcode
writer.Write(user.ToUpper());
writer.Write(password.ToUpper());
writer.Write(SanitizationMgr.Verify()); //App hash
writer.Write(Program.Seed);
}
}
}
I'm not sure if that could be what's messing it up, and binary protocol doesn't seem to have much info on the web, especially where C# is involved. Any comment's would be helpful. =)
Solved with this, not sure if it's correct, but it seems to give my handlers just what they need.
public void HandleConnection(object state)
{
TcpClient client = threadListener.AcceptTcpClient();
NetworkStream stream = client.GetStream();
byte[] data = new byte[1024];
uint contentLength = 0;
var packet = new MemoryStream();
while (true)
{
int recvCount = stream.Read(data, 0, data.Length);
if (recvCount == 0) break;
if (contentLength == 0 && recvCount < headerSize)
{
LogManager.Error("Got incomplete header!");
Dispose();
}
if(contentLength == 0) //Get the payload length
contentLength = BitConverter.ToUInt16(data, 1);
packet.Write(data, (int) packet.Position, recvCount); //Buffer the data we got into our MemStream
if (packet.Length < contentLength + headerSize) //if it's not enough, continue trying to read
continue;
//We have a full packet, pass it on
//LogManager.Debug(Utility.ToHexDump(packet));
processPacket(packet);
//reset for next packet
contentLength = 0;
packet = new MemoryStream();
}
LogManager.Debug("Client disconnected");
client.Close();
Dispose();
}
You should just treat it as a stream. Don't rely on any particular chunking behaviour.
Is the amount of data you need always the same? If not, you should change the protocol (if you can) to prefix the logical "chunk" of data with the length in bytes.
In this case you're using BinaryWriter on one side, so attaching a BinaryReader to the NetworkStream returned by TcpClient.GetStream() would seem like the easiest approach. If you really want to capture all the data for a chunk at a time though, you should go back to my idea of prefixing the data with its length. Then just loop round until you've got all the data.
(Make sure you've got enough data to read the length though! If your length prefix is 4 bytes, you don't want to read 2 bytes and miss the next 2...)