I'm trying to GetResponseStream from a GET request and convert this into byte[]. The Stream is non-seekable so I can't access stream.Length. response.ContentLength is unreliable.
private byte[] streamToBytes(Stream stream, int bufferSize = 4096)
{
byte[] buffer = new byte[bufferSize];
int read = 0;
int pos = 0;
List<byte> bytes = new List<byte>();
while (true) { // `bufferSize > read` does **not** mean stream end
while (bufferSize == (read = stream.Read(buffer, pos, bufferSize))) {
pos += read;
bytes.AddRange(buffer);
}
if (read > 0) {
byte[] _buffer = new byte[read];
Array.Copy(buffer, _buffer, read);
bytes.AddRange(_buffer);
} else break;
}
return bytes.ToArray();
}
An ArgumentOutOfRangeException gets thrown on the Read in the second iteration of the while loop. MSDN says
ArgumentOutOfRangeException offset or count is negative.
I know this can't be true because offset (pos) is 0 + read >= 0 and count (bufferSize) is 4096 so why am I getting exceptions thrown at me?
I'm trying to keep streamToBytes as generic as possible so I can use it in future async methods, too.
If how the request is made helps, here are the relevant bits
HttpWebRequest request = (HttpWebRequest)WebRequest.Create((new Uri("http://google.com")).ToString());
request.Method = "GET";
request.KeepAlive = true;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
byte[] responseBytes = streamToBytes(stream);
As a simpler alternative:
using(var memStream = new MemoryStream())
{
stream.CopyTo(memStream);
return memStream.ToArray();
}
Your code has two mistakes:
The offset on Read is an offset into the buffer, not the position in the stream (as #Ulugbek already noted). This means you don't need the pos variable anymore.
You cannot assume that bufferSize == read even before the stream reached its end. You need to check for > 0 instead.
So your reading loop becomes:
while ((read = stream.Read(buffer, 0, bufferSize)) > 0)
{
bytes.AddRange(buffer.Take(read));
}
You can now drop the special handling for the last block. Simplifying your code to:
private byte[] streamToBytes(Stream stream, int bufferSize = 4096)
{
byte[] buffer = new byte[bufferSize];
int read = 0;
List<byte> bytes = new List<byte>();
while ((read = stream.Read(buffer, 0, bufferSize)) > 0)
{
bytes.AddRange(buffer.Take(read));
}
return bytes.ToArray();
}
Using List<byte> instead of MemoryStream isn't a great idea either. AddRange need to iterate over the bytes individually, instead of using a low level copy operation. Replacing the list with a memory stream, the code becomes:
private byte[] streamToBytes(Stream stream, int bufferSize = 4096)
{
byte[] buffer = new byte[bufferSize];
int read = 0;
using(var bytes = new MemoryStream())
{
while ((read = stream.Read(buffer, 0, bufferSize)) > 0)
{
bytes.Write(buffer, 0, read);
}
return bytes.ToArray();
}
}
You could even split it into two parts, one does the copying into the memory stream, the other handles the creating of the memory stream and turning it into a byte array:
private static void CopyStream(Stream source, Stream destination, int bufferSize = 4096)
{
byte[] buffer = new byte[bufferSize];
int read = 0;
while ((read = source.Read(buffer, 0, bufferSize)) > 0)
{
destination.Write(buffer, 0, read);
}
}
private static byte[] StreamToBytes(Stream stream, int bufferSize = 4096)
{
using(var memStream = new MemoryStream())
{
CopyStream(stream, memStream);
return memStream.ToArray();
}
}
This is probably very similar to what Stream.CopyTo does internally.
Change
stream.Read(buffer, pos, bufferSize)
to
stream.Read(buffer, 0, bufferSize)
Related
So I have this file transfer function that I made which works fine with small files but does not work with big files (above 5 or 10mbs). Basically what I do is I split the file into chunks and add a 'header' which defines what type of data it is so that the client can understand. I am also using this 'header' approach because I also want to be able to upload files and do other stuff at the same time. The problem is that the client is receiving half of the bytes even tho the server is sending all the bytes so I am assuming the header gets corrupted and if (header == "s^") return false.
Here is my code
Server (Uploader):
int max_buffer = 1024 * 1024 * 2;
using (var fs = new FileStream(fo_path, FileMode.Open, FileAccess.Read))
{
byte[] header = Encoding.Unicode.GetBytes("s^fd");
byte[] buffer = new byte[max_buffer];
int bytesRead = 0;
long bytesToRead = fs.Length;
while (bytesToRead > 0)
{
int n = fs.Read(buffer, 0, max_buffer);
if (n == 0) break;
if (n != buffer.Length)
Array.Resize(ref buffer, n);
byte[] data = new byte[header.Length + n];
Buffer.BlockCopy(header, 0, data, 0, header.Length);
Buffer.BlockCopy(buffer, 0, data, header.Length, n);
_client.Send(data);
bytesRead += n;
bytesToRead -= n;
}
}
Client (Downloader):
private static MemoryStream fileStream = new MemoryStream(); //global scope
byte[] buffer = new byte[2048];
int recieved = _clientSocket.Receive(buffer, SocketFlags.None);
if (recieved == 0) return;
byte[] data = new byte[recieved];
Array.Copy(buffer, data, recieved);
string header = Encoding.Unicode.GetString(data, 0, 4);
if (header == "s^")
{
string header_cmd = Encoding.Unicode.GetString(data, 4, 4);
if (header_cmd == "fd")
{
byte[] fileBytes = new byte[data.Length - 32];
Buffer.BlockCopy(data, 32, fileBytes, 0, fileBytes.Length);
fileStream.Write(fileBytes, 0, fileBytes.Length);
fo_writeSize += fileBytes.Length;
if (fo_writeSize >= fo_size) //fo_size is the file size that the server sends before sending the actual file itself
{
var fs = new FileStream(fo_path, FileMode.Create, FileAccess.Write);
fileStream.WriteTo(fs);
fileStream.Close();
fs.Close();
}
}
}
Here is the method I like to use. I believe, there is nothing new with this code.
public static byte[] ReadFully(Stream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, just
// use 1K.
if (initialLength < 1)
{
initialLength = 1024;
}
byte[] buffer = new byte[initialLength];
int read = 0;
int chunk;
while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0)
{
read += chunk;
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
My goal is to read data sent from TCP Clients e.g. box{"id":1,"aid":1}
It is a command to interpret in my application in Jason-like text.
And this text is not necessarily at the same size each time.
Next time there can be run{"id":1,"aid":1,"opt":1}.
The method called by this line;
var serializedMessageBytes = ReadFully(_receiveMemoryStream, 1024);
Please click to see; Received data in receiveMemoryStream
Although we can see the data in the stream,
in the ReadFully method, "chunck" always return 0 and the method returns {byte[0]}.
Any help effort greatly appreciated.
Looking at your stream in the Watch window, the Position of the stream (19) is at the end of the data, hence there is nothing left to read. This is possibly because you have just written data to the stream and have not subsequently reset the position.
Add a stream.Position = 0; or stream.Seek(0, System.IO.SeekOrigin.Begin); statement at the start of the function if you are happy to always read from the start of the stream, or check the code that populates the stream. Note though that some stream implementations do not support seeking.
I get the following exception when I attempt to download an item from a specific location
public void Mydownload(string uri, string destinationPath, CancellationToken cancellationToken)
{
try
{
WebRequest request = WebRequest.Create(uri);
WebResponse response = request.GetResponse();
using (FileStream file = File.Create(destinationPath))
{
long sz = response.ContentLength;
int bytesRead = 0;
int chunkBytes;
int progress = 0;
byte[] buffer = new byte[4096];
Stream stream = response.GetResponseStream();
while ((chunkBytes = stream.Read(buffer, bytesRead, buffer.Length)) > 0)
{
file.Write(buffer, 0, chunkBytes);
bytesRead += chunkBytes;
int currentProgress = (int)(((double)sz) / bytesRead);
}
}
}
catch(Exception c)
{
Console.Write(c.ToString()); //Specified argument was out of the range of valid values.Parameter name: size
}
}
I noticed that the program crashes during the second iteration of the while loop on the statement
while ((chunkBytes = stream.Read(buffer, bytesRead, buffer.Length)) > 0)
I am not sure what I might be doing wrong any suggestions ?
The second argument to stream.Read is the offset of the output buffer to begin writing to. You are doing:
stream.Read(buffer, bytesRead, buffer.Length)
yet bytesRead is incremented as the data is written:
file.Write(buffer, 0, chunkBytes);
bytesRead += chunkBytes;
On the first iteration bytesRead is 0 so the start index is in range for the target buffer, but on the second it will probably have been incremented by the length of the buffer after the first read. At this point it will be an out of range index, hence the exception.
You always want to read from the start of the buffer i.e.
stream.Read(buffer, 0, buffer.Length);
What I'm trying to do is to received a large number of bytes (about 5MB data) sent from the client side
Below is the code where data(byte[]) is received
byte[] receivedBytesRaw = new byte[4000];
//first, initialize network stream
NetworkStream stream = client.GetStream();
//The bytesNeeded is the size of bytes which is a protocol sent by the client side indicating the size of byte which will be sent
int bytesNeeded = 4000;
int bytesReceived = 0;
do
{
int bytesRead = stream.Read(receivedBytesRaw, bytesReceived, bytesNeeded - bytesReceived);
networkValidation.addBytesToList(receivedBytesRaw, ref receivedBytes);
bytesReceived += bytesRead;
} while (bytesReceived < bytesNeeded);
But now I'm stuck on a problem:
Everytime when data arrives, the do while loop loops for the first time, and the return value (i) is 26, then it loops again, this time, when it goes to " i = stream.Read(receivedBytesRaw, 0, receivedBytesRaw.Length);", the program seems waiting for the client side to send data and have no response, also, when I check "receivedBytesRaw", the data was incomplete, only the first 13 bytes was received, the remaining space in the byte array remains null, and the stream.DataAvailable is false
Why the server side received incomplete data?
Note: when I try to send small data (a string), it's ok
=====================================================================
Edited
Below is the client side code which sends data:
private int sendData(byte[] dataSend, string IP, int portNumber)
{
TcpClient clientSide = new TcpClient();
int result = -1;
try
{
clientSide.Connect(IP, portNumber);
}
catch (Exception ex)
{
return 2;
}
NetworkStream netStream = clientSide.GetStream();
if (netStream.CanWrite)
{
byte[] replyMsg = new byte[1024];
netStream.Write(dataSend, 0, dataSend.Length);
netStream.Flush();
result = 0;
}
else
{
result = 1;
}
return result;
}
Because it's a stream, and can be partial received. Are you sure you are always receiving packages with te size of 2048 bytes?
int i = 0;
int bytesNeeded = 200;
int bytesReceived = 0;
do
{
//read byte from client
int bytesRead = stream.Read(receivedBytesRaw, bytesReceived, bytesNeeded-bytesReceived);
bytesReceived += bytesRead;
// merge byte array to another byte array
} while (bytesReceived < bytesNeeded);
I think you need a frame protocol, try create a protocol like, writing the size of the data that follows.
example: (psuedo)
void SendData(byte[] data)
{
// get the 4 bytes of a int value.
byte[] dataLength = BitConverter.GetBytes(data.Lenght);
// write the length to the stream.
stream.Write(dataLength, 0, dataLength.Length);
// write the data bytes.
stream.Write(data, 0, data.Length);
}
void Receive()
{
// read 4 bytes from the stream.
ReadBuffer(buffer, 4);
// convert those 4 bytes to an int.
int dataLength = BitConverter.ToInt32(buffer, 0);
// read bytes with dataLength as count.
ReadBuffer(buffer, dataLength);
}
// read until the right amount of bytes are read.
void ReadBuffer(byte[] buffer, int length)
{
int i = 0;
int bytesNeeded = length;
int bytesReceived = 0;
do
{
//read byte from client
int bytesRead = stream.Read(buffer, bytesReceived, bytesNeeded-bytesReceived);
bytesReceived += bytesRead;
// merge byte array to another byte array
} while (bytesReceived < bytesNeeded); // <- you should do this async.
}
This is just an example..
Another solution you could try is using async reads.
I made a class that reads until all bytes are read. If it isn't a problem that the complete file is read, you could try this:
Example:
This example show that you can read a simple protocol. ReadPacket handles a length + data message. So the sender will first send an int value containing the length of data that follows.
The StartReading method reads a filename and the filedata. It will store up to 10mb max filesize. But this isn't originally designed for receiving files.
const int MaxFileSize = 10 * 1024 * 1024;
private void Example()
{
Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
socket.Connect("localhost", 12345);
StartReading(socket);
}
private void StartReading(Socket socket)
{
ReadPacket(socket, (filenameData) =>
{
if (filenameData.Count == 0)
{
// disconnected
return;
}
// parse the filename
string filename = Encoding.UTF8.GetString(filenameData.Array, filenameData.Offset, filenameData.Count);
Trace.WriteLine("Receiving file :" + filename);
ReadPacket(socket, (fileData) =>
{
if (fileData.Count == 0)
{
// disconnected
return;
}
Trace.WriteLine("Writing file :" + filename);
// write to the file
using (FileStream stream = new FileStream(filename, FileMode.Create, FileAccess.Write))
stream.Write(fileData.Array, fileData.Offset, fileData.Count);
// start waiting for another packet.
StartReading(socket);
});
});
}
private void ReadPacket(Socket socket, Action<ArraySegment<byte>> endRead)
{
// read header. (length of data) (4 bytes)
EasySocketReader.ReadFromSocket(socket, 4, (headerBufferSegment) =>
{
// if the ReadFromSocket returns 0, the socket is closed.
if (headerBufferSegment.Count == 0)
{
// disconnected;
endRead(new ArraySegment<byte>());
return;
}
// Get the length of the data that follows
int length = BitConverter.ToInt32(headerBufferSegment.Array, headerBufferSegment.Offset);
// Check the length
if (length > MaxFileSize)
{
// disconnect
endRead(new ArraySegment<byte>());
return;
}
// Read bytes specified in length.
EasySocketReader.ReadFromSocket(socket, length, (dataBufferSegment) =>
{
// if the ReadFromSocket returns 0, the socket is closed.
if (dataBufferSegment.Count == 0)
{
endRead(new ArraySegment<byte>());
return;
}
endRead(dataBufferSegment);
});
});
}
The EasySocketReader class can be found on my blog: http://csharp.vanlangen.biz/network-programming/async-sockets/asyncsocketreader/
The original EasyPacketReader can be found here: http://csharp.vanlangen.biz/network-programming/async-sockets/easypacketreader/
For the sending part, you could use something like this:
private void SendFile(Socket socket, string filename)
{
byte[] filenameData = Encoding.UTF8.GetBytes(filename);
socket.Send(BitConverter.GetBytes(filenameData.Length));
socket.Send(filenameData);
int fileSize;
byte[] fileData;
using (FileStream stream = new FileStream(filename, FileMode.Open, FileAccess.Read))
{
fileSize = (int)stream.Length;
if (fileSize > MaxFileSize)
throw new ArgumentOutOfRangeException("File too big");
fileData = new byte[fileSize];
stream.Read(fileData, 0, fileSize);
}
socket.Send(BitConverter.GetBytes(fileSize));
socket.Send(fileData);
}
Is there any way to use Stream.CopyTo to copy only certain number of bytes to destination stream? what is the best workaround?
Edit:
My workaround (some code omitted):
internal sealed class Substream : Stream
{
private readonly Stream stream;
private readonly long origin;
private readonly long length;
private long position;
public Substream(Stream stream, long length)
{
this.stream = stream;
this.origin = stream.Position;
this.position = stream.Position;
this.length = length;
}
public override int Read(byte[] buffer, int offset, int count)
{
var n = Math.Max(Math.Min(count, origin + length - position), 0);
int bytesRead = stream.Read(buffer, offset, (int) n);
position += bytesRead;
return bytesRead;
}
}
then to copy n bytes:
var substream = new Substream(stream, n);
substream.CopyTo(stm);
The implementation of copying streams is not overly complicated. If you want to adapt it to copy only a certain number of bytes then it shouldn't be too difficult to tweak the existing method, something like this
public static void CopyStream(Stream input, Stream output, int bytes)
{
byte[] buffer = new byte[32768];
int read;
while (bytes > 0 &&
(read = input.Read(buffer, 0, Math.Min(buffer.Length, bytes))) > 0)
{
output.Write(buffer, 0, read);
bytes -= read;
}
}
The check for bytes > 0 probably isn't strictly necessary, but can't do any harm.
What's wrong with just copying the bytes you need using a buffer?
long CopyBytes(long bytesRequired, Stream inStream, Stream outStream)
{
long readSoFar = 0L;
var buffer = new byte[64*1024];
do
{
var toRead = Math.Min(bytesRequired - readSoFar, buffer.Length);
var readNow = inStream.Read(buffer, 0, (int)toRead);
if (readNow == 0)
break; // End of stream
outStream.Write(buffer, 0, readNow);
readSoFar += readNow;
} while (readSoFar < bytesRequired);
return readSoFar;
}