Develop Reference

C# Filestream Read - Recycle array?

I am working with filestream read: https://msdn.microsoft.com/en-us/library/system.io.filestream.read%28v=vs.110%29.aspx
What I'm trying to do is read a large file in a loop a certain number of bytes at a time; not the whole file at once. The code example shows this for reading:
int n = fsSource.Read(bytes, numBytesRead, numBytesToRead);
The definition of "bytes" is: "When this method returns, contains the specified byte array with the values between offset and (offset + count - 1) replaced by the bytes read from the current source."
I want to only read in 1 mb at a time so I do this:
using (FileStream fsInputFile = new FileStream(strInputFileName, FileMode.Open, FileAccess.Read)) {
int intBytesToRead = 1024;
int intTotalBytesRead = 0;
int intInputFileByteLength = 0;
byte[] btInputBlock = new byte[intBytesToRead];
byte[] btOutputBlock = new byte[intBytesToRead];
intInputFileByteLength = (int)fsInputFile.Length;
while (intInputFileByteLength - 1 >= intTotalBytesRead)
{
if (intInputFileByteLength - intTotalBytesRead < intBytesToRead)
{
intBytesToRead = intInputFileByteLength - intTotalBytesRead;
}
// *** Problem is here ***
int n = fsInputFile.Read(btInputBlock, intTotalBytesRead, intBytesToRead);
intTotalBytesRead += n;
fsOutputFile.Write(btInputBlock, intTotalBytesRead - n, n);
}
fsOutputFile.Close(); }
Where the problem area is stated, btInputBlock works on the first cycle because it reads in 1024 bytes. But then on the second loop, it doesn't recycle this byte array. It instead tries to append the new 1024 bytes into btInputBlock. As far as I can tell, you can only specify the offset and length of the file you want to read and not the offset and length of btInputBlock. Is there a way to "re-use" the array that is being dumped into by Filestream.Read or should I find another solution?
Thanks.
P.S. The exception on the read is: "Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection."

Your code can be simplified somewhat
int num;
byte[] buffer = new byte[1024];
while ((num = fsInputFile.Read(buffer, 0, buffer.Length)) != 0)
{
//Do your work here
fsOutputFile.Write(buffer, 0, num);
}
Note that Read takes in the Array to fill, the offset (which is the offset of the array where the bytes should be placed, and the (max) number of bytes to read.

That's because you're incrementing intTotalBytesRead, which is an offset for the array, not for the filestream. In your case it should always be zero, which will overwrite previous byte data in the array, rather than append it at the end, using intTotalBytesRead.
int n = fsInputFile.Read(btInputBlock, intTotalBytesRead, intBytesToRead); //currently
int n = fsInputFile.Read(btInputBlock, 0, intBytesToRead); //should be
Filestream doesn't need an offset, every Read picks up where the last one left off.
See https://msdn.microsoft.com/en-us/library/system.io.filestream.read(v=vs.110).aspx
for details

Your Read call should be Read(btInputBlock, 0, intBytesToRead). The 2nd parameter is the offset into the array you want to start writing the bytes to. Similarly for Write you want Write(btInputBlock, 0, n) as the 2nd parameter is the offset in the array to start writing bytes from. Also you don't need to call Close as the using will clean up the FileStream for you.
using (FileStream fsInputFile = new FileStream(strInputFileName, FileMode.Open, FileAccess.Read))
{
int intBytesToRead = 1024;
byte[] btInputBlock = new byte[intBytesToRead];
while (fsInputFile.Postion < fsInputFile.Length)
{
int n = fsInputFile.Read(btInputBlock, 0, intBytesToRead);
intTotalBytesRead += n;
fsOutputFile.Write(btInputBlock, 0, n);
}
}

Can't read FileStream into byte[] correctly

I have some C# code to call as TF(true,"C:\input.txt","C:\noexistsyet.file"), but when I run it, it breaks on FileStream.Read() for reading the last chunk of the file into the buffer, getting an index-out-of-bounds ArgumentException.
To me, the code seems logical with no overflow for trying to write to the buffer. I thought I had all that set up with rdlen and _chunk, but maybe I'm looking at it wrong. Any help?
My error: ArgumentException was unhandled: Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection.
public static bool TF(bool tf, string filepath, string output)
{
long _chunk = 16 * 1024; //buffer count
long total_size = 0
long rdlen = 0;
long wrlen = 0;
long full_chunks = 0;
long end_remain_buf_len = 0;
FileInfo fi = new FileInfo(filepath);
total_size = fi.Length;
full_chunks = total_size / _chunk;
end_remain_buf_len = total_size % _chunk;
fi = null;
FileStream fs = new FileStream(filepath, FileMode.Open);
FileStream fw = new FileStream(output, FileMode.Create);
for (long chunk_pass = 0; chunk_pass < full_chunks; chunk_pass++)
{
int chunk = (int)_chunk * ((tf) ? (1 / 3) : 3); //buffer count for xbuffer
byte[] buffer = new byte[_chunk];
byte[] xbuffer = new byte[(buffer.Length * ((tf) ? (1 / 3) : 3))];
//Read chunk of file into buffer
fs.Read(buffer, (int)rdlen, (int)_chunk); //ERROR occurs here
//xbuffer = do stuff to make it *3 longer or *(1/3) shorter;
//Write xbuffer into chunk of completed file
fw.Write(xbuffer, (int)wrlen, chunk);
//Keep track of location in file, for index/offset
rdlen += _chunk;
wrlen += chunk;
}
if (end_remain_buf_len > 0)
{
byte[] buffer = new byte[end_remain_buf_len];
byte[] xbuffer = new byte[(buffer.Length * ((tf) ? (1 / 3) : 3))];
fs.Read(buffer, (int)rdlen, (int)end_remain_buf_len); //error here too
//xbuffer = do stuff to make it *3 longer or *(1/3) shorter;
fw.Write(xbuffer, (int)wrlen, (int)end_remain_buf_len * ((tf) ? (1 / 3) : 3));
rdlen += end_remain_buf_len;
wrlen += chunk;
}
//Close opened files
fs.Close();
fw.Close();
return false; //no functionality yet lol
}

The Read() method of Stream (the base class of FileStream) returns an int indicating the number of bytes read, and 0 when it has no more bytes to read, so you don't even need to know the file size beforehand:
public static void CopyFileChunked(int chunkSize, string filepath, string output)
{
byte[] chunk = new byte[chunkSize];
using (FileStream reader = new FileStream(filepath, FileMode.Open))
using (FileStream writer = new FileStream(output, FileMode.Create))
{
int bytes;
while ((bytes = reader.Read(chunk , 0, chunkSize)) > 0)
{
writer.Write(chunk, 0, bytes);
}
}
}
Or even File.Copy() may do the trick, if you can live with letting the framework decide about the chunk size.

I think it's failing on this line:
fw.Write(xbuffer, (int)wrlen, chunk);
You are declaring xbuffer as
byte[] xbuffer = new byte[(buffer.Length * ((tf) ? (1 / 3) : 3))];
Since 1 / 3 is an integer division, it returns 0.And you are declaring xbuffer with the size 0 hence the error.You can fix it by casting one of the operand to a floating point type or using literals.But then you still need to cast the result back to integer.
byte[] xbuffer = new byte[(int)(buffer.Length * ((tf) ? (1m / 3) : 3))];
The same problem also present in the chunk declaration.

How to write an array of doubles to a file very fast in C#?

I want to write something like this to a file:
FileStream output = new FileStream("test.bin", FileMode.Create, FileAccess.ReadWrite);
BinaryWriter binWtr = new BinaryWriter(output);
double [] a = new double [1000000]; //this array fill complete
for(int i = 0; i < 1000000; i++)
{
binWtr.Write(a[i]);
}
And unfortunately this code's process last very long!
(in this example about 10 seconds!)
The file format is binary.
How can I make that faster?

You should be able to speed up the process by converting your array of doubles to an array of bytes, and then write the bytes in one shot.
This answer shows how to do the conversion (the code below comes from that answer):
static byte[] GetBytes(double[] values) {
var result = new byte[values.Length * sizeof(double)];
Buffer.BlockCopy(values, 0, result, 0, result.Length);
return result;
}
With the array of bytes in hand, you can call Write that takes an array of bytes:
var byteBuf = GetBytes(a);
binWtr.Write(byteBuf);

You're writing the bytes 1 by 1, of course it's going to be slow.
You could do the writing in memory to an array and then write the array to disk all at once like this :
var arr = new double[1000000];
using(var strm = new MemoryStream())
using (var bw = new BinaryWriter(strm))
{
foreach(var d in arr)
{
bw.Write(d);
}
bw.Flush();
File.WriteAllBytes("myfile.bytes",strm.ToArray());
}

How to point to the particular index of BinaryStream?

I have a byte array consist of 32 bytes.
I want to read 4 bytes from index position 16 to 19.
How can i point binary reader to start reading from index 16.
I am trying these commands
byte[] trace ; // 32 byte array
using (FileStream s = File.OpenRead(filename))
using (BinaryReader r = new BinaryReader(s))
{
r.baseStream.Seek(position,SeekOrigin.Begin);
byte[] by = r.ReadBytes(4);
}
but i don't know what to put at position?

I think I got it (although your sample in the question is not very clear).
You have the byte array trace with 32 elements in it and you want to read 4 bytes starting with position 16.
Assuming that endianness is not a variable, you can use this to read the 4 bytes as an int value or byte array:
using(var memStream = new MemoryStream(trace))
{
//position the stream
using(var reader = new BinaryReader(memStream)
{
memStream.Seek(16, SeekOrigin.Begin);
var intValue = reader.ReadInt32();
memStream.Seek(16, SeekOrigin.Begin);
//now read a byte array
var byteArray = reader.ReadBytes(4);
}
}

Need help Creating a big array from small byte arrays

i got the following code:
byte[] myBytes = new byte[10 * 10000];
for (long i = 0; i < 10000; i++)
{
byte[] a1 = BitConverter.GetBytes(i);
byte[] a2 = BitConverter.GetBytes(true);
byte[] a3 = BitConverter.GetBytes(false);
byte[] rv = new byte[10];
System.Buffer.BlockCopy(a1, 0, rv, 0, a1.Length);
System.Buffer.BlockCopy(a2, 0, rv, a1.Length, a2.Length);
System.Buffer.BlockCopy(a3, 0, rv, a1.Length + a2.Length, a3.Length);
}
everything works as it should. i was trying to convert this code so everything will be written into myBytes but then i realised, that i use a long and if its value will be higher then int.MaxValue casting will fail.
how could one solve this?
another question would be, since i dont want to create a very large bytearray in memory, how could i send it directry to my .WriteBytes(path, myBytes); function ?

If the final destination for this is, as suggested, a file: then write to a file more directly, rather than buffering in memory:
using (var file = File.Create(path)) // or append file FileStream etc
using (var writer = new BinaryWriter(file))
{
for (long i = 0; i < 10000; i++)
{
writer.Write(i);
writer.Write(true);
writer.Write(false);
}
}
Perhaps the ideal way of doing this in your case would be to pass a single BinaryWriter instance to each object in turn as you serialize them (don't open and close the file per-object).

Why don't you just Write() the bytes out as you process them rather than converting to a massive buffer, or use a smaller buffer at least?