I am trying to upload large files to 3rd part service by chunks. But I have problem with last chunk. Last chunk would be always smaller then 5mb, but all chunks incl. the last have all the same size - 5mb
My code:
int chunkSize = 1024 * 1024 * 5;
using (Stream streamx = new FileStream(file.Path, FileMode.Open, FileAccess.Read))
{
byte[] buffer = new byte[chunkSize];
int bytesRead = 0;
long bytesToRead = streamx.Length;
while (bytesToRead > 0)
{
int n = streamx.Read(buffer, 0, chunkSize);
if (n == 0) break;
// do work on buffer...
// uploading chunk ....
var partRequest = HttpHelpers.InvokeHttpRequestStream
(
new Uri(endpointUri + "?partNumber=" + i + "&uploadId=" + UploadId),
"PUT",
partHeaders,
buffer
); // upload buffer
bytesRead += n;
bytesToRead -= n;
}
streamx.Dispose();
}
buffer is uploaded on 3rd party service.
Solved, someone posted updated code in comment, but after some seconds deleted this comment. But there was solution. I added this part after
if (n == 0)
this code, which resizes last chunk on the right size
// Let's resize the last incomplete buffer
if (n != buffer.Length)
Array.Resize(ref buffer, n);
Thank you all.
I post full working code:
int chunkSize = 1024 * 1024 * 5;
using (Stream streamx = new FileStream(file.Path, FileMode.Open, FileAccess.Read))
{
byte[] buffer = new byte[chunkSize];
int bytesRead = 0;
long bytesToRead = streamx.Length;
while (bytesToRead > 0)
{
int n = streamx.Read(buffer, 0, chunkSize);
if (n == 0) break;
// Let's resize the last incomplete buffer
if (n != buffer.Length)
Array.Resize(ref buffer, n);
// do work on buffer...
// uploading chunk ....
var partRequest = HttpHelpers.InvokeHttpRequestStream
(
new Uri(endpointUri + "?partNumber=" + i + "&uploadId=" + UploadId),
"PUT",
partHeaders,
buffer
); // upload buffer
bytesRead += n;
bytesToRead -= n;
}
}
Related
I'm uploading images to Azure Storage, and need to implement chunking to upload heavy image.
But I'm getting exception such as "Offset and length out of bound"
int fileSize = imageModel.ImageByteArray.Length;
int blockSize = 100*1024 , numberOfBlocks = 0;
if (fileSize < blockSize)
blockSize = fileSize;
if (fileSize % blockSize == 0)
numberOfBlocks = fileSize / blockSize;
else
numberOfBlocks = fileSize / blockSize + 1;
byte[] blockArray = new byte[blockSize];
for (int i = 0; i < numberOfBlocks; i++)
{
Array.Copy(imageModel.ImageByteArray, i * blockSize, blockArray, 0, blockSize);
await blob.UploadFromByteArrayAsync(blockArray, i, imageModel.ImageByteArray.Length);
Progress = (float)i / numberOfBlocks;
}
var ResponseURL = blob.Uri.OriginalString;
Can anyone help me with the issue?
This code always copies blockSize bytes:
Array.Copy(imageModel.ImageByteArray, i * blockSize, blockArray, 0, blockSize);
However, the last block may not be a full blockSize:
if (fileSize % blockSize == 0)
numberOfBlocks = fileSize / blockSize;
else
numberOfBlocks = fileSize / blockSize + 1;
To fix this, you'll need to handle the last block specially. It may be a full blockSize bytes or it may be less.
So I have this:
public static long FindPosition(Stream stream, byte[] byteSequence)
{
if (byteSequence.Length > stream.Length)
return -1;
byte[] buffer = new byte[byteSequence.Length];
using (BufferedStream bufStream = new BufferedStream(stream, byteSequence.Length))
{
int i;
while ((i = bufStream.Read(buffer, 0, byteSequence.Length)) == byteSequence.Length)
{
if (byteSequence.SequenceEqual(buffer))
return bufStream.Position - byteSequence.Length;
else
bufStream.Position -= byteSequence.Length - PadLeftSequence(buffer, byteSequence);
}
}
return -1;
}
private static int PadLeftSequence(byte[] bytes, byte[] seqBytes)
{
int i = 1;
while (i < bytes.Length)
{
int n = bytes.Length - i;
byte[] aux1 = new byte[n];
byte[] aux2 = new byte[n];
Array.Copy(bytes, i, aux1, 0, n);
Array.Copy(seqBytes, aux2, n);
if (aux1.SequenceEqual(aux2))
return i;
i++;
}
return i;
}
Which works perfectly to get an offset that has a specific set of bytes, but now I want to do the inverse, find a set of bytes from a specific offset.
How can I do that?
Try this example:
long offset = 100L; // Offset
int bytesCount = 20; // Number of bytes to read
byte[] buffer = new byte[bytesCount];
stream.Seek( offset, SeekOrigin.Begin ); // Set offset from Begin of a stream
stream.Read( buffer, 0, bytesCount ); // Read bytesCount from previous set offset
More detail about Seek and Read
Hi i currently use the following code to split a file into muliple 2mb smaller parts.
const int BUFFER_SIZE = 20 * 1024;
byte[] buffer = new byte[BUFFER_SIZE];
using (Stream input = File.OpenRead(inputFile)) {
int index = 0;
while (input.Position < input.Length) {
using (Stream output = File.Create(path)) {
int remaining = chunkSize, bytesRead;
while (remaining > 0 && (bytesRead = input.Read(buffer, 0,
Math.Min(remaining, BUFFER_SIZE))) > 0) {
output.Write(buffer, 0, bytesRead);
remaining -= bytesRead;
}
}
}
index++;
}
This works perfectly, and will split a 10mb file into 5 x 2mb files 0.part,2.part ect...
I would like to know how I would generate just part 3 again knowing the chunkSize always stays at 2mb. I can achieve this by wrapping in an if,else and evaluating index, but with a 1GB file this process can take a while to loop through. I'd like to understand this function more and how I can just get the part of the file I require?
input.Position property is settable. If you know that you need part 3, set Position to 2*chunkSize to skip the first two chunks, and do the innermost while loop once to copy from that position to the output:
int desiredChunkNumber = 3;
using (Stream input = File.OpenRead(inputFile)) {
input.Position = (desiredChunkNumber - 1) * chunkSize;
using (Stream output = File.Create(path)) {
int remaining = chunkSize, bytesRead;
while (remaining > 0 && (bytesRead = input.Read(buffer, 0,
Math.Min(remaining, BUFFER_SIZE))) > 0) {
output.Write(buffer, 0, bytesRead);
remaining -= bytesRead;
}
}
}
I'm having trouble seperating the channel buffers into a new file.
Here is the code for extracting each channels buffer:
int samplesDesired = 10000;
byte[] buffer = new byte[samplesDesired * 4];
short[] left = new short[samplesDesired];
short[] right = new short[samplesDesired];
using (WaveFileReader pcm = new WaveFileReader(filePath))
{
int bytesRead = pcm.Read(buffer, 0, 10000);
int index = 0;
for (int i = 0; i < bytesRead / 4; i++)
{
left[i] = BitConverter.ToInt16(buffer, index);
index += 2;
right[i] = BitConverter.ToInt16(buffer, index);
index += 2;
}
}
And here is how I try to create a file from the gathered buffers:
using(var leftChannelFile = new WaveFileWriter("test.wav", new WaveFormat()))
{
leftChannelFile.WriteSamples(left, 0, left.Length);
}
The problem is, when I try to play the "file.wav", it is 0 seconds long and 19,5 KB large. Any idea on why is that happening?
Either you set the header of the resulting files indicating it is one channel/mono,
OR
you add a second empty channel to your result files (all 0 Bytes)
The Microsoft website has the code snippet:
using (FileStream fsSource = new FileStream(pathSource,
FileMode.Open, FileAccess.Read))
{
// Read the source file into a byte array.
byte[] bytes = new byte[fsSource.Length];
int numBytesToRead = (int)fsSource.Length;
int numBytesRead = 0;
while (numBytesToRead > 0)
{
// Read may return anything from 0 to numBytesToRead.
int n = fsSource.Read(bytes, numBytesRead, numBytesToRead);
// Break when the end of the file is reached.
if (n == 0)
break;
numBytesRead += n;
numBytesToRead -= n;
}
}
What concerns me is that fsSource.Length is a long, whereas numBytesRead is an int so at most only 2 * int.MaxValue can be read into bytes (the head and the tail of the stream). So my questions are:
Is there some reason that this is OK?
If not, how should you read a FileStream into a byte[].
In this situation I wouldn't even bother processing the FileStream manually; use File.ReadAllBytes instead:
byte[] bytes = File.ReadAllBytes(pathSource);
To answer your question:
The sample code is good for most of applications where we are not reaching extremes.
If you have really long stream like say a video, use BufferedStream. Sample code is available at MSDN site
Example using ReadAllBytes:
private byte[] m_cfgBuffer;
m_cfgBuffer = File.ReadAllBytes(m_FileName);
StringBuilder PartNbr = new StringBuilder();
StringBuilder Version = new StringBuilder();
int i, j;
byte b;
i = 356; // We know that the cfg file header ends at position 356 (1st hex(80))
b = m_cfgBuffer[i];
while (b != 0x80) // Scan for 2nd hex(80)
{
i++;
b = m_cfgBuffer[i];
}
// Now extract the part number - 6 bytes after hex(80)
m_PartNbrPos = i + 5;
for (j = m_PartNbrPos; j < m_PartNbrPos + 6; j++)
{
char cP = (char)m_cfgBuffer[j];
PartNbr.Append(cP);
}
m_PartNbr = PartNbr.ToString();
// Now, extract version number - 6 bytes after part number
m_VersionPos = (m_PartNbrPos + 6) + 6;
for (j = m_VersionPos; j < m_VersionPos + 2; j++)
{
char cP = (char)m_cfgBuffer[j];
Version.Append(cP);
}
m_Version = Version.ToString();