I've created a simple buffer manager class to be used with asyncroneous sockets. This will protect against memory fragmentation and improve performance. Any suggestions for further improvements or other approaches?
public class BufferManager
{
private int[] free;
private byte[] buffer;
private readonly int blocksize;
public BufferManager(int count, int blocksize)
{
buffer = new byte[count * blocksize];
free = new int[count];
this.blocksize = blocksize;
for (int i = 0; i < count; i++)
free[i] = 1;
}
public void SetBuffer(SocketAsyncEventArgs args)
{
for (int i = 0; i < free.Length; i++)
{
if (1 == Interlocked.CompareExchange(ref free[i], 0, 1))
{
args.SetBuffer(buffer, i * blocksize, blocksize);
return;
}
}
args.SetBuffer(new byte[blocksize], 0, blocksize);
}
public void FreeBuffer(SocketAsyncEventArgs args)
{
int offset = args.Offset;
byte[] buff = args.Buffer;
args.SetBuffer(null, 0, 0);
if (buffer == buff)
free[offset / blocksize] = 1;
}
}
Edit:
The orignal answer below addresses a code construction issue of overly tight coupling. However, considering the solution as whole I would avoid using just one large buffer and handing over slices of it in this way. You expose your code to buffer overrun (and shall we call it buffer "underrun" issues). Instead I would manage an array of byte arrays each being a discrete buffer. Offset handed over is always 0 and size is always the length of the buffer. Any bad code that attempts to read/write parts beyond the boundaries will be caught.
Original answer
You've coupled the class to SocketAsyncEventArgs where in fact all it needs is a function to assign the buffer, change SetBuffer to:-
public void SetBuffer(Action<byte[], int, int> fnSet)
{
for (int i = 0; i < free.Length; i++)
{
if (1 == Interlocked.CompareExchange(ref free[i], 0, 1))
{
fnSet(buffer, i * blocksize, blocksize);
return;
}
}
fnSet(new byte[blocksize], 0, blocksize);
}
Now you can call from consuming code something like this:-
myMgr.SetBuffer((buf, offset, size) => myArgs.SetBuffer(buf, offset, size));
I'm not sure that type inference is clever enough to resolve the types of buf, offset, size in this case. If not you will have to place the types in the argument list:-
myMgr.SetBuffer((byte[] buf, int offset, int size) => myArgs.SetBuffer(buf, offset, size));
However now your class can be used to allocate a buffer for all manner of requirements that also use the byte[], int, int pattern which is very common.
Of course you need to decouple the free operation to but thats:-
public void FreeBuffer(byte[] buff, int offset)
{
if (buffer == buff)
free[offset / blocksize] = 1;
}
This requires you to call SetBuffer on the EventArgs in consuming code in the case for SocketAsyncEventArgs. If you are concerned that this approach reduces the atomicity of freeing the buffer and removing it from the sockets use, then sub-class this adjusted buffer manager and include SocketAsyncEventArgs specific code in the sub-class.
I've created a new class with a completely different approach.
I have a server class that receives byte arrays. It will then invoke different delegates handing them the buffer objects so that other classes can process them. When those classes are done they need a way to push the buffers back to the stack.
public class SafeBuffer
{
private static Stack bufferStack;
private static byte[][] buffers;
private byte[] buffer;
private int offset, lenght;
private SafeBuffer(byte[] buffer)
{
this.buffer = buffer;
offset = 0;
lenght = buffer.Length;
}
public static void Init(int count, int blocksize)
{
bufferStack = Stack.Synchronized(new Stack());
buffers = new byte[count][];
for (int i = 0; i < buffers.Length; i++)
buffers[i] = new byte[blocksize];
for (int i = 0; i < buffers.Length; i++)
bufferStack.Push(new SafeBuffer(buffers[i]));
}
public static SafeBuffer Get()
{
return (SafeBuffer)bufferStack.Pop();
}
public void Close()
{
bufferStack.Push(this);
}
public byte[] Buffer
{
get
{
return buffer;
}
}
public int Offset
{
get
{
return offset;
}
set
{
offset = value;
}
}
public int Lenght
{
get
{
return buffer.Length;
}
}
}
Related
I am trying to write a massive object to AWS S3 (e.g. 25 GB).
Currently I can get it working in two ways:
Write the content to a file on local disk, then send the file to S3 using multi-part upload
Write the content to a MemoryStream, then send that stream to S3 using multi-part upload
However, I don't like either approach, because I need to reserve a large amount of disk space or memory for the operation. I am generating this content in code, so I was hoping to open a stream to an S3 object, and generate the content directly to that object. But I can't see how to make that work.
Is it possible to build a massive object in S3 without representing the entire object in a local file or memory first?
(Note: My question is very similar to this question, but that question doesn't have a useful answer.)
I was able to get it working by breaking the overall payload into chunks, and sending each individual chunk as a separate MemoryStream.
Technically this solution still uses a MemoryStream, but that's OK, since I can control how much memory is used by adjusting the chunk size. For my test, I created a 25GB file while keeping memory usage well below that (~2 GB IIRC).
Here is my solution:
private const string BucketName = "YOUR-BUCKET-NAME-HERE";
private static readonly RegionEndpoint BucketRegion = RegionEndpoint.USEast1;
private const string Key = "massive-file-test";
// We're going to send 100 chunks of 256 MB each, for a total of 25 GB.
// The content will be the asterisk ("*") repeated for the desired size.
private const int ChunkSizeMb = 256;
private const int TotalSizeGb = 25;
public static void Main(string[] args)
{
Console.WriteLine($"Writing object to {BucketName}, {Key}");
int totalChunks = TotalSizeGb * 1024 / ChunkSizeMb;
int chunkSizeBytes = ChunkSizeMb * 1024 * 1024;
string payload = new String('*', chunkSizeBytes);
// Initiate the request.
InitiateMultipartUploadRequest initiateRequest = new InitiateMultipartUploadRequest
{
BucketName = BucketName,
Key = Key
};
List<UploadPartResponse> uploadResponses = new List<UploadPartResponse>();
IAmazonS3 s3Client = new AmazonS3Client(BucketRegion);
InitiateMultipartUploadResponse initResponse = s3Client.InitiateMultipartUpload(initiateRequest);
// Open a stream to build the input.
for (int i = 0; i < totalChunks; i++)
{
// Write the next chunk to the input stream.
Console.WriteLine($"Writing chunk {i} of {totalChunks}");
using (var stream = ToStream(payload))
{
// Write the next chunk to s3.
UploadPartRequest uploadRequest = new UploadPartRequest
{
BucketName = BucketName,
Key = Key,
UploadId = initResponse.UploadId,
PartNumber = i + 1,
PartSize = chunkSizeBytes,
InputStream = stream,
};
uploadResponses.Add(s3Client.UploadPart(uploadRequest));
}
}
// Complete the request.
CompleteMultipartUploadRequest completeRequest = new CompleteMultipartUploadRequest
{
BucketName = BucketName,
Key = Key,
UploadId = initResponse.UploadId
};
completeRequest.AddPartETags(uploadResponses);
s3Client.CompleteMultipartUpload(completeRequest);
Console.WriteLine("Script is complete. Press any key to exit...");
Console.ReadKey();
}
private static Stream ToStream(string s)
{
var stream = new MemoryStream();
var writer = new StreamWriter(stream);
writer.Write(s);
writer.Flush();
stream.Position = 0;
return stream;
}
Here is what AnonCoward started, finished off by adding seeking - it's a trivial op for a stream that does nothing except write asterisks to its buffer. If you were generating more complex data it would be hard work but for seeking all you need to do is set the position and say "yep, done that" because no matter where you seek to in the stream the behavior of creating asterisks is always the same
class AsteriskGeneratingStream : Stream
{
long _pos = 0;
long _length = 0;
public AsteriskGeneratingStream(long length)
{
_length = length;
}
public override long Length => _length;
public override int Read(byte[] buffer, int offset, int count)
{
// Create the data as needed
if (count + _pos > _length)
count = (int)(_length - _pos);
for (int i = offset; i < count; i++)
buffer[i] = (byte)'*';
_pos += count;
return count;
}
public override bool CanRead => true;
public override long Seek(long offset, SeekOrigin origin)
{
if(origin == SeekOrigin.Begin) //lets just trust that the caller will be sensible and not set e.g. negative offset
_pos = offset;
else if(origin == SeekOrigin.Current)
_pos += offset;
else if(origin == SeekOrigin.End)
_pos = _length + offset;
return _pos;
}
public override bool CanSeek => true;
public override bool CanWrite => false;
public override long Position { get => _pos; set => _pos = value; }
public override void Flush() { }
public override void SetLength(long value) { _length = value; }
public override void Write(byte[] buffer, int offset, int count) { throw new NotImplementedException(); }
}
class Program
{
static void Main(string[] args)
{
long objectSize = 25L * 1024 * 1024;
var s3 = new AmazonS3Client(Amazon.RegionEndpoint.USWest1);
var xfer = new TransferUtility(s3,new TransferUtilityConfig
{
MinSizeBeforePartUpload = 5L * 1024 * 1024
});
var helper = new AsteriskGeneratingStream(objectSize);
xfer.Upload(helper, "bucket-name", "object-key");
}
}
Note, I can't guarantee it'll work right off the bat because I'm on a cellphone and can't test this via c# fiddle but let's see how it blows up! 😀
If you can create the object on the fly, or at least cache fairly small segments, you can create a stream that serves the data up to S3. Note, that unless you can also create any part of the object out of order, you need to prevent the AWS SDK from using a multi-part upload, which will slow down the transfer speed.
class DataStream : Stream
{
long _pos = 0;
long _length = 0;
public DataStream(long length)
{
_length = length;
}
public override long Length => _length;
public override int Read(byte[] buffer, int offset, int count)
{
// Create the data as needed, on demand
// For this example, just cycle through 0 to 256 in the data over and over again
if (count + _pos > _length)
{
count = (int)(_length - _pos);
}
for (int i = 0; i < count; i++)
{
buffer[i + offset] = (byte)((_pos + i) % 256);
}
_pos += count;
return count;
}
public override bool CanRead => true;
// Stub out all other methods. For a seekable stream
// Seek() and Postion need to be implemented, along with CanSeek changed
public override long Seek(long offset, SeekOrigin origin) { throw new NotImplementedException(); }
public override bool CanSeek => false;
public override bool CanWrite => false;
public override long Position { get => _pos; set => throw new NotImplementedException(); }
public override void Flush() { throw new NotImplementedException(); }
public override void SetLength(long value) { throw new NotImplementedException(); }
public override void Write(byte[] buffer, int offset, int count) { throw new NotImplementedException(); }
}
class Program
{
static void Main(string[] args)
{
long objectSize = 25L * 1024 * 1024;
var s3 = new AmazonS3Client(Amazon.RegionEndpoint.USWest1);
// Prevent a multi-part upload, which requires a seekable stream
var xfer = new TransferUtility(s3, new TransferUtilityConfig
{
MinSizeBeforePartUpload = objectSize + 1
});
var helper = new DataStream(objectSize);
xfer.Upload(helper, "bucket-name", "object-key");
}
}
In order to clean some messy code and get a better understanding of the SocketAsyncEventArgs class, I'd to know what's the most efficient technique to reassemble partially received messages from SocketAsyncEventArgs buffers.
To give you the big picture, I'm connected to a TCP server using a C# Socket client that will essentially receive data. The data received is message-based delimited by a \n character.
As you're probably already aware of, when using the ReceiveAsync method, this is almost a certitude that the last received message will be uncompleted such as you'll have to locate the index of the last complete message, copy the incomplete buffer section and keep it as start for the next received buffer and so on.
The thing is, I wish to abstract this operation from the upper layer and call the ProcessReceiveDataImpl as soon I get completed messages in the _tmpBuffer. I found that my Buffer.BlockCopy is not much readable (very old code also (-:) but anyway I wish to know what are you doing in this typical use case?
Code to reassemble messages:
public class SocketClient
{
private const int _receiveBufferSize = 8192;
private byte[] _remBuffer = new byte[2 * _receiveBufferSize];
private byte[] _tmpBuffer = new byte[2 * _receiveBufferSize];
private int _remBufferSize = 0;
private int _tmpBufferSize = 0;
private void ProcessReceiveData(SocketAsyncEventArgs e)
{
// the buffer to process
byte[] curBuffer = e.Buffer;
int curBufferSize = e.BytesTransferred;
int curBufferOffset = e.Offset;
int curBufferLastIndex = e.BytesTransferred - 1;
int curBufferLastSplitIndex = int.MinValue;
if (_remBufferSize > 0)
{
curBufferLastSplitIndex = GetLastSplitIndex(curBuffer, curBufferOffset, curBufferSize);
if (curBufferLastSplitIndex != curBufferLastIndex)
{
// copy the remain + part of the current into tmp
Buffer.BlockCopy(_remBuffer, 0, _tmpBuffer, 0, _remBufferSize);
Buffer.BlockCopy(curBuffer, curBufferOffset, _tmpBuffer, _remBufferSize, curBufferLastSplitIndex + 1);
_tmpBufferSize = _remBufferSize + curBufferLastSplitIndex + 1;
ProcessReceiveDataImpl(_tmpBuffer, _tmpBufferSize);
Buffer.BlockCopy(curBuffer, curBufferLastSplitIndex + 1, _remBuffer, 0, curBufferLastIndex - curBufferLastSplitIndex);
_remBufferSize = curBufferLastIndex - curBufferLastSplitIndex;
}
else
{
// copy the remain + entire current into tmp
Buffer.BlockCopy(_remBuffer, 0, _tmpBuffer, 0, _remBufferSize);
Buffer.BlockCopy(curBuffer, curBufferOffset, _tmpBuffer, _remBufferSize, curBufferSize);
ProcessReceiveDataImpl(_tmpBuffer, _remBufferSize + curBufferSize);
_remBufferSize = 0;
}
}
else
{
curBufferLastSplitIndex = GetLastSplitIndex(curBuffer, curBufferOffset, curBufferSize);
if (curBufferLastSplitIndex != curBufferLastIndex)
{
// we must copy the unused byte into remaining buffer
_remBufferSize = curBufferLastIndex - curBufferLastSplitIndex;
Buffer.BlockCopy(curBuffer, curBufferLastSplitIndex + 1, _remBuffer, 0, _remBufferSize);
// process the msg
ProcessReceiveDataImpl(curBuffer, curBufferLastSplitIndex + 1);
}
else
{
// we can process the entire msg
ProcessReceiveDataImpl(curBuffer, curBufferSize);
}
}
}
protected virtual void ProcessReceiveDataImpl(byte[] buffer, int bufferSize)
{
}
private int GetLastSplitIndex(byte[] buffer, int offset, int bufferSize)
{
for (int i = offset + bufferSize - 1; i >= offset; i--)
{
if (buffer[i] == '\n')
{
return i;
}
}
return -1;
}
}
Your input is very important and appreciated!
Thank you!
Updated:
Also, rather then calling the ProcessReceiveDataImpl and block further receive operations, will it be useful to queue completed messages and make them available to the consumer?
I have a class that can open memory mapped files, read and write to it :
public class Memory
{
protected bool _lock;
protected Mutex _locker;
protected MemoryMappedFile _descriptor;
protected MemoryMappedViewAccessor _accessor;
public void Open(string name, int size)
{
_descriptor = MemoryMappedFile.CreateOrOpen(name, size);
_accessor = _descriptor.CreateViewAccessor(0, size, MemoryMappedFileAccess.ReadWrite);
_locker = new Mutex(true, Guid.NewGuid().ToString("N"), out _lock);
}
public void Close()
{
_accessor.Dispose();
_descriptor.Dispose();
_locker.Close();
}
public Byte[] Read(int count, int index = 0, int position = 0)
{
Byte[] bytes = new Byte[count];
_accessor.ReadArray<Byte>(position, bytes, index, count);
return bytes;
}
public void Write(Byte[] data, int count, int index = 0, int position = 0)
{
_locker.WaitOne();
_accessor.WriteArray<Byte>(position, data, index, count);
_locker.ReleaseMutex();
}
Usually I use it this way :
var data = new byte[5];
var m = new Memory();
m.Open("demo", sizeof(data));
m.Write(data, 5);
m.Close();
I would like to implement some kind of lazy loading for opening and want to open file only when I am ready to write there something, e.g. :
public void Write(string name, Byte[] data, int count, int index = 0, int position = 0)
{
_locker.WaitOne();
Open(name, sizeof(byte) * count); // Now I don't need to call Open() before the write
_accessor.WriteArray<Byte>(position, data, index, count);
_locker.ReleaseMutex();
}
Question : when I call "Write" method several times (in a loop) it will cause member variables (like _locker) to reinitialise and I would like to know - is it safe to do it this way, can it cause memory leaks or unpredictable behavior with mutex?
If you open in the write method using a lock, it's safe to close before you release the mutex.
When you are dealing with unmanaged resources and disposable object, it's always better to implement IDispose interface correctly. Here is some more information.
Then you can initialse Memory instance in a using clause
using (var m = new Memory())
{
// Your read write
}
I want to be able to compute the hashes of arbitrarily sized file chunks of a file in C#.
eg: Compute the hash of the 3rd gigabyte in 4gb file.
The main problem is that I don't want to load the entire file at memory, as there could be several files and the offsets could be quite arbitrary.
AFAIK, the HashAlgorithm.ComputeHash allows me to either use a byte buffer, of a stream. The stream would allow me to compute the hash efficiently, but for the entire file, not just for a specific chunk.
I was thinking to create aan alternate FileStream object and pass it to ComputeHash, where I would overload the FileStream methods and have read only for a certain chunk in a file.
Is there a better solution than this, preferably using the built in C# libraries ?
Thanks.
You should pass in either:
A byte array containing the chunk of data to compute the hash from
A stream that restricts access to the chunk you want to computer the hash from
The second option isn't all that hard, here's a quick LINQPad program I threw together. Note that it lacks quite a bit of error handling, such as checking that the chunk is actually available (ie. that you're passing in a position and length of the stream that actually exists and doesn't fall off the end of the underlying stream).
Needless to say, if this should end up as production code I would add a lot of error handling, and write a bunch of unit-tests to ensure all edge-cases are handled correctly.
You would construct the PartialStream instance for your file like this:
const long gb = 1024 * 1024 * 1024;
using (var fileStream = new FileStream(#"d:\temp\too_long_file.bin", FileMode.Open))
using (var chunk = new PartialStream(fileStream, 2 * gb, 1 * gb))
{
var hash = hashAlgorithm.ComputeHash(chunk);
}
Here's the LINQPad test program:
void Main()
{
var buffer = Enumerable.Range(0, 256).Select(i => (byte)i).ToArray();
using (var underlying = new MemoryStream(buffer))
using (var partialStream = new PartialStream(underlying, 64, 32))
{
var temp = new byte[1024]; // too much, ensure we don't read past window end
partialStream.Read(temp, 0, temp.Length);
temp.Dump();
// should output 64-95 and then 0's for the rest (64-95 = 32 bytes)
}
}
public class PartialStream : Stream
{
private readonly Stream _UnderlyingStream;
private readonly long _Position;
private readonly long _Length;
public PartialStream(Stream underlyingStream, long position, long length)
{
if (!underlyingStream.CanRead || !underlyingStream.CanSeek)
throw new ArgumentException("underlyingStream");
_UnderlyingStream = underlyingStream;
_Position = position;
_Length = length;
_UnderlyingStream.Position = position;
}
public override bool CanRead
{
get
{
return _UnderlyingStream.CanRead;
}
}
public override bool CanWrite
{
get
{
return false;
}
}
public override bool CanSeek
{
get
{
return true;
}
}
public override long Length
{
get
{
return _Length;
}
}
public override long Position
{
get
{
return _UnderlyingStream.Position - _Position;
}
set
{
_UnderlyingStream.Position = value + _Position;
}
}
public override void Flush()
{
throw new NotSupportedException();
}
public override long Seek(long offset, SeekOrigin origin)
{
switch (origin)
{
case SeekOrigin.Begin:
return _UnderlyingStream.Seek(_Position + offset, SeekOrigin.Begin) - _Position;
case SeekOrigin.End:
return _UnderlyingStream.Seek(_Length + offset, SeekOrigin.Begin) - _Position;
case SeekOrigin.Current:
return _UnderlyingStream.Seek(offset, SeekOrigin.Current) - _Position;
default:
throw new ArgumentException("origin");
}
}
public override void SetLength(long length)
{
throw new NotSupportedException();
}
public override int Read(byte[] buffer, int offset, int count)
{
long left = _Length - Position;
if (left < count)
count = (int)left;
return _UnderlyingStream.Read(buffer, offset, count);
}
public override void Write(byte[] buffer, int offset, int count)
{
throw new NotSupportedException();
}
}
You can use TransformBlock and TransformFinalBlock directly. That's pretty similar to what HashAlgorithm.ComputeHash does internally.
Something like:
using(var hashAlgorithm = new SHA256Managed())
using(var fileStream = new File.OpenRead(...))
{
fileStream.Position = ...;
long bytesToHash = ...;
var buf = new byte[4 * 1024];
while(bytesToHash > 0)
{
var bytesRead = fileStream.Read(buf, 0, (int)Math.Min(bytesToHash, buf.Length));
hashAlgorithm.TransformBlock(buf, 0, bytesRead, null, 0);
bytesToHash -= bytesRead;
if(bytesRead == 0)
throw new InvalidOperationException("Unexpected end of stream");
}
hashAlgorithm.TransformFinalBlock(buf, 0, 0);
var hash = hashAlgorithm.Hash;
return hash;
};
Your suggestion - passing in a restricted access wrapper for your FileStream - is the cleanest solution. Your wrapper should defer everything to the wrapped Stream except the Length and Position properties.
How? Simply create a class that inherits from Stream. Make the constructor take:
Your source Stream (in your case, a FileStream)
The chunk start position
The chunk end position
As an extension - this is a list of all the Streams that are available http://msdn.microsoft.com/en-us/library/system.io.stream%28v=vs.100%29.aspx#inheritanceContinued
To easily compute the hash of a chunk of a larger stream, use these two methods:
HashAlgorithm.TransformBlock
HashAlgorithm.TransformFinalBlock
Here's a LINQPad program that demonstrates:
void Main()
{
const long gb = 1024 * 1024 * 1024;
using (var stream = new FileStream(#"d:\temp\largefile.bin", FileMode.Open))
{
stream.Position = 2 * gb; // 3rd gb-chunk
byte[] buffer = new byte[32768];
long amount = 1 * gb;
using (var hashAlgorithm = SHA1.Create())
{
while (amount > 0)
{
int bytesRead = stream.Read(buffer, 0,
(int)Math.Min(buffer.Length, amount));
if (bytesRead > 0)
{
amount -= bytesRead;
if (amount > 0)
hashAlgorithm.TransformBlock(buffer, 0, bytesRead,
buffer, 0);
else
hashAlgorithm.TransformFinalBlock(buffer, 0, bytesRead);
}
else
throw new InvalidOperationException();
}
hashAlgorithm.Hash.Dump();
}
}
}
To answer your original question ("Is there a better solution..."):
Not that I know of.
This seems to be a very special, non-trivial task, so a little extra work might be involved anyway. I think your approach of using a custom Stream-class goes in the right direction, I'd probably do exactly the same.
And Gusdor and xander have already provided very helpful information on how to implement that — good job guys!
I have this method that I need to call and use in my application, but I don't know really know how to do it exactly.
This is the function that I need to call.
[DllImport(dll_Path)]
public static extern int DTS_GetDataToBuffer(int Position, int Length, char* Buffer, int* DataRead);
In my code, I have this function and I'm missing its implementation.
internal static void GetDataToBuffer(int position, int length, out byte[] data, out int dataRead)
{
unsafe
{
// the code I need
}
}
I think most of this is very selfexplanatory. I need to implement the latter function so I can be able to read the data into the buffer and the amount of data read (which should actually be the same as data.Length, but the manufacturer has this as separate option, so I need it).
Can anyone help? Is this clear enough?
Thank you
Edit: Here is the unmanaged declaration from the .h file. Hope it helps.
extern NAG_DLL_EXPIMP int DTS_GetDataToBuffer(int Position,
int Length,
unsigned char *Buffer,
int *DataRead );
Edit #2:
Positon - the position from which to star reading the data.
Length - The amount of data to read (this would be the buffer size).
DataRead - the actual data size that was read.
I don't think you really need to use unsafe pointers here.
Declare function as
[DllImport(dll_Path)]
public static extern int DTS_GetDataToBuffer(
int position,
int length,
byte[] buffer,
ref int dataRead);
Reasonable C# wrapper for this function:
internal static byte[] GetDataToBuffer()
{
// set BufferSize to your most common data length
const int BufferSize = 1024 * 8;
// list of data blocks
var chunks = new List<byte[]>();
int dataRead = 1;
int position = 0;
int totalBytes = 0;
while(true)
{
var chunk = new byte[BufferSize];
// get new block of data
DTS_GetDataToBuffer(position, BufferSize, chunk, ref dataRead);
position += BufferSize;
if(dataRead != 0)
{
totalBytes += dataRead;
// append data block
chunks.Add(chunk);
if(dataRead < BufferSize)
{
break;
}
}
else
{
break;
}
}
switch(chunks.Count)
{
case 0: // no data blocks read - return empty array
return new byte[0];
case 1: // single data block
if(totalBytes < BufferSize)
{
// truncate data block to actual data size
var data = new byte[totalBytes];
Array.Copy(chunks[0], data, totalBytes);
return data;
}
else // single data block with size of Exactly BufferSize
{
return chunks[0];
}
default: // multiple data blocks
{
// construct new array and copy all data blocks to it
var data = new byte[totalBytes];
position = 0;
for(int i = 0; i < chunks.Count; ++i)
{
// copy data block
Array.Copy(chunks[i], 0, data, position, Math.Min(totalBytes, BufferSize));
position += BufferSize;
// we need to handle last data block correctly,
// it might be shorted than BufferSize
totalBytes -= BufferSize;
}
return data;
}
}
}
I can't test this but I think you should let the Marshaler do you conversion(s):
[DllImport(dll_Path)]
public static extern int DTS_GetDataToBuffer(out byte[] data, out int dataRead);
i agree you don't need to use unsafe block. you are using pinvoke, i hope below links might be useful :
http://msdn.microsoft.com/en-us/magazine/cc164123.aspx
http://www.pinvoke.net/
and there are post on stackoverflow too