Develop Reference

How do I get a byte array from an ArraySegment

If someone passes me an ArraySegment<byte> foo and this segment points into a larger buffer, what's the idiomatic way to copy this segment into a fresh new byte[] ?
I tried accessing at foo.Array but this seems to point to the beginning of the larger buffer, not the beginning of the segment.
e.g. the larger buffer could be "blahfoobar" and the ArraySegment points to "foo". I want to get a byte[] with just "foo".
I'm sure it's dead simple, but coming from C++, I can't figure the lingo used in c#.

Creating a new array would be entirely to miss the point of the API, which is to represent a pre-existing segment. In more recent .NET version, Span<T> and ReadOnlySpan<T> would be better choices - they allow you to create an abstraction over contiguous memory without needing the consumer to worry about the Offset etc, as that can be imposed externally. There are constructors on Span<T> and ReadOnlySpan<T> that allow you to deal with aspects so that the consumer never needs to know about them, with the knowledge that on recent runtimes: the JIT will elide bounds checks on spans.

using System;
using System.Text;
namespace ConsoleApp1
{
class Program
{
private static readonly byte[] _initArray = Encoding.ASCII.GetBytes("blahfoobar");
//private static byte[] ArraySegmentToArray(ArraySegment<byte> segment)
//{
// var result = new byte[segment.Count];
// for (int i = 0; i < segment.Count; i++)
// {
// result[i] = segment.Array[i + segment.Offset];
// }
// return result;
//}
private static byte[] ArraySegmentToArray(ArraySegment<byte> segment) =>
segment.ToArray();
static void Main(string[] args)
{
var segFoo = new ArraySegment<byte>(_initArray, 4, 3);
var test = ArraySegmentToArray(segFoo);
}
}
}
But of course it is bad practice. Because if you turn segment to array you allocate memory, and if you use ArraySegment stuff as pointers , you don't allocate memory, actually, that is the main idea of using ArraySegment, because if not the array could be provided as parameters :)
P.S. Commented code just for understanding the idea.

C# Struct Layouts and unexpected bench marking result?

I'm not really micro-managing the performance of an application, but I'm curios on the below scenario.
For Structs, by default, C# compiler generates the layout, LayoutType. Sequential. This means the fields should stay in the order defined by the programmer. I believe that this is to support interoperability with unmanaged code. However most user defined Structs have nothing to do with interoperability. I have read that for better performance, we can explicitly specify the LayoutKind.Auto, and let the CLR to decide the best possible layout. In order to test this, I thought of doing a quick benchmark on both layouts. However my result says the default layout (LayoutType.Sequnetial) is bit quicker than the explicit layout (LayoutType.Auto). I was expecting the reverse.
Below is the test I ran on my machine (x86 running .NET 4)
//uses LayoutKind.Sequence by default
public struct StructSeq
{
private readonly Byte mb;
private readonly Int16 mx;
public string a;
public string b;
public string c;
public string d;
}
[StructLayout(LayoutKind.Auto)]
public struct StructAuto
{
private readonly Byte mb;
private readonly Int16 mx;
public string a;
public string b;
public string c;
public string d;
}
public sealed class Program
{
public static void Main()
{
StructSeq sq = new StructSeq();
Stopwatch sw1 = new Stopwatch();
sw1.Start();
for (int i = 0; i < 10000; i++)
{
sq = ProcessStructSeq(sq);
}
sw1.Stop();
Console.WriteLine("Struct LayoutKind.Sequence (default) {0}", sw1.Elapsed.TotalMilliseconds);
StructAuto so = new StructAuto();
Stopwatch sw2 = new Stopwatch();
sw2.Start();
for (int i = 0; i < 10000; i++)
{
so = ProcessStructAuto(so);
}
sw2.Stop();
Console.WriteLine("Struct LayoutKind.Auto (explicit) {0}", sw2.Elapsed.TotalMilliseconds);
Console.ReadLine();
}
public static StructSeq ProcessStructSeq(StructSeq structSeq)
{
structSeq.a = "1";
structSeq.b = "2";
structSeq.c = "3";
structSeq.d = "4";
return structSeq;
}
public static StructAuto ProcessStructAuto(StructAuto structAuto)
{
structAuto.a = "1";
structAuto.b = "2";
structAuto.c = "3";
structAuto.d = "4";
return structAuto;
}
}
Below is a sample result I get on my machine (x86 running .NET 4)
Struct LayoutKind.Sequence (default) 0.7488
Struct LayoutKind.Auto (explicit) 0.7643
I ran this test multiple times and I always get Struct LayoutKind.Sequence (default) < Struct LayoutKind.Auto (explicit)
Even though it is a micro milliseconds difference, I ‘m expecting the Struct LayoutKind.Auto (explicit) to be lower than the Struct LayoutKind.Sequence (default).
Does anyone know the reason for this? Or is it my benchmarking is not accurate enough give me the right result?

I have tested your code on my system, and found that the average time taken is the same when the test is run a large number of times, with each test run slightly favoring one or the other alternative. This applies both to debug and release builds.
Also, as a quick check, I looked at the x86 code in the debugger, and I see no difference in the generated code whatsoever. So with your program as it is, the difference you observed in your measurements essentially seems to be noise.

Honestly, it's so close that it wouldn't make any sort of visible difference unless you were processing a few million of these structs. In fact, running it multiple times may yield different results. I would up the number of iterations and try to run the program without the debugger attached to see if anything changes.
Just using structs doesn't immediately make your code faster though, there are many pitfalls that make structs far slower than their class equivalents.
If you want to optimize this benchmark, you should pass the structs to the process methods as references and not return another struct (avoiding the creation of 2 additional structs for the method), which should provide a much larger speedup than the different layout kinds:
public static void ProcessStructSeq(ref StructSeq structSeq)
{
structSeq.a = "1";
structSeq.b = "2";
structSeq.c = "3";
structSeq.d = "4";
}
public static void ProcessStructAuto(ref StructAuto structAuto)
{
structAuto.a = "1";
structAuto.b = "2";
structAuto.c = "3";
structAuto.d = "4";
}
Also, there's a point where structs become slower than their class counterparts, and that's estimated to be at about 16 bytes according to this MSDN article and further explained in this StackOverflow question.

I believe there is no difference due to how your fields are laid out. The way you declared them, the padding will be the same either way. If you try interlacing the fields of different sizes, you should see a difference, at least in size, if not in speed.
Also, according to this blog post, a struct with a reference field is changed to auto layout (meaning you were benching literally the exact same thing!).
public struct MyStruct
{
private byte b1;
public long a;
private byte b2;
public long b;
private byte b3;
public long c;
private byte b4;
public long d;
}

Reading Serialized MFC CArray in C#

MFC CArray was Serialized and saved to a database. I need to read this data into a C# project. I am able to retrieve the data as byte[] from the database. I then write the byte[] to a MemoryStream. Now I need to read the data from the MemoryStream.
Someone has apparently solved this before, but did not write their solution.
http://social.msdn.microsoft.com/Forums/eu/csharpgeneral/thread/17393adc-1f1e-4e12-8975-527f42e5393e
I followed these projects in my attempt to solve the problem.
http://www.codeproject.com/Articles/32741/Implementing-MFC-Style-Serialization-in-NET-Part-1
http://www.codeproject.com/Articles/32742/Implementing-MFC-Style-Serialization-in-NET-Part-2
The first thing in the byte[] is the size of the array, and I can retrieve that with binaryReader.readInt32(). However, I cannot seem to get back the float values. If I try binaryReader.readSingle() or
public void Read(out float d) {
byte[] bytes = new byte[4];
reader.Read(bytes, m_Index, 4);
d = BitConverter.ToSingle(bytes, 0);
}
I do not get back the correct data. What am I missing?
EDIT Here is the C++ code that serializes the data
typedef CArray<float, float> FloatArray;
FloatArray floatArray;
// fill floatArray
CSharedFile memoryFile(GMEM_MOVEABLE | GMEM_ZEROINIT);
CArchive ar(&memoryFile, CArchive::store);
floatArray.Serialize(ar);
ar.Close();
EDIT 2
By reading backward, I was able to get all of the floats, and was also able to determine that the size for CArray is byte[2], or Int16. Does anyone know if this is always the case?

Using the codeproject articles above, here is a C# implementation of CArray which will allow you to deserialize a serialized MFC CArray.
// Deriving from the IMfcArchiveSerialization interface is not mandatory
public class CArray : IMfcArchiveSerialization {
public Int16 size;
public List<float> floatValues;
public CArray() {
floatValues = new List<float>();
}
virtual public void Serialize(MfcArchive ar) {
if(ar.IsStoring()) {
throw new NotImplementedException("MfcArchive can't store");
}
else {
// be sure to read in the order in which they were stored
ar.Read(out size);
for(int i = 0; i < size; i++) {
float floatValue;
ar.Read(out floatValue);
floatValues.Add(floatValue);
}
}
}
}

Having trouble with extension methods for byte arrays

I'm working with a device that sends back an image, and when I request an image, there is some undocumented information that comes before the image data. I was only able to realize this by looking through the binary data and identifying the image header information inside.
I originally had a normal method and converted it to an extension method. The original question here was related to the compiler complaining about not having Array as the first parameter (I had Byte[]), but it turns out that I had made an error and forgot to delete the first argument in the calling code. In other words, I used to have:
Byte[] new_buffer = RemoveUpToByteArray(buffer, new byte[] { 0x42, 0x4D });
and after changing to an extension method, I had erroneously used:
buffer.RemoveUpToByteArray( buffer, new byte[] { 0x42, 0x4D });
Anyhow, that's all fixed now because I realized my mistake as I was entering the code example into SO. However, I have a new problem that is simply lack of understanding of extension methods and reference vs. value types. Here's the code:
public static void RemoveFromByteArrayUntil(this Byte[] array, Byte[] until)
{
Debug.Assert(until.Count() > 0);
int num_header_bytes = until.Count();
int header_start_pos = 0; // the position of the header bytes, defined by [until]
byte first_header_byte = until[0];
while(header_start_pos != -1) {
header_start_pos = Array.IndexOf(array, first_header_byte, header_start_pos);
if(header_start_pos == -1)
break;
// if we get here, then we've found the first header byte, and we need to look
// for the next ones sequentially
for(int header_ctr=1; header_ctr<num_header_bytes; header_ctr++) {
// we're going to loop over each of the header bytes, but will
// bail out of this loop if there isn't a match
if(array[header_start_pos + header_ctr] != until[header_ctr]) {
// no match, so bail out. but before doing that, advance
// header_start_pos so the outer loop won't find the same
// occurrence of the first header byte over and over again
header_start_pos++;
break;
}
}
// if we get here, we've found the header!
// create a new byte array of the new size
int new_size = array.Count() - header_start_pos;
byte[] output_array = new byte[new_size];
Array.Copy(array, header_start_pos, output_array, 0, new_size);
// here is my problem -- I want to change what array points to, but
// when this code returns, array goes back to its original value, which
// leads me to believe that the first argument is passed by value.
array = output_array;
return;
}
// if we get here, we didn't find a header, so throw an exception
throw new HeaderNotInByteArrayException();
}
My problem now is that it looks like the first this argument to the extension method is passed by value. I want to reassign what array points to, but in this case, it looks like I'll have to just manipulate array's data instead.

Extension methods are static methods that only appear to be instance methods. You can consider the instance the extension method is working on to be read only (by value). Assigning to the instance method of byte[] that is the first parameter of your extension won't work. You won't be able to get away from assigning, but you could modify your extension then write your assignment like this:
buffer = buffer.RemoveUpToByteArray(header);
Make your extension return the byte array result, and don't try to assign to buffer within the extension. Your extension would then be something like this:
public static class MyExtensionMethods
{
public static byte[] RemoveUpToByteArray(this byte[] buffer, byte[] header)
{
byte[] result = buffer;
// your logic to remove header from result
return result;
}
}
I hope this helps.
EDIT:
The above is correct for value types only. If the type you are extending is a reference type, then you would not have an issue operating directly on the type like you are trying to do above. Sadly, a byte array is a struct, and thus derived from System.ValueType. Consider the following, which would be perfectly legal inside an extension, and would give the desired result:
public class MyBytes
{
public byte[] ByteArray { get; set; }
}
public static class MyExtensionMethods
{
// Notice the void return here...
public static void MyClassExtension(this MyBytes buffer, byte[] header)
{
buffer.ByteArray = header;
}
}

I am sorry that I do not know what specific problem you are encountering, or how to resolve it [certainly checking namespace is referenced and resolving any conflicts with similarly named methods is a start], but I did notice one or two oddities.
Consider the following sample solution,
using System.Linq;
namespace Sample.Extensions
{
public static class ByteExtensions
{
public static void RemoveHeader (this byte[] buffer, byte[] header)
{
// take first sequence of bytes, compare to header, if header
// is present, return only content
//
// NOTE: Take, SequenceEqual, and Skip are standard Linq extensions
if (buffer.Take (header.Length).SequenceEqual (header))
{
buffer = buffer.Skip (header.Length).ToArray ();
}
}
}
}
This compiles and runs in VS2010RC. To demonstrate usage,
using Sample.Extensions;
namespace Sample
{
class Program
{
static void Main (string[] args)
{
byte[] buffer = new byte[] { 00, 01, 02 };
byte[] header = new byte[] { 00, 01 };
buffer.RemoveHeader (header);
// hm, so everything compiles and runs, but buffer == { 00, 01, 02 }
}
}
}
So we will not receive a compile or run-time error but clearly it will not operate as intended. This is because extensions must still comply with standard method semantics, meaning parameters are passed by value. We cannot change buffer to point to our new array.
We can resolve this issue by rewriting our method to conventional function semantics,
public static byte[] RemoveHeaderFunction (this byte[] buffer, byte[] header)
{
byte[] stripped = null;
if (stripped.Take (header.Length).SequenceEqual (header))
{
stripped = stripped.Skip (header.Length).ToArray ();
}
else
{
stripped = buffer.ToArray ();
}
return stripped;
}
Now
using Sample.Extensions;
namespace Sample
{
class Program
{
static void Main (string[] args)
{
byte[] buffer = new byte[] { 00, 01, 02 };
byte[] header = new byte[] { 00, 01 };
// old way, buffer will still contain { 00, 01, 02 }
buffer.RemoveHeader (header);
// new way! as a function, we obtain new array of { 02 }
byte[] stripped = buffer.RemoveHeaderFunction (header);
}
}
}
Unfortunately, arrays are immutable value types [may be using these terms incorrectly]. The only way to modify your "array" in-place is to change the container to a mutable reference-type, like a List<byte>.
If you are really keen on "passing by ref", in-place, side-effect semantics, then one option may be the following
using System.Linq;
namespace Sample.Extensions
{
public static class ListExtensions
{
public static void RemoveHeader<T> (this List<T> list, List<T> header)
{
if (list.Take (header.Count).SequenceEqual (header))
{
list.RemoveRange (0, header.Count);
}
}
}
}
As for usage,
static void Main (string[] args)
{
byte[] buffer = new byte[] { 00, 01, 02 };
byte[] header = new byte[] { 00, 01 };
List<byte> bufferList = buffer.ToList ();
// in-place side-effect header removal
bufferList.RemoveHeader (header.ToList ());
}
Under the hood, List<T> is maintaining an array of type T. At certain thresholds, it is simply manipulating the underlying array and\or instantiating new arrays for us.
Hope this helps! :)

What is the equivalent of "ByteBuffer.flip" & "ByteBuffer.slice" in .NET?

I need to port code from Java to C#. In the Java code, the methods "ByteBuffer.flip()" and "ByteBuffer.slice" is used, and I don't know how to translate this.
I've read this question (An equivalent of javax.nio.Buffer.flip() in c#), but although an answer is given, I cannot figure how to apply it. According to Tom Hawtin, I should "Set the limit to the current position and then set the position to zero" in the underlying array. I am unsure as of how to change these values. (If you could explain the underlying logic, it would help me a lot :)
As for the ByteBuffer.slice, I have no clue on how to translate it.
EDIT: If it can be clearer with the actual code, I'll post it:
Java:
ByteBuffer buff;
buff.putShort((short) 0);
buff.put(customArray);
buff.flip();
buff.putShort((short) 0);
ByteBuffer b = buff.slice();
short size = (short) (customFunction(b) + 2);
buff.putShort(0, size);
buff.position(0).limit(size);
So far, my translation in C#.NET:
BinaryWriter b = new BinaryWriter(); //ByteBuffer buff;
b.Write((short)0); // buff.putShort((short) 0);
b.Write(paramStream.ToArray()); // buff.put(customArray);
b.BaseStream.SetLength(b.BaseStream.Position); // buff.flip; (not sure)
b.BaseStream.Position = 0; // buff.flip; too (not sure)
b.Write((short)0); // buff.putShort((short) 0)
??? // ByteBuffer b = buff.slice();
// Not done but I can do it, short size = (short) (customFunction(b) + 2);
??? // How do I write at a particular position?
??? // buff.position(0).limit(size); I don't know how to do this
Thank you!
EDIT: Changed b.BaseStream.SetLength(b.BaseStream.Length); to b.BaseStream.SetLength(b.BaseStream.Position);, based on the Java docs.

(See See http://java.sun.com/javase/6/docs/api/java/nio/ByteBuffer.html#slice%28%29 and http://java.sun.com/javase/6/docs/api/java/nio/Buffer.html#flip%28%29 for java's calls)
Flip is a quick way to reset the buffer. So for example
(pseudocode)
void flip()
{
Length = currentPos;
currentPos = 0;
}
Allows you to quickly setup the buffer you presumably just wrote to for reading from the beginning.
Update:
Splice is a bit trickier due to the requirement that "Changes to this buffer's content will be visible in the new buffer, and vice versa; the two buffers' position, limit, and mark values will be independent". There unfortunately is no concept of a shared portion of buffer (that i know of - theres always using arrays, detailed below) without making your own class. The closest thing you could do is this:
Old Code:
ByteBuffer b = buff.slice();
New Code (assuming a List)
List<Byte> b= buff;
int bStart = buffPos; // buffPos is your way of tracking your mark
the downside to the code above is that there is no way for c# to hold the new starting point of the new buffer and still share it. You'll have to manually use the new starting point whenever you do anything, from for loops (for i=bStart;...) to indexing (newList[i + bStart]...)
Your other option is to do use Byte[] arrays instead, and do something like this:
Byte[] b = &buff[buffPos];
... however that requires unsafe operations to be enabled, and I cannot vouch for its saftey, due to the garbage collector and my avoidance of the "unsafe" features.
Outside of that, theres always making your own ByteBuffer class.

Untested, but if I understand the java bits correctly, this would give you an idea on how to implement.
public class ByteBuffer {
private int _Position;
private int _Capacity;
private byte[] _Buffer;
private int _Start;
private ByteBuffer(int capacity, int position, int start, byte[] buffer) {
_Capacity = capacity;
_Position = position;
_Start = start;
_Buffer = buffer;
}
public ByteBuffer(int capacity) : this(capacity, 0 , 0, new byte[capacity]) {
}
public void Write(byte item) {
if (_Position >= _Capacity) {
throw new InvalidOperationException();
}
_Buffer[_Start + _Position++] = item;
}
public byte Read() {
if (_Position >= _Capacity) {
throw new InvalidOperationException();
}
return _Buffer[_Start + _Position++];
}
public void Flip() {
_Capacity = _Position;
_Position = _Start;
}
public ByteBuffer Slice() {
return new ByteBuffer(_Capacity-_Position, 0, _Position, _Buffer);
}
}