Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 4 years ago.
Improve this question
Basically, why does this work?
System.IO.Stream stream = new MemoryStream();
int a = 4;
byte[] barray = new byte[a];
stream.Write(barray, 0, Marshal.SizeOf(a));
When this doesn't:
System.IO.Stream stream = new MemoryStream();
int a = 3;
byte[] barray = new byte[a];
stream.Write(barray, 0, Marshal.SizeOf(a));
This is the error I get:
The offset and length were greater than allowed for the matrix, or the number is greater than the number of elements from the index to the end of the source collection.
When using Marshel.SizeOf(a) you ask the size of the object in memory. Since a is an int the size is always 4.
When you say byte[] barray = new byte[a]; you say:
Create an array called barray of type byte with length a. Thus in the first code block you create an array of length 4 and in the second one you create an array of length 3. Both array's contain only zero's.
Then you say: write the (empty) array to the stream, starting at position 0 and with length 4 (Marshel.SizeOf(a) is always 4 because a is an int).
The first example array has a length of 4 and thus works. The second example only contains 3 bytes and thus the length is not correct and you get an error.
If you wish to save the int to the stream as bytes explicitly you could call BitConverter:
System.IO.Stream stream = new MemoryStream();
int a = 4;
byte[] barray = System.BitConverter.GetBytes(a);
stream.Write(barray, 0, Marshal.SizeOf(a));
Now you say: create an array called barray that is filled with the binary representation of integer variable a.
And then write that filled array to the stream.
Related
Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 1 year ago.
Improve this question
In julia I am able to read the first 4 bytes from the file using this code. (It is the length of the message to read)
# Read the length in first 4 bytes
msglen = read(R.buffer, UInt32)
# Then read up to that length
bytes = read(R.buffer, msglen)
But when I try to read the same file in Rust the length value comes up way too big:
let mut f = std::fs::File::open("../20181002.bytes").unwrap();
let mut buf = Vec::new();
f.read_to_end(&mut buf).expect("file reading failed");
let mut dst = [0u8; 4];
let mut read_index = 0usize;
// select first 4 bytes of buf and clone into dst
dst.clone_from_slice(&buf[read_index..(read_index+4)]);
println!("Dst: {:?}", &buf[read_index..(read_index+4)]);
let length = u32::from_le_bytes(dst);
println!("Len: {}", length);
Dst: [31, 139, 8, 0]
Len: 559903
I think the first 4 bytes should be 1f8b 0800 0000 0000
If its of help, here is how its written in C#:
public static void Write(Stream stream, byte[] bytes)
{
var lengthBuffer = BitConverter.GetBytes(bytes.Length);
// Write object length.
stream.Write(lengthBuffer, offset: 0, count: 4);
// Write object.
stream.Write(bytes, offset: 0, count: bytes.Length);
}
1f8b is the magic number for gzip encoded files. Sorry to waste your time - leaving the code up in case it interests anyone.
Q: Is there any benefit of storing the length of a large array within the array itself?
Explanation:
Let's say we compress some large binary serialized object by using the GZipStream class of the System.IO.Compression namespace.
The output will be a Base64 string of some compressed byte array.
At some later point the Base64 string gets converted back to a byte array and the data needs to be decompressed.
While compressing the data we create a new byte array with the size of the compressed byte array + 4.
In the first 4 bytes we store the length/size of the compressed byte array and we then BlockCopy the length and the data to the new array. This new array gets converted into a Base64 string.
While decompressing we convert the Base64 string into a byte array.
Now we can extract the length of the actual compressed data by using the BitConverter class which will extract a Int32 from the first 4 bytes.
We then allocate a byte array with the length that we got from the first 4 bytes and let the Stream write the decompressed bytes to the byte array.
I can't image that something like this actually has any benefit at all.
It adds more complexity to the code and more operations need to be executed.
Readability is reduced too.
The BlockCopy operations alone should consume so much resources that this just cannot have a benefit, right?
Compression example code:
byte[] buffer = new byte[0xffff] // Some large binary serialized object
// Compress in-memory.
using (var mem = new MemoryStream())
{
// The actual compression takes place here.
using (var zipStream = new GZipStream(mem, CompressionMode.Compress, true)) {
zipStream.Write(buffer, 0, buffer.Length);
}
// Store compressed byte data here.
var compressedData = new byte[mem.Length];
mem.Position = 0;
mem.Read(compressedData, 0, compressedData.Length);
/* Increase the size by 4 to accommadate for an Int32 that
** will store the total length of the compressed data. */
var zipBuffer = new byte[compressedData.Length + 4];
// Store length of compressedData array in the first 4 bytes.
Buffer.BlockCopy(compressedData, 0, zipBuffer, 4, compressedData.Length);
// Store the compressedData array after the first 4 bytes which store the length.
Buffer.BlockCopy(BitConverter.GetBytes(buffer.Length), 0, zipBuffer, 0, 4);
return Convert.ToBase64String(zipBuffer);
}
Decompression example code:
byte[] zipBuffer = Convert.FromBase64String("some base64 string");
using (var inStream = new MemoryStream())
{
// The length of the array that was stored in the first 4 bytes.
int dataLength = BitConverter.ToInt32(zipBuffer, 0);
// Allocate array with specific size.
byte[] buffer = new byte[dataLength];
// Writer data to buffer array.
inStream.Write(zipBuffer, 4, zipBuffer.Length - 4);
inStream.Position = 0;
// Decompress data.
using (var zipStream = new GZipStream(inStream, CompressionMode.Decompress)) {
zipStream.Read(buffer, 0, buffer.Length);
}
... code
... code
... code
}
You tagged the question as C#, wich means .NET, so the question is irrelevant:
The Framework already store the length with the Array. It is how the array classes do the sanity checks on Indexers. It how it prevents overflow attacks in managed code. That help alone is worth any minor inefficiency (note that the JiT is actually able to prune most of the checks. With a loop for example, it will simply look at the running variable once per loop).
You would have to go all the way into unmanaged code and handling naked pointers to have a hope to get rid of it. But why would you? The difference is so small, it falls under the speed rant. If it maters, you propably got a realtime programming case. And starting those with .NET was a bad idea.
I am trying to write an Encoded file.The file has 9 to 12 bit symbols. While writing a file I guess that it is not written correctly the 9 bit symbols because I am unable to decode that file. Although when file has only 8 bit symbols in it. Everything works fine. This is the way I am writing a file
File.AppendAllText(outputFileName, WriteBackContent, ASCIIEncoding.Default);
Same goes for reading with ReadAllText function call.
What is the way to go here?
I am using ZXing library to encode my file using RS encoder.
ReedSolomonEncoder enc = new ReedSolomonEncoder(GenericGF.AZTEC_DATA_12);//if i use AZTEC_DATA_8 it works fine beacuse symbol size is 8 bit
int[] bytesAsInts = Array.ConvertAll(toBytes.ToArray(), c => (int)c);
enc.encode(bytesAsInts, parity);
byte[] bytes = bytesAsInts.Select(x => (byte)x).ToArray();
string contentWithParity = (ASCIIEncoding.Default.GetString(bytes.ToArray()));
WriteBackContent += contentWithParity;
File.AppendAllText(outputFileName, WriteBackContent, ASCIIEncoding.Default);
Like in the code I am initializing my Encoder with AZTEC_DATA_12 which means 12 bit symbol. Because RS Encoder requires int array so I am converting it to int array. And writing to file like here.But it works well with AZTEC_DATA_8 beacue of 8 bit symbol but not with AZTEC_DATA_12.
Main problem is here:
byte[] bytes = bytesAsInts.Select(x => (byte)x).ToArray();
You are basically throwing away part of the result when converting the single integers to single bytes.
If you look at the array after the call to encode(), you can see that some of the array elements have a value higher than 255, so they cannot be represented as bytes. However, in your code quoted above, you cast every single element in the integer array to byte, changing the element when it has a value greater than 255.
So to store the result of encode(), you have to convert the integer array to a byte array in a way that the values are not lost or modified.
In order to make this kind of conversion between byte arrays and integer arrays, you can use the function Buffer.BlockCopy(). An example on how to use this function is in this answer.
Use the samples from the answer and the one from the comment to the answer for both conversions: Turning a byte array to an integer array to pass to the encode() function and to turn the integer array returned from the encode() function back into a byte array.
Here are the sample codes from the linked answer:
// Convert byte array to integer array
byte[] result = new byte[intArray.Length * sizeof(int)];
Buffer.BlockCopy(intArray, 0, result, 0, result.Length);
// Convert integer array to byte array (with bugs fixed)
int bytesCount = byteArray.Length;
int intsCount = bytesCount / sizeof(int);
if (bytesCount % sizeof(int) != 0) intsCount++;
int[] result = new int[intsCount];
Buffer.BlockCopy(byteArray, 0, result, 0, byteArray.Length);
Now about storing the data into files: Do not turn the data into a string directly via Encoding.GetString(). Not all bit sequences are valid representations of characters in any given character set. So, converting a random sequence of random bytes into a string will sometimes fail.
Instead, either store/read the byte array directly into a file via File.WriteAllBytes() / File.ReadAllBytes() or use Convert.ToBase64() and Convert.FromBase64() to work with a base64 encoded string representation of the byte array.
Combined here is some sample code:
ReedSolomonEncoder enc = new ReedSolomonEncoder(GenericGF.AZTEC_DATA_12);//if i use AZTEC_DATA_8 it works fine beacuse symbol size is 8 bit
int[] bytesAsInts = Array.ConvertAll(toBytes.ToArray(), c => (int)c);
enc.encode(bytesAsInts, parity);
// Turn int array to byte array without loosing value
byte[] bytes = new byte[bytesAsInts.Length * sizeof(int)];
Buffer.BlockCopy(bytesAsInts, 0, bytes, 0, bytes.Length);
// Write to file
File.WriteAllBytes(outputFileName, bytes);
// Read from file
bytes = File.ReadAllBytes(outputFileName);
// Turn byte array to int array
int bytesCount = bytes.Length * 40;
int intsCount = bytesCount / sizeof(int);
if (bytesCount % sizeof(int) != 0) intsCount++;
int[] dataAsInts = new int[intsCount];
Buffer.BlockCopy(bytes, 0, dataAsInts, 0, bytes.Length);
// Decoding
ReedSolomonDecoder dec = new ReedSolomonDecoder(GenericGF.AZTEC_DATA_12);
dec.decode(dataAsInts, parity);
This question already has answers here:
c# how to add byte to byte array
(7 answers)
Closed 7 years ago.
I have a byte array which contains an image; I converted the image to byte array using this method:
public byte[] imageToByteArray(System.Drawing.Image imageIn)
{
MemoryStream ms = new MemoryStream();
imageIn.Save(ms,System.Drawing.Imaging.ImageFormat.Gif);
return ms.ToArray();
}
This is the byte array:
Bitmap bmp = Image.FromFile("xxx");
byte[] buffer = ImageToByteArray(bmp);
Now I would like to add some minor minor information about the image in the byte array,such as the position it should be drawn to,etc.
How could it be done? Lets say i want to add these 2 values:1209,540.
First I don't think you can use these values 1209,540 since byte's max val is 255.
But if your values are between that range 0-255, why don't you add two bytes at the end of the array and when the conversion is to be made remove the last two values?
in the example below how do i get the length number using PrefixStyle with ProtoBuf-net?
and what's the difference between PrefixStyle.Base128 and PrefixStyle.Fixed32?
Thanks!
PerfTest clone;
using (MemoryStream ms = new MemoryStream())
{
Serializer.SerializeWithLengthPrefix(ms, obj,PrefixStyle.Base128);
byte[] raw = ms.ToArray();
ms.Position = 0;
clone = Serializer.DeserializeWithLengthPrefix<PerfTest>(ms,PrefixStyle.Base128);
}
Edit: Using the code below the byte array has a length of 22. Why does TryReadLengthPrefix return 21? Surely is should return 22?
PerfTest clone;
using (MemoryStream ms = new MemoryStream())
{
Serializer.SerializeWithLengthPrefix(ms, obj,PrefixStyle.Base128);
byte[] raw = ms.ToArray();
ms.Position = 0;
int bArrayLen = ms.ToArray().Length; //returns 22
int len;// set to 21. Why not 22?
Serializer.TryReadLengthPrefix(ms, PrefixStyle.Base128,out len);
clone = Serializer.DeserializeWithLengthPrefix<PerfTest>(ms,PrefixStyle.Fixed32);
}
Fixed32 always uses 4 bytes for the length prefix - this seems to be not uncommon for people packing messages manually (indeed, I even had to add different-endian versions at some point, due to repeat requests).
The preference though should be Base128, which uses "varint" encoding, so small numbers take less space to encode. Additionally, this prefix style can be used to make a sequence of objects wire-compatible with a single object with a repeated field, which has some useful applications.
Re getting the length; if you are using DeserializeWithLenghPrefix or DeserializeItems you shouldnt need to - it handles it internally. But if you need this, look at Serializer.TryReadLengthPrefix.
To clarify: the purpose of the *WithLengthPrefix methods is to allow separation of different objects/messages in a single stream - most commonly: network sockets. This is necessary because the protocol itself otherwise assumes an entire stream is a single message, so would keep trying to read until the stream/socket was closed.
Re the 22 vs 21 bytes; that is because the length prefix itself takes some length :) actually if the array is 22 I'm a little surprised the payload isn't 20 - I byte field headed (to make the composed stream itself a valid protobuf stream), 1 byte length, and 20 bytes payload. I'm on a mobile at the moment, so I can't run the sample to investigate; it is possible to omit the field-header (optionally), though - so it could be 1 byte length, 21 bytes payload. I'll look later.