Related
I want to split a large array of UTF-8 encoded data, so that decoding it into chars can be parallelized.
It seems that there's no way to find out how many bytes Encoding.GetCharCount reads. I also can't use GetByteCount(GetChars(...)) since it decodes the entire array anyways, which is what I'm trying to avoid.
UTF-8 has well-defined byte sequences and is considered self-synchronizing, meaning given any position in bytes you can find where the character at that position begins at.
The UTF-8 spec (Wikipedia is the easiest link) defines the following byte sequences:
0_______ : ASCII (0-127) char
10______ : Continuation
110_____ : Two-byte character
1110____ : Three-byte character
11110___ : Four-byte character
So, the following method (or something similar) should get your result:
Get the byte count for bytes (bytes.Length, et. al.)
Determine how many sections to split into
Select byte byteCount / sectionCount
Test byte against table:
If byte & 0x80 == 0x00 then you can make this byte part of either section
If byte & 0xE0 == 0xC0 then you need to seek ahead one byte, and keep it with the current section
If byte & 0xF0 == 0xE0 then you need to seek ahead two bytes, and keep it with the current section
If byte & 0xF8 == 0xF0 then you need to seek ahead three bytes, and keep it with the current section
If byte & 0xC0 == 0x80 then you are in a continuation, and should seek ahead until the first byte that does not fit val & 0xB0 == 0x80, then keep up to (but not including) this value in the current section
Select byteStart through byteCount + offset where offset can be defined by the test above
Repeat for each section.
Of course, if we redefine our test as returning the current char start position, we have two cases:
1. If (byte[i] & 0xC0) == 0x80 then we need to move around the array
2. Else, return the current i (since it's not a continuation)
This gives us the following method:
public static int GetCharStart(ref byte[] arr, int index) =>
(arr[index] & 0xC0) == 0x80 ? GetCharStart(ref arr, index - 1) : index;
Next, we want to get each section. The easiest way is to use a state-machine (or abuse, depending on how you look at it) to return the sections:
public static IEnumerable<byte[]> GetByteSections(byte[] utf8Array, int sectionCount)
{
var sectionStart = 0;
var sectionEnd = 0;
for (var i = 0; i < sectionCount; i++)
{
sectionEnd = i == (sectionCount - 1) ? utf8Array.Length : GetCharStart(ref utf8Array, (int)Math.Round((double)utf8Array.Length / sectionCount * i));
yield return GetSection(ref utf8Array, sectionStart, sectionEnd);
sectionStart = sectionEnd;
}
}
Now I built this in this manner because I want to use Parallel.ForEach to demonstrate the result, which makes it super easy if we have an IEnumerable, and it also allows me to be extremely lazy with the processing: we only continue to gather sections when needed, which means we can lazily process it and do it on-demand, which is a good thing, no?
Lastly, we need to be able to get a section of bytes, so we have the GetSection method:
public static byte[] GetSection(ref byte[] array, int start, int end)
{
var result = new byte[end - start];
for (var i = 0; i < result.Length; i++)
{
result[i] = array[i + start];
}
return result;
}
Finally, the demonstration:
var sourceText = "Some test 平仮名, ひらがな string that should be decoded in parallel, this demonstrates that we work flawlessly with Parallel.ForEach. The only downside to using `Parallel.ForEach` the way I demonstrate is that it doesn't take order into account, but oh-well.";
var source = Encoding.UTF8.GetBytes(sourceText);
Console.WriteLine(sourceText);
var results = new ConcurrentBag<string>();
Parallel.ForEach(GetByteSections(source, 10),
new ParallelOptions { MaxDegreeOfParallelism = 1 },
x => { Console.WriteLine(Encoding.UTF8.GetString(x)); results.Add(Encoding.UTF8.GetString(x)); });
Console.WriteLine();
Console.WriteLine("Assemble the result: ");
Console.WriteLine(string.Join("", results.Reverse()));
Console.ReadLine();
The result:
Some test ???, ???? string that should be decoded in parallel, this demonstrates that we work flawlessly with Parallel.ForEach. The only downside to using `Parallel.ForEach` the way I demonstrate is that it doesn't take order into account, but oh-well.
Some test ???, ??
?? string that should b
e decoded in parallel, thi
s demonstrates that we work
flawlessly with Parallel.
ForEach. The only downside
to using `Parallel.ForEach`
the way I demonstrate is
that it doesn't take order into account, but oh-well.
Assemble the result:
Some test ???, ???? string that should be decoded in parallel, this demonstrates that we work flawlessly with Parallel.ForEach. The only downside to using `Parallel.ForEach` the way I demonstrate is that it doesn't take order into account, but oh-well.
Not perfect, but it does the job. If we change MaxDegreesOfParallelism to a higher value, our string gets jumbled:
Some test ???, ??
e decoded in parallel, thi
flawlessly with Parallel.
?? string that should b
to using `Parallel.ForEach`
ForEach. The only downside
that it doesn't take order into account, but oh-well.
s demonstrates that we work
the way I demonstrate is
So, as you can see, super easy. You'll want to make modifications to allow for correct order-reassembly, but this should demonstrate the trick.
If we modify the GetByteSections method as follows, the last section is no longer ~2x the size of the remaining ones:
public static IEnumerable<byte[]> GetByteSections(byte[] utf8Array, int sectionCount)
{
var sectionStart = 0;
var sectionEnd = 0;
var sectionSize = (int)Math.Ceiling((double)utf8Array.Length / sectionCount);
for (var i = 0; i < sectionCount; i++)
{
if (i == (sectionCount - 1))
{
var lengthRem = utf8Array.Length - i * sectionSize;
sectionEnd = GetCharStart(ref utf8Array, i * sectionSize);
yield return GetSection(ref utf8Array, sectionStart, sectionEnd);
sectionStart = sectionEnd;
sectionEnd = utf8Array.Length;
yield return GetSection(ref utf8Array, sectionStart, sectionEnd);
}
else
{
sectionEnd = GetCharStart(ref utf8Array, i * sectionSize);
yield return GetSection(ref utf8Array, sectionStart, sectionEnd);
sectionStart = sectionEnd;
}
}
}
The result:
Some test ???, ???? string that should be decoded in parallel, this demonstrates that we work flawlessly with Parallel.ForEach. The only downside to using `Parallel.ForEach` the way I demonstrate is that it doesn't take order into account, but oh-well. We can continue to increase the length of this string to demonstrate that the last section is usually about double the size of the other sections, we could fix that if we really wanted to. In fact, with a small modification it does so, we just have to remember that we'll end up with `sectionCount + 1` results.
Some test ???, ???? string that should be de
coded in parallel, this demonstrates that we work flawless
ly with Parallel.ForEach. The only downside to using `Para
llel.ForEach` the way I demonstrate is that it doesn't tak
e order into account, but oh-well. We can continue to incr
ease the length of this string to demonstrate that the las
t section is usually about double the size of the other se
ctions, we could fix that if we really wanted to. In fact,
with a small modification it does so, we just have to rem
ember that we'll end up with `sectionCount + 1` results.
Assemble the result:
Some test ???, ???? string that should be decoded in parallel, this demonstrates that we work flawlessly with Parallel.ForEach. The only downside to using `Parallel.ForEach` the way I demonstrate is that it doesn't take order into account, but oh-well. We can continue to increase the length of this string to demonstrate that the last section is usually about double the size of the other sections, we could fix that if we really wanted to. In fact, with a small modification it does so, we just have to remember that we'll end up with `sectionCount + 1` results.
And finally, if for some reason you split into an abnormally large number of sections compared to input size (my input size of ~578 bytes at 250 chars demonstrates this) you'll hit an IndexOutOfRangeException in GetCharStart, the following version fixes that:
public static int GetCharStart(ref byte[] arr, int index)
{
if (index > arr.Length)
{
index = arr.Length - 1;
}
return (arr[index] & 0xC0) == 0x80 ? GetCharStart(ref arr, index - 1) : index;
}
Of course this leaves you with a bunch of empty results, but when you reassemble the string doesn't change, so I'm not even going to bother posting the full scenario test here. (I leave it up to you to experiment.)
Great answer Mathieu and Der, adding a python variant 100% based on your answer which works great:
def find_utf8_split(data, bytes=None):
bytes = bytes or len(data)
while bytes > 0 and data[bytes - 1] & 0xC0 == 0x80:
bytes -= 1
if bytes > 0:
if data[bytes - 1] & 0xE0 == 0xC0: bytes = bytes - 1
if data[bytes - 1] & 0xF0 == 0xE0: bytes = bytes - 1
if data[bytes - 1] & 0xF8 == 0xF0: bytes = bytes - 1
return bytes
This code finds a UTF-8 compatible split in a given byte string. It does not do the split as that would take more memory, that is left to the rest of the code.
For example you could:
position = find_utf8_split(data)
leftovers = data[position:]
text = data[:position].decode('utf-8')
I'm maintaining a program that has the following code to read a file to a byte array:
using (FileStream fileStream = new FileStream(filePath, FileMode.Open))
{
fileStream.Position = 0;
int fileSize = (int)fileStream.Length;
int readSize;
int remain = fileSize;
var pos = 0;
byteData = new byte[fileSize];
while (remain > 0)
{
readSize = fileStream.Read(byteData, pos, Math.Min(1024, remain));
pos += readSize;
remain -= readSize;
}
}
And then afterwards outputs this byte array as a Base64 string:
var value = "File contents:" + Environment.NewLine + Convert.ToBase64String(byteData)
The issue we are occasionally seeing is that the output is just a string of A's, like "AAAAAAAAAAAAAAAAAAAAAA", but longer. I've figured out that if you output a byte array that has been initialized to a given length but not assigned a value (i.e. each byte is still the initial value of 0) it will output in Base64 as a series of A's, so my hypothesis is that the byte array is being created to the size of the file, but then the value of each byte isn't being assigned. Looking at the code I can't see any obvious issues with it though, so if anyone knows better I'd be very grateful.
For posterity, I did end up changing it to File.ReadAllBytes, however I also found out that the issue was with the file itself, and an empty byte array was actually correct. I.e. each byte was still the initial value of 0, so a corresponding base64 string of "A"s was also correct.
I tried to create a function for print all the articles of a billing with some max length vars, but when this max length is exceeded Index out of range errors appears or in another cases total and quantity Doesn't appear in line:
Max Length variables:
private Int32 MaxCharPage = 36;
private Int32 MaxCharArticleName = 15;
private Int32 MaxCharArticleQuantity = 4;
private Int32 MaxCharArticleSellPrice = 6;
private Int32 MaxCharArticleTotal = 8;
private Int32 MaxCharArticleLineSpace = 1;
This is my current function:
private IEnumerable<String> ArticleLine(Double Quantity, String Name, Double SellPrice, Double Total)
{
String QuantityChunk = (String)(Quantity).ToString("#,##0.00");
String PriceChunk = (String)(SellPrice).ToString("#,##0.00");
String TotalChunk = (String)(Total).ToString("#,##0.00");
// full chunks with "size" length
for (int i = 0; i < Name.Length / this.MaxCharArticleName; ++i)
{
String Chunk = String.Empty;
if (i == 0)
{
Chunk = QuantityChunk + new String((Char)32, (MaxCharArticleQuantity + MaxCharArticleLineSpace - QuantityChunk.Length)) +
Name.Substring(i * this.MaxCharArticleName, this.MaxCharArticleName) +
new String((Char)32, (MaxCharArticleLineSpace)) +
new String((Char)32, (MaxCharArticleSellPrice - PriceChunk.Length)) +
PriceChunk +
new String((Char)32, (MaxCharArticleLineSpace)) +
new String((Char)32, (MaxCharArticleTotal - TotalChunk.Length)) +
TotalChunk;
}
else
{
Chunk = new String((Char)32, (MaxCharArticleQuantity + MaxCharArticleLineSpace)) +
Name.Substring(i * this.MaxCharArticleName, this.MaxCharArticleName);
}
yield return Chunk;
}
if (Name.Length % this.MaxCharArticleName > 0)
{
String chunk = Name.Substring(Name.Length / this.MaxCharArticleName * this.MaxCharArticleName);
yield return new String((Char)32, (MaxCharArticleQuantity + MaxCharArticleLineSpace)) + chunk;
}
}
private void AddArticle(Double Quantity, String Name, Double SellPrice, Double Total)
{
Lines.AddRange(ArticleLine(Quantity, Name, SellPrice, Total).ToList());
}
For example:
private List<String> Lines = new List<String>();
private void Form1_Load(object sender, EventArgs e)
{
AddArticle(2.50, "EGGS", 0.50, 1.25); // Problem: Dont show the numbers like (Quantity, Sellprice, Total)
//AddArticle(100.52, "HAND SOAP /w EP", 5.00, 502.60); //OutOfRangeException
AddArticle(105.6, "LONG NAME ARTICLE DESCRIPTION", 500.03, 100.00);
//AddArticle(100, "LONG NAME ARTICLE DESCRIPTION2", 1500.03, 150003.00); // OutOfRangeException
foreach (String line in Lines)
{
Console.WriteLine(line);
}
}
Console Output:
LINE1: EGGS
LINE2:105.6LONG NAME ARTIC 500.03 100.00
LINE3: LE DESCRIPTION
Desired output:
LINE:2.50 EGGS 0.50 1.25
LINE:100. HAND SOAP /w EP 5.00 502.60
LINE: 52
LINE:105. LONG NAME ARTIC 500.03 100.00
LINE: 60 LE DESCRIPTION
LINE:100. LONG NAME ARTIC 1,500. 150,003.
LINE: 00 LE DESCRIPTION2 03 00
You are attempting to take values (string values, ultimately) and extract specified lengths from them (chunks), where the each individual chunk of a given group goes on one line, and the next chunk(s) go on the next line(s). There are several challenges in your posted code. One of the biggest is that you may not understand what the / operator does. It is, of course, used for division, but it is integer division, meaning you get a whole number as the result. E.g. 3 / 15 gives you 0, whereas 17 / 15 would give you 1. This means your loop never runs unless the length of the value is greater than the specified chunk limit.
Another possible issue is that you only perform this check against Name, not the other items (though you may have omitted the code for them for brevity reasons).
Your current code is also creating a lot of unnecessary strings, which will lead to performance degradation. Remember that in C# strings are immutable - meaning they cannot be changed. When you "change" the value of a string, you are actually creating another copy of the string.
The crux of your requirement is how to "chunk" the values in such a way that the correct output is achieved. One way to do this is to create an extension method that will take a string value and "chunkify" it for you. One such example is found here: Split String Into Array of Chunks. I've modified it slightly to use a List<T>, but an array would work just as well.
public static List<string> SplitIntoChunks(this string toSplit, int chunkSize)
{
int stringLength = toSplit.Length;
int chunksRequired = (int)Math.Ceiling(decimal)stringLength / (decimal)chunkSize);
List<string> chunks = new List<string>();
int lengthRemaining = stringLength;
for (int i = 0; i < chunksRequired; i++)
{
int lengthToUse = Math.Min(lengthRemaining, chunkSize);
int startIndex = chunkSize * i;
chunks.Add(toSplit.Substring(startIndex, lengthToUse));
lengthRemaining = lenghtRemaining - lengthToUse;
}
return chunks;
}
Say you have a string named myString. You would use the above method like this: string[] chunks = myString.SplitIntoChunks(15);, and you would receive an array of 15 character strings (depending on the size of the string).
A quick walk-through of the code (as there is not much explanation on the page). The size of the chunk is passed into the extension method. The length of the string is recorded, and the number of chunks for that string is determined using the Math.Ceiling function.
Then a for loop is constructed with the number of chunks required as the limit. Inside the loop, the length of the chunk is determined (using the lower of either the chunk size or the remaining length of the string), the starting index is calculated based on the chunk size and the loop index, and then the chunk is extracted via Substring. Finally remaining length is calculated, and once the loop ends the chunks are returned.
One way to use this extension method in your code would look like this. The extension method will need to be in a separate, static class (I suggest building a library that has a class dedicated solely to extension methods, as they come in quite handy). Note that I haven't had time to test this, and it's a bit kludgy for my tastes, but it should at least get you going in the right direction.
private IEnumerable<string> ArticleLine(double quantity, string name, double sellPrice, double total)
{
List<string> quantityChunks = quantity.ToString("#,##0.00").SplitIntoChunks(maxCharArticleQuantity);
List<string> nameChunks = name.SplitIntoChunks(maxCharArticleName);
List<string> sellPriceChunks = sellPrice.ToString("#,##0.00").SplitIntoChunks(maxCharArticleSellPrice);
List<string> totalChunks = total.ToString("#,##0.00").SplitIntoChunks(maxCharArticleTotal);
int maxLines = (new List<int>() { quantityChunks.Count,
nameChunks.Count,
sellPriceChunks.Count,
totalChunks.Count }).Max();
for (int i = 0; i < maxLines; i++)
{
lines.Add(String.Format("{0}{1}{2}{3}",
quantityChunks.Count > i ?
quantityChunks[i].PadLeft(maxCharArticleQuantity) :
String.Empty.PadLeft(maxCharArticleQuantity),
nameChunks.Count > i ?
nameChunks[i].PadLeft(maxCharArticleName) :
String.Empty.PadLeft(maxCharArticleName, ' '),
sellPriceChunks.Count > i ?
sellPriceChunks[i].PadLeft(maxCharArticleSellPrice) :
String.Empty.PadeLeft(maxCharArticleSellPrice),
totalChunks.Count > i ?
totalChunks[i].PadLeft(maxCharArticleTotal) :
String.Empty.PadLeft(maxCharArticleTotal));
}
return lines;
}
The above code does a couple of things. First, it calls SplitIntoChunks on each of the four variables.
Next, it uses the Max() extension method (you'll need to add a reference to System.Linq if you don't already have one, plus a using directive). to determine the maximum number of lines needed (based on the highest count of the four lists).
Finally, it uses a 4for loop to build the lines. Note that I use the ternary operator (?:) to build each line. The reason I went this route is so that the lines would be properly formatted, even if one or more of the 4 values didn't have something for that line. In other words:
quantityChunks.Count > i
If the count of items in quantityChunks is greater than i, then there is an entry for that item for that line.
quanityChunks[i].PadLeft(maxCharArticleQuantity, ' ')
If the condition evaluates to true, we use the corresponding entry (and pad it with spaces to keep alignment).
String.Empty.PadLeft(maxCharArticleQuantity, ' ')
If the condition is false, we simply put in spaces for the maximum number of characters for that position in the line.
Then you can print the output like this:
foreach(string line in lines)
{
Console.WriteLine(line);
}
The portions of the code where I check for the maximum number of lines and the use of a ternary operator in the String.Format feel very kludgy to me, but I don't have time to finesse it into something more respectable. I hope this at least points you in the right direction.
EDIT
Fixed the PadLeft syntax and removed the character specified, as space is used by default.
Does anyone know of an implementation of the php function mb_strcut in C#?
http://php.net/manual/en/function.mb-strcut.php
mb_strcut() extracts a substring from a string similarly to mb_substr(), but operates on bytes instead of characters. If the cut position happens to be between two bytes of a multi-byte character, the cut is performed starting from the first byte of that character. This is also the difference to the substr() function, which would simply cut the string between the bytes and thus result in a malformed byte sequence.
Thanks Dash could have not written the below without your help
public static string LimitByteLength(string input, int startByte, int byteLength)
{
var maxLength = startByte + byteLength;
return
new string(
input.SkipWhile((c, i) => GetByteCount(input.Substring(0, i + 1)) <= startByte)
.TakeWhile((c, i) => GetByteCount(input.Substring(0, i + 1)) <= maxLength).ToArray());
}
private static int GetByteCount(string input)
{
return Encoding.Unicode.GetByteCount(input);
}
I'm currently making a game but I seem to have problems reading values from a text file. For some reason, when I read the value, it gives me the ASCII code of the value rather than the actual value itself when I wrote it to the file. I've tried about every ASCII conversion function and string conversion function, but I just can't seem to figure it out.
I use a 2D array of integers. I use a nested for loop to write each element into the file. I've looked at the file and the values are correct, but I don't understand why it's returning the ASCII code. Here's the code I'm using to write and read to file:
Writing to file:
for (int i = 0; i < level.MaxRows(); i++)
{
for (int j = 0; j < level.MaxCols(); j++)
{
fileWrite.Write(level.GetValueAtIndex(i, j) + " ");
//Console.WriteLine(level.GetValueAtIndex(i, j));
}
//add new line
fileWrite.WriteLine();
}
And here's the code where I read the values from the file:
string str = "";
int iter = 0; //used to iterate in each column of array
for (int i = 0; i < level.MaxRows(); i++)
{
iter = 0;
//TODO: For some reason, the file is returning ASCII code, convert to int
//keep reading characters until a space is reached.
str = fileRead.ReadLine();
//take the above string and extract the values from it.
//Place each value in the level.
foreach (char id in str)
{
if (id != ' ')
{
//convert id to an int
num = (int)id;
level.ChangeTile(i, iter, num);
iter++;
}
}
This is the latest version of the loop that I use to read the values. Reading other values is fine; it's just when I get to the array, things go wrong. I guess my question is, why did the conversion to ASCII happen? If I can figure that out, then I might be able to solve the issue. I'm using XNA 4 to make my game.
This is where the convertion to ascii is happening:
fileWrite.Write(level.GetValueAtIndex(i, j) + " ");
The + operator implicitly converts the integer returned by GetValueAtIndex into a string, because you are adding it to a string (really, what did you expect to happen?)
Furthermore, the ReadLine method returns a String, so I am not sure why you'd expect a numeric value to magically come back here. If you want to write binary data, look into BinaryWriter
This is where you are converting the characters to character codes:
num = (int)id;
The id variable is a char, and casting that to int gives you the character code, not the numeric value.
Also, this converts a single character, not a whole number. If you for example have "12 34 56 " in your text file, it will get the codes for 1, 2, 3, 4, 5 and 6, not 12, 34 and 56.
You would want to split the line on spaces, and parse each substring:
foreach (string id in str.Split(' ')) {
if (id.Length > 0) {
num = Int32.Parse(id);
level.ChangeTile(i, iter, num);
iter++;
}
}
Update: I've kept the old code (below) with the assumption that one record was on each line, but I've also added a different way of doing it that should work with multiple integers on a line, separated by a space.
Multiple records on one line
str = fileRead.ReadLine();
string[] values = str.Split(new Char[] {' '});
foreach (string value in values)
{
int testNum;
if (Int32.TryParse(str, out testnum))
{
// again, not sure how you're using iter here
level.ChangeTile(i, iter, num);
}
}
One record per line
str = fileRead.ReadLine();
int testNum;
if (Int32.TryParse(str, out testnum))
{
// however, I'm not sure how you're using iter here; if it's related to
// parsing the string, you'll probably need to do something else
level.ChangeTile(i, iter, num);
}
Please note that the above should work if you write out each integer line-by-line (i.e. how you were doing it via the WriteLine which you remarked out in your code above). If you switch back to using a WriteLine, this should work.
You have:
foreach (char id in str)
{
//convert id to an int
num = (int)id;
A char is an ASCII code (or can be considered as such; technically it is a unicode code-point, but that is broadly comparable assuming you are writing ANSI or low-value UTF-8).
What you want is:
num = (int)(id - '0');
This:
fileWrite.Write(level.GetValueAtIndex(i, j) + " ");
converts the int returned from level.GetValueAtIndex(i, j) into a string. Assuming the function returns the value 5 for a particular i and j then you write "5 " into the file.
When you then read it is being read as a string which consists of chars and you get the ASCII code of 5 when you cast it simply to an int. What you need is:
foreach (char id in str)
{
if (id != ' ')
{
//convert id to an int
num = (int)(id - '0'); // subtract the ASCII value for 0 from your current id
level.ChangeTile(i, iter, num);
iter++;
}
}
However this only works if you only ever are going to have single digit integers (only 0 - 9). This might be better:
foreach (var cell in fileRead.ReadLine().Split(' '))
{
num = Int.Parse(cell);
level.ChangeTile(i, iter, num);
iter++;
}