I had not found a way of adding two bytestrings in google documentation. I tried to convert them to strings, concatenate, than convert back. But it gets stuck on conversion back to ByteString.
if (someList.Count > 3)
{
var bigString = "";
for (int i = 0; i < someList.Count; i++)
{
string partString= someList.ElementAt(i).ToBase64();
bigString += partString;
Logger.Write($"{i}");
}
Logger.Write("here");
var chunk = ByteString.FromBase64(bigString);
Logger.Write("here2");
SomeFunc(chunk);
someList.Clear();
}
It gets "here", but never to "here2".
UPD: ByteStrings in someList contain WaveIn audioData
I would definitely not suggest converting to base64 and back - there's no need for that, and it causes issues as base64 strings aren't naturally joinable like this due to padding.
It looks like you're not actually trying to concatenate two ByteStrings, but some arbitrary number. I'd suggest writing them all to a MemoryStream, then creating a new ByteString from that:
using var tempStream = new MemoryStream();
foreach (var byteString in someList)
{
byteString.WriteTo(tempStream);
}
tempStream.Position = 0;
var combinedString = ByteString.FromStream(tempStream);
Related
This question already has answers here:
Why does adding a new value to list<> overwrite previous values in the list<>
(2 answers)
Closed 11 months ago.
I'm using a for loop to read the lines of a file one by one. I then use the data in each line of the file to change the properties of the objects in an array.
I've run through the code in Debug mode, and it all seems to run fine. It reads the line of the file which corresponds to the i value of the for loop correctly, it defines an array of this data based on the commas, and it creates a temporary object which stores these values in the correct format. I know all of this because, as I've said, I have checked the values in Debug mode.
However, on the last line inside of the loop it seems to change every element in the array to the values stored in scrOldScore, whereas I want it to keep the values that it read in from
previous lines and just update the element of the array corresponding to the i of the for loop.
With each iteration of the for loop, the array holds identical data in each element that isn't null, and with each iteration that data changes to the most recently defined scrOldScore.
string str;
Data d = new Data();
for (int i = 0; i < arr.Length; i++)
{
str = File.ReadLines(FileName).Skip(i).Take(1).First(); ;
string[] string = new string[4];
string = str.Split(',');
d.Property01 = string[0];
d.Property02 = Convert.ToInt32(string[1]);
d.Property03 = string[2];
d.Property04 = string[3];
arr[i] = d;
}
Thanks for any help :)
The code doesn't work because it's putting the same ScoreData object instance in every array item. You need to create a new ScoreData object instance inside the loop.
But also the original code re-opens and reads through the file a little further on every iteration of the loop. This is grossly inefficient. You can make things run in a fraction of the time by keeping the same file handle, which you can do by using the file, rather than the array, as the main item for the loop:
int i = 0;
var lines = File.ReadLines(strFileName);
foreach(var line in lines)
{
var data = line.Split(',');
ScrScores[i] = new ScoreData() {
Name = data[0],
Score = int.Parse(data[1]),
Accuracy = data[2],
ReactionTime = data[3]
};
i++;
}
Of course, there's a chance here the file might be larger than the array. If that's possible, you should probably be using a List<ScoreData> rather than an array anyway:
var ScrScores = new List<ScrScores>();
var lines = File.ReadLines(FileName);
foreach(var line in lines)
{
var data = line.Split(',');
ScrScores.Add(new ScoreData() {
Name = data[0],
Score = int.Parse(data[1]),
Accuracy = data[2],
ReactionTime = data[3]
});
}
Failing both of these, I would open a StreamReader object, rather than using File.ReadLines(), and call it's ReadLine() method in each for loop iteration.
using (var rdr = new StreamReader(FileName))
{
for (int i = 0; i < ScrScores.Length; i++)
{
var data = rdr.ReadLine().Split(',');
ScrScores[i] = new ScoreData() {
Name = data[0],
Score = int.Parse(data[1]),
Accuracy = data[2],
ReactionTime = data[3]
};
}
}
One last thing we can do, since it looks like we're replacing all the elements in an array of known size, is replace the entire array. To do this we can read exactly that many items from the file and project them into ScoreData objects as we go:
ScrScores = File.ReadLines(FileName).
Select(line => line.Split(',')).
Select(data => new ScoreData() {
Name = data[0],
Score = int.Parse(data[1]),
Accuracy = data[2],
ReactionTime = data[3]
}).
Take(ScrScores.Length).
ToArray();
Technically, this is one line of code (only one semi-colon and could be extended to fit on the same line).
I'll add that hungarian notation variable prefixes like str and arr are a hold-over from the VB6 era. Today, tooling has improved and even Microsoft's own coding guidelines — where the practice originated — now specifically say (in bold type, no less) "Do not use Hungarian notation". All the code examples I provided reflect this recommendation.
I have a binary file I am reading and printing into a textbox while wrapping at a set point, but it is wrapping at places it shouldn't be. I want to ignore all line feed characters except those I have defined.
There isn't a single Newline byte, rather it seems to be a series of them. I think I found the series of Hex values 00-01-01-0B that seem to correspond with where the line feeds should be.
How do I ignore existing line breaks, and use what I want instead?
This is where I am at:
shortFile = new FileStream(#"tempfile.dat", FileMode.Open, FileAccess.Read);
DisplayArea.Text = "";
byte[] block = new byte[1000];
shortFile.Position = 0;
while (shortFile.Read(block, 0, 1000) > 0)
{
string trimmedText = System.Text.Encoding.Default.GetString(block);
DisplayArea.Text += trimmedText + "\n";
}
I had just figured it out a couple minutes before dlatikay posted, but really appreciated seeing that he also had the right idea. I just replaced all control characters with spaces.
for (int i = 0; i < block.Length; i++)
{
if (block[i] < 32)
{
block[i] = 0x20;
}
}
I would like to consecutively read from a text file that is generated by my program. The problem is that after parsing the file for the first time, my program reads the last line of the file before it can begin re-parsing, which causes it to accumulates unwanted data.
3 photos: first is creating tournament and showing points, second is showing text file and the third is showing that TeamA got more 3 points
StreamReader = new StreamReader("Torneios.txt");
torneios = 0;
while (!rd.EndOfStream)
{
string line = rd.ReadLine();
if (line == "Tournament")
{
torneios++;
}
else
{
string[] arr = line.Split('-');
equipaAA = arr[0];
equipaBB = arr[1];
res = Convert.ToChar(arr[2]);
}
}
rd.Close();
That is what I'm using at the moment.
To avoid mistakes like these, I highly recommend using File.ReadAllText or File.ReadAllLines unless you are using large files (in which case they are not good choices), here is an example of an implementation of such:
string result = File.ReadAllText("textfilename.txt");
Regarding your particular code, an example using File.ReadAllLines which achieves this is:
string[] lines = File.ReadAllLines("textfilename.txt");
for(int i = 0; i < lines.Length; i++)
{
string line = lines[i];
//Do whatever you want here
}
Just to make it clear, this is not a good idea if the files you intend to read from are large (such as binary files).
I am having the problem that whitespace of some sort is being inserted between characters when I am converting a Queue<byte> list into a string for comparison. I do not think that they are actual whitespace characters, however, because the Queue retains only seven values and when debugging I am still able to see the seven character values. See image:
Relevant code:
Queue<byte> bufKeyword = new Queue<byte>(7);
// Remove old byte from queue and add new one
if (bufKeyword.Count == 7) bufKeyword.Dequeue();
bufKeyword.Enqueue((byte)fsInput.ReadByte());
// Check buffer string for match
StringBuilder bufKeywordString = new StringBuilder();
foreach (byte qByte in bufKeyword) {
bufKeywordString.Append(Encoding.ASCII.GetString(BitConverter.GetBytes(qByte)));
}
string _bufKeywordString = bufKeywordString.ToString();
Console.WriteLine("{0}", _bufKeywordString); //DEBUG - SEE IMAGE
StringBuilder bufWriteString = new StringBuilder();
if (_bufKeywordString.StartsWith("time=")) //Does not work because of 'whitespace'
{
for (int i = 1; i < 25; i++) { bufWriteString.Append(fsInput.ReadByte()); } // Read next 24 bytes
fileWriteQueue.Enqueue(bufWriteString.ToString()); // Add this data to write queue
fileWriteQueueCount++;
fileBytesRead += 24; // Change to new spot in file
}
There is no BitConverter.GetBytes for byte argument. byte gets converted to short, and BitConverter.GetBytes(short) returns an array of two elements.
So instead of
bufKeywordString.Append(Encoding.ASCII.GetString(BitConverter.GetBytes(qByte)));
try
bufKeywordString.Append(Encoding.ASCII.GetString(new byte[] {qByte});
I want to deserialize a list of 1 million pairs of (String,Guid) for a performance critical app. The format can be anything I choose, and serialization does not have the same performance requirements.
What sort of approach is best? Text or binary? Write each pair (string,guid) consecutively, or write all strings followed by all guids?
I started playing with LinqPad, (and the simpler example of deserializing strings only) and found that (slightly counter-intuitively), using a TextReader and ReadLine() was a fair bit faster than using a BinaryReader and ReadString(). (Is the filesystem cache playing tricks on me?)
public string[] DeSerializeBinary()
{
var tmr = System.Diagnostics.Stopwatch.StartNew();
long ms = 0;
string[] arr = null;
using (var rdr = new BinaryReader(new FileStream(file, FileMode.Open, FileAccess.Read)))
{
var num = rdr.ReadInt32();
arr = new String[num];
for (int i = 0; i < num; i++)
{
arr[i] = rdr.ReadString();
}
tmr.Stop();
ms = tmr.ElapsedMilliseconds;
Console.WriteLine("DeSerializeBinary took {0}ms", ms);
}
return arr;
}
public string[] DeserializeText()
{
var tmr = System.Diagnostics.Stopwatch.StartNew();
long ms = 0;
string[] arr = null;
using (var rdr = File.OpenText(file))
{
var num = Int32.Parse(rdr.ReadLine());
arr = new String[num];
for (int i = 0; i < num; i++)
{
arr[i] = rdr.ReadLine();
}
tmr.Stop();
ms = tmr.ElapsedMilliseconds;
Console.WriteLine("DeserializeText took {0}ms", ms);
}
return arr;
}
Some Edits:
I used RamMap to clear the file system cache, and it turns out there was very little difference to Text & Binary reader for strings only.
I have a fairly simple class that holds the string and guid. It also holds an int index which corresponds to its position in the list. Obviously there's no need to include this in serialization.
In a test for (binary) deSerializing Strings and Guids alternately, I get around 500ms.
Ideal timing is 50ms, or as close as I can get. However, a simple experiment showed it takes at least 120ms to read the (compressed) file into memory from a reasonably fast SSD drive, without any sort of parsing at all. So 50ms seems unlikely.
Our strings have no theoretical length restrictions. However, we can assume that the performance target only applies if they are all 20 characters or less.
Timings include opening the file.
Reading the Strings is the clear bottleneck now (hence my experiments with serializing strings only). The JIT_NewFast took 30% before I preallocated an array of 16bytes for reading GUIDs.
It's not surprising that reading a bunch of strings is faster with StreamReader than with BinaryReader. StreamReader reads in blocks from the underlying stream, and parses the strings from that buffer. BinaryReader doesn't have a buffer like that. It reads the string length from the underlying stream, and then reads that many characters. So BinaryReader makes more calls to the base stream's Read method.
But there's more to deserializing a (String, Guid) pair than just reading. You also have to parse the Guid. If you write the file in binary then the Guid is written in binary, which makes it much easier and faster to create a Guid structure. If it's a string, then you have to call new Guid(string) to parse the text and create a Guid, after you split the line into its two fields.
Hard to say which of those will be faster.
I can't imagine that we're talking about a whole lot of time here. Certainly reading a file with a million lines will take around a second. Unless the string is really long. A GUID is only 36 characters if you count the separators, right?
With BinaryWriter, you can write the file like this:
writer.Write(count); // integer number of records
foreach (var pair in pairs)
{
writer.Write(pair.theString);
writer.Write(pair.theGuid.ToByteArray());
}
And to read it, you have:
count = reader.ReadInt32();
byte[] guidBytes = new byte[16];
for (int i = 0; i < count; ++i)
{
string s = reader.ReadString();
reader.Read(guidBytes, 0, guidBytes.Length);
pairs.Add(new Pair(s, new Guid(guidBytes));
}
Whether that's faster than splitting a string and calling the Guid constructor that takes a string parameter, I don't know.
I suspect that any difference is going to be pretty slight. I'd probably go with the simplest method: a text file.
If you want to get really crazy, you can write a custom format that you can easily slurp up in just a couple of large reads (a header, an index, and two arrays for strings and GUIDs), and do everything else in memory. That would almost certainly be faster. But faster enough to warrant the extra work? Doubtful.
Update
Or maybe not doubtful. Here's some code that writes and reads a custom binary format. The format is:
count (int32)
guids (count * 16 bytes)
strings (one big concatenated string)
index (index of each string's starting character in the big string)
I assume you're using a Dictionary<string, Guid> to hold these things. But your data structure doesn't really matter. The code would be substantially the same.
Note that I tested this very briefly. I won't say that the code is 100% bug free, but I think you can get the idea of what I'm doing.
private void WriteGuidFile(string filename, Dictionary<string, Guid>guids)
{
using (var fs = File.Create(filename))
{
using (var writer = new BinaryWriter(fs, Encoding.UTF8))
{
List<int> stringIndex = new List<int>(guids.Count);
StringBuilder bigString = new StringBuilder();
// write count
writer.Write(guids.Count);
// Write the GUIDs and build the string index
foreach (var pair in guids)
{
writer.Write(pair.Value.ToByteArray(), 0, 16);
stringIndex.Add(bigString.Length);
bigString.Append(pair.Key);
}
// Add one more entry to the string index.
// makes deserializing easier
stringIndex.Add(bigString.Length);
// Write the string that contains all of the strings, combined
writer.Write(bigString.ToString());
// write the index
foreach (var ix in stringIndex)
{
writer.Write(ix);
}
}
}
}
Reading is just slightly more involved:
private Dictionary<string, Guid> ReadGuidFile(string filename)
{
using (var fs = File.OpenRead(filename))
{
using (var reader = new BinaryReader(fs, Encoding.UTF8))
{
// read the count
int count = reader.ReadInt32();
// The guids are in a huge byte array sized 16*count
byte[] guidsBuffer = new byte[16*count];
reader.Read(guidsBuffer, 0, guidsBuffer.Length);
// Strings are all concatenated into one
var bigString = reader.ReadString();
// Index is an array of int. We can read it as an array of
// ((count+1) * 4) bytes.
byte[] indexBuffer = new byte[4*(count+1)];
reader.Read(indexBuffer, 0, indexBuffer.Length);
var guids = new Dictionary<string, Guid>(count);
byte[] guidBytes = new byte[16];
int startix = 0;
int endix = 0;
for (int i = 0; i < count; ++i)
{
endix = BitConverter.ToInt32(indexBuffer, 4*(i+1));
string key = bigString.Substring(startix, endix - startix);
Buffer.BlockCopy(guidsBuffer, (i*16),
guidBytes, 0, 16);
guids.Add(key, new Guid(guidBytes));
startix = endix;
}
return guids;
}
}
}
A couple of notes here. First, I'm using BitConverter to convert the data in the byte arrays to integers. It would be faster to use unsafe code and just index into the arrays using an int32*.
You might gain some speed by using pointers to index into the guidBuffer and calling Guid Constructor (Int32, Int16, Int16, Byte, Byte, Byte, Byte, Byte, Byte, Byte, Byte) rather than using Buffer.BlockCopy to copy the GUID into the temporary array.
You could make the string index an index of lengths rather than the starting positions. That would eliminate the need for the extra value at the end of the array, but it's unlikely that it'd make any difference in the speed.
There might be other optimization opportunities, but I think you get the general idea here.