Develop Reference

Convert RepeatedField<ByteString> to Byte[]

Defined a gRPC Python service in a Docker.
The services in my PROTO file:
rpc ClientCommand(ClientRequest) returns (ClientResponse){}
Definition of "ClientResponse":
message ClientResponse{
int32 request_id = 1;
repeated int32 prediction_status = 2;
repeated string prediction_info = 3;
repeated string prediction_error = 4;
repeated string prediction_result_name = 5;
repeated bytes prediction_result = 6;
repeated bytes prediction_config = 7;
repeated bytes prediction_log = 8;
}
On client side, I want to catch the repeated bytes and convert them into a file(I know it better works with a stream but for the moment I would like to do it like this).
The repeated strings and integer I can perfectly convert to List --> OK
The repeated bytes I would like to convert to Byte[]. Their type: Google.ProtoBuf.Collections.RepeatedField<Google.ProtoBuf.ByteString>.
At first it seems to be impossible to convert this type to a Byte[]. Could somebody help me with this please? My solution temporary:
byte[] test = new byte[100];
Google.Protobuf.ByteString[] test2 = new Google.Protobuf.ByteString[100];
response.PredictionResult.CopyTo(test2,0);
test2.CopyTo(test,0);
WriteFile(#"C:\programs\file.txt", test);

Meanwhile I figure it out:
public void WriteFileResult(string fileName,Google.Protobuf.Collections.RepeatedField<ByteString> data)
{
byte[] bytes = new byte[data.Count];
ByteString[] dataByteString = new ByteString[data.Count];
string path = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
if (!path.EndsWith(#"\")) path += #"\";
if (File.Exists(Path.Combine(path, fileName)))
File.Delete(Path.Combine(path, fileName));
using (FileStream fs = new FileStream(Path.Combine(path, fileName), FileMode.CreateNew, FileAccess.Write))
{
data.CopyTo(dataByteString, 0);
bytes = dataByteString[0].ToByteArray();
fs.Write(bytes, 0, (int)bytes.Length);
_serviceResponseModel.Json = false;
//fs.Close()
}
}

How do I replicate the functionality of tail -f in C# [duplicate]

I want to read file continuously like GNU tail with "-f" param. I need it to live-read log file.
What is the right way to do it?

More natural approach of using FileSystemWatcher:
var wh = new AutoResetEvent(false);
var fsw = new FileSystemWatcher(".");
fsw.Filter = "file-to-read";
fsw.EnableRaisingEvents = true;
fsw.Changed += (s,e) => wh.Set();
var fs = new FileStream("file-to-read", FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
using (var sr = new StreamReader(fs))
{
var s = "";
while (true)
{
s = sr.ReadLine();
if (s != null)
Console.WriteLine(s);
else
wh.WaitOne(1000);
}
}
wh.Close();
Here the main reading cycle stops to wait for incoming data and FileSystemWatcher is used just to awake the main reading cycle.

You want to open a FileStream in binary mode. Periodically, seek to the end of the file minus 1024 bytes (or whatever), then read to the end and output. That's how tail -f works.
Answers to your questions:
Binary because it's difficult to randomly access the file if you're reading it as text. You have to do the binary-to-text conversion yourself, but it's not difficult. (See below)
1024 bytes because it's a nice convenient number, and should handle 10 or 15 lines of text. Usually.
Here's an example of opening the file, reading the last 1024 bytes, and converting it to text:
static void ReadTail(string filename)
{
using (FileStream fs = File.Open(filename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
// Seek 1024 bytes from the end of the file
fs.Seek(-1024, SeekOrigin.End);
// read 1024 bytes
byte[] bytes = new byte[1024];
fs.Read(bytes, 0, 1024);
// Convert bytes to string
string s = Encoding.Default.GetString(bytes);
// or string s = Encoding.UTF8.GetString(bytes);
// and output to console
Console.WriteLine(s);
}
}
Note that you must open with FileShare.ReadWrite, since you're trying to read a file that's currently open for writing by another process.
Also note that I used Encoding.Default, which in US/English and for most Western European languages will be an 8-bit character encoding. If the file is written in some other encoding (like UTF-8 or other Unicode encoding), It's possible that the bytes won't convert correctly to characters. You'll have to handle that by determining the encoding if you think this will be a problem. Search Stack overflow for info about determining a file's text encoding.
If you want to do this periodically (every 15 seconds, for example), you can set up a timer that calls the ReadTail method as often as you want. You could optimize things a bit by opening the file only once at the start of the program. That's up to you.

To continuously monitor the tail of the file, you just need to remember the length of the file before.
public static void MonitorTailOfFile(string filePath)
{
var initialFileSize = new FileInfo(filePath).Length;
var lastReadLength = initialFileSize - 1024;
if (lastReadLength < 0) lastReadLength = 0;
while (true)
{
try
{
var fileSize = new FileInfo(filePath).Length;
if (fileSize > lastReadLength)
{
using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
fs.Seek(lastReadLength, SeekOrigin.Begin);
var buffer = new byte[1024];
while (true)
{
var bytesRead = fs.Read(buffer, 0, buffer.Length);
lastReadLength += bytesRead;
if (bytesRead == 0)
break;
var text = ASCIIEncoding.ASCII.GetString(buffer, 0, bytesRead);
Console.Write(text);
}
}
}
}
catch { }
Thread.Sleep(1000);
}
}
I had to use ASCIIEncoding, because this code isn't smart enough to cater for variable character lengths of UTF8 on buffer boundaries.
Note: You can change the Thread.Sleep part to be different timings, and you can also link it with a filewatcher and blocking pattern - Monitor.Enter/Wait/Pulse. For me the timer is enough, and at most it only checks the file length every second, if the file hasn't changed.

This is my solution
static IEnumerable<string> TailFrom(string file)
{
using (var reader = File.OpenText(file))
{
while (true)
{
string line = reader.ReadLine();
if (reader.BaseStream.Length < reader.BaseStream.Position)
reader.BaseStream.Seek(0, SeekOrigin.Begin);
if (line != null) yield return line;
else Thread.Sleep(500);
}
}
}
so, in your code you can do
foreach (string line in TailFrom(file))
{
Console.WriteLine($"line read= {line}");
}

You could use the FileSystemWatcher class which can send notifications for different events happening on the file system like file changed.

private void button1_Click(object sender, EventArgs e)
{
if (folderBrowserDialog.ShowDialog() == DialogResult.OK)
{
path = folderBrowserDialog.SelectedPath;
fileSystemWatcher.Path = path;
string[] str = Directory.GetFiles(path);
string line;
fs = new FileStream(str[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
tr = new StreamReader(fs);
while ((line = tr.ReadLine()) != null)
{
listBox.Items.Add(line);
}
}
}
private void fileSystemWatcher_Changed(object sender, FileSystemEventArgs e)
{
string line;
line = tr.ReadLine();
listBox.Items.Add(line);
}

If you are just looking for a tool to do this then check out free version of Bare tail

.pkpass create fail because of manifest pass.json string format?

This is a very strange question.
I using C# to create a pass.json and save it to memoryStream, it work normally. After that I create the manifest.json SHA1 data which including that pass.json, the string of manifest.json like this and it is totally correct.
{"icon.png": "9423bd00e2b01c59a3265c38b5062fac7da0752d",
"icon#2x.png": "4d1db55bdaca70b685c013529a1c0dcbd7046524",
"logo.png": "ee5b053e63dbfe3b78378c15d163331d68a0ede8",
"logo#2x.png": "2f9e3a55bded1163620719a4d6c1ad496ed40c17",
"pass.json": "fd68bf77757d3057263a9aca0e5110ddd933934a"}
After generate pkpass as my phone, it can't open. I change the pass.json SHA1 code as "fd68bf77757d3057263a9aca0e5110ddd933934a" without using a value to save it, it work.
The coding like following:
// This version run success
var strPass = JavascriptSerialize(details);
var sw = new StreamWriter(assetsFolder + #"pass.json");
sw.Write(strPass);
sw.Close();
manifest.passjson = GetSha1Hash(assetsFolder + manifest.GetAssetBoardingPass(libPkPass_object_boardingPass.JsonObjects.AssetTypes.passjson));
//manifest.passjson = "2f9e3a55bded1163620719a4d6c1ad496ed40c17"
// end
// This version run fail
var strPass = JavascriptSerialize(details);
MemoryStream stream = new MemoryStream();
StreamWriter writer = new StreamWriter(strPass);
writer.Write(s);
writer.Flush();
stream.Position = 0;
var a = GetSha1HashMemory(passStream);
private static string GetSha1HashMemory(Stream passStream)
{
//var bs = new BufferedStream(passStream);
using (SHA1Managed sha = new SHA1Managed())
{
byte[] checksum = sha.ComputeHash(passStream);
string sendCheckSum = BitConverter.ToString(checksum)
.Replace("-", string.Empty);
return sendCheckSum.ToString().ToLower();
}
}
manifest.passjson = a;
//manifest.passjson = "2f9e3a55bded1163620719a4d6c1ad496ed40c17" (same data )
//end
What is going on?????? I can find out any question that string is wrong.
The pkpass provide in here (sendspace).
Can any body told me where is wrong?
Big Thank!

Two mistakes :
ComputeHash(Stream) and using Stream
ComputeHash(Stream) : ComputeHash stream only using System.IO.Stream, but not MemoryStream, change to ComputeHash(bytes[]) can handle it
using Stream: I try to pass the stream to other function, it is not a good example, the stream need to create a new one and it may replace some bytes at your computer stream. In this case, I just need to call this function will out open new one, it will fix
StringBuilder formatted;
using (var sha1 = new SHA1Managed())
{
//var bytePass = ReadFully(passStream);
var bytePass = passStream.ToArray();
var hash = sha1.ComputeHash(bytePass);
formatted = new StringBuilder(2 * hash.Length);
foreach (var b in hash)
{
formatted.AppendFormat("{0:X2}", b);
}
}
manifest.passjson = formatted.ToString().ToLower();

Unable to re-construct a file using byte array retrieved from another file (chunk-by-chunk)

I am currently trying to construct file B by extracting a certain length of bytes from file A (chunk-by-chunk). The size of file B is 38052441 bytes, and its location in file A is from byte 34 onward. If I do it in one shot, I manage to extract file B from file A without any issue, as shown in the snippet below.
test = new byte[38052441];
//madefilePath: file A, madecabfilePath: file B
using (BinaryReader reader = new BinaryReader(new FileStream(madefilePath, FileMode.Open)))
using (BinaryWriter bw = new BinaryWriter(File.Open(madecabfilePath, FileMode.OpenOrCreate)))
{
reader.BaseStream.Seek(34, SeekOrigin.Begin);
reader.Read(test, 0, 38052441);
bw.Write(test);
bw.Close();
reader.Close();
}
Howerver, if I try to do it in multiple query (I have to do this, because this feature will be ported to compact framework in the future), I kept on getting a corrupted file. Currently, I am testing by getting the first 20Mb, write into a file, then get the remaining bytes and write it into the file again.
int max = 38052474;
int offset = 34;
int weight = 20000000;
bool isComplete = false;
test = null;
test = new byte[weight];
using (BinaryWriter bw = new BinaryWriter(File.Open(madecabfilePath, FileMode.OpenOrCreate)))
using (BinaryReader reader = new BinaryReader(new FileStream(madefilePath, FileMode.Open)))
{
while (!isComplete)
{
if (offset + weight < max)
{
reader.BaseStream.Seek(offset, SeekOrigin.Begin);
reader.Read(test, 0, weight);
bw.Write(test);
offset = offset + weight;
}
else
{
weight = max - offset;
test = null;
test = new byte[weight];
reader.BaseStream.Seek(offset, SeekOrigin.Begin);
reader.Read(test, 0, weight);
bw.Write(test);
//Terminate everything
reader.Close();
bw.Close();
isComplete = true;
}
}
}
I think the issue lies with my logic, but I can't figure out why. Any help is appreciated. Thank you.

BinaryReader.Read() returns the number of bytes that were actually read. So you can simplify your logic and probably fix some issues with something like:
using (BinaryWriter bw = new BinaryWriter(File.Open(madecabfilePath, FileMode.OpenOrCreate)))
using (BinaryReader reader = new BinaryReader(new FileStream(madefilePath, FileMode.Open)))
{
reader.BaseStream.Seek(offset, SeekOrigin.Begin);
while (!isComplete)
{
int charsRead = reader.Read(test, 0, weight);
if (charsRead == 0)
{
isComplete = true;
}
else
{
bw.Write(test, 0, charsRead);
}
}
}
Note that you don't need to explicitly close bw or reader, as the using statement will do that for you. Also note that after the first Seek() call the position in the BinaryReader will be kept track of.

Editing a text file in place through C#

I have a huge text file, size > 4GB and I want to replace some text in it programmatically. I know the line number at which I have to replace the text but the problem is that I do not want to copy all the text (along with my replaced line) to a second file. I have to do this within the source file. Is there a way to do this in C#?
The text which has to be replaced is exactly the same size as the source text (if this helps).

Since the file is so large you may want to take a look at the .NET 4.0 support for memory mapped files. Basically you'll need to move the file/stream pointer to the location in the file, overwrite that location, then flush the file to disk. You won't need to load the entire file into memory.
For example, without using memory mapped files, the following will overwrite a part of an ascii file. Args are the input file, the zero based start index and the new text.
static void Main(string[] args)
{
string inputFilename = args[0];
int startIndex = int.Parse(args[1]);
string newText = args[2];
using (FileStream fs = new FileStream(inputFilename, FileMode.Open, FileAccess.Write))
{
fs.Position = startIndex;
byte[] newTextBytes = Encoding.ASCII.GetBytes(newText);
fs.Write(newTextBytes, 0, newTextBytes.Length);
}
}

Unless the new text is exactly the same size as the old text, you will have to re-write the file. There is no way around it. You can at least do this without keeping the entire file in memory.

Hello I tested the following -works well.This caters to variable length lines separated by Environment.NewLine. if you have fixed length lines you can straightaway seek to it.For converting bytes to string and vice versa you can use Encoding.
static byte[] ReadNextLine(FileStream fs)
{
byte[] nl = new byte[] {(byte) Environment.NewLine[0],(byte) Environment.NewLine[1] };
List<byte> ll = new List<byte>();
bool lineFound = false;
while (!lineFound)
{
byte b = (byte)fs.ReadByte();
if ((int)b == -1) break;
ll.Add(b);
if (b == nl[0]){
b = (byte)fs.ReadByte();
ll.Add(b);
if (b == nl[1]) lineFound = true;
}
}
return ll.Count ==0?null: ll.ToArray();
}
static void Main(string[] args)
{
using (FileStream fs = new FileStream(#"c:\70-528\junk.txt", FileMode.Open, FileAccess.ReadWrite))
{
int replaceLine=1231;
byte[] b = null;
int lineCount=1;
while (lineCount<replaceLine && (b=ReadNextLine(fs))!=null ) lineCount++;//Skip Lines
long seekPos = fs.Position;
b = ReadNextLine(fs);
fs.Seek(seekPos, 0);
string line=new string(b.Select(x=>(char)x).ToArray());
line = line.Replace("Text1", "Text2");
b=line.ToCharArray().Select(x=>(byte)x).ToArray();
fs.Write(b, 0, b.Length);
}
}

I'm guessing you'll want to use the FileStream class and seek to your positon, and place your updated data.