I want to read file continuously like GNU tail with "-f" param. I need it to live-read log file.
What is the right way to do it?
More natural approach of using FileSystemWatcher:
var wh = new AutoResetEvent(false);
var fsw = new FileSystemWatcher(".");
fsw.Filter = "file-to-read";
fsw.EnableRaisingEvents = true;
fsw.Changed += (s,e) => wh.Set();
var fs = new FileStream("file-to-read", FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
using (var sr = new StreamReader(fs))
{
var s = "";
while (true)
{
s = sr.ReadLine();
if (s != null)
Console.WriteLine(s);
else
wh.WaitOne(1000);
}
}
wh.Close();
Here the main reading cycle stops to wait for incoming data and FileSystemWatcher is used just to awake the main reading cycle.
You want to open a FileStream in binary mode. Periodically, seek to the end of the file minus 1024 bytes (or whatever), then read to the end and output. That's how tail -f works.
Answers to your questions:
Binary because it's difficult to randomly access the file if you're reading it as text. You have to do the binary-to-text conversion yourself, but it's not difficult. (See below)
1024 bytes because it's a nice convenient number, and should handle 10 or 15 lines of text. Usually.
Here's an example of opening the file, reading the last 1024 bytes, and converting it to text:
static void ReadTail(string filename)
{
using (FileStream fs = File.Open(filename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
// Seek 1024 bytes from the end of the file
fs.Seek(-1024, SeekOrigin.End);
// read 1024 bytes
byte[] bytes = new byte[1024];
fs.Read(bytes, 0, 1024);
// Convert bytes to string
string s = Encoding.Default.GetString(bytes);
// or string s = Encoding.UTF8.GetString(bytes);
// and output to console
Console.WriteLine(s);
}
}
Note that you must open with FileShare.ReadWrite, since you're trying to read a file that's currently open for writing by another process.
Also note that I used Encoding.Default, which in US/English and for most Western European languages will be an 8-bit character encoding. If the file is written in some other encoding (like UTF-8 or other Unicode encoding), It's possible that the bytes won't convert correctly to characters. You'll have to handle that by determining the encoding if you think this will be a problem. Search Stack overflow for info about determining a file's text encoding.
If you want to do this periodically (every 15 seconds, for example), you can set up a timer that calls the ReadTail method as often as you want. You could optimize things a bit by opening the file only once at the start of the program. That's up to you.
To continuously monitor the tail of the file, you just need to remember the length of the file before.
public static void MonitorTailOfFile(string filePath)
{
var initialFileSize = new FileInfo(filePath).Length;
var lastReadLength = initialFileSize - 1024;
if (lastReadLength < 0) lastReadLength = 0;
while (true)
{
try
{
var fileSize = new FileInfo(filePath).Length;
if (fileSize > lastReadLength)
{
using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
fs.Seek(lastReadLength, SeekOrigin.Begin);
var buffer = new byte[1024];
while (true)
{
var bytesRead = fs.Read(buffer, 0, buffer.Length);
lastReadLength += bytesRead;
if (bytesRead == 0)
break;
var text = ASCIIEncoding.ASCII.GetString(buffer, 0, bytesRead);
Console.Write(text);
}
}
}
}
catch { }
Thread.Sleep(1000);
}
}
I had to use ASCIIEncoding, because this code isn't smart enough to cater for variable character lengths of UTF8 on buffer boundaries.
Note: You can change the Thread.Sleep part to be different timings, and you can also link it with a filewatcher and blocking pattern - Monitor.Enter/Wait/Pulse. For me the timer is enough, and at most it only checks the file length every second, if the file hasn't changed.
This is my solution
static IEnumerable<string> TailFrom(string file)
{
using (var reader = File.OpenText(file))
{
while (true)
{
string line = reader.ReadLine();
if (reader.BaseStream.Length < reader.BaseStream.Position)
reader.BaseStream.Seek(0, SeekOrigin.Begin);
if (line != null) yield return line;
else Thread.Sleep(500);
}
}
}
so, in your code you can do
foreach (string line in TailFrom(file))
{
Console.WriteLine($"line read= {line}");
}
You could use the FileSystemWatcher class which can send notifications for different events happening on the file system like file changed.
private void button1_Click(object sender, EventArgs e)
{
if (folderBrowserDialog.ShowDialog() == DialogResult.OK)
{
path = folderBrowserDialog.SelectedPath;
fileSystemWatcher.Path = path;
string[] str = Directory.GetFiles(path);
string line;
fs = new FileStream(str[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
tr = new StreamReader(fs);
while ((line = tr.ReadLine()) != null)
{
listBox.Items.Add(line);
}
}
}
private void fileSystemWatcher_Changed(object sender, FileSystemEventArgs e)
{
string line;
line = tr.ReadLine();
listBox.Items.Add(line);
}
If you are just looking for a tool to do this then check out free version of Bare tail
Related
In native C#, how can I read from the end of a file?
This is pertinent because I need to read a log file, and it doesn't make sense to read 10k, to read the last 3 lines.
To read the last 1024 bytes:
using (var reader = new StreamReader("foo.txt"))
{
if (reader.BaseStream.Length > 1024)
{
reader.BaseStream.Seek(-1024, SeekOrigin.End);
}
string line;
while ((line = reader.ReadLine()) != null)
{
Console.WriteLine(line);
}
}
Maybe something like this will work for you:
using (var fs = File.OpenRead(filePath))
{
fs.Seek(0, SeekOrigin.End);
int newLines = 0;
while (newLines < 3)
{
fs.Seek(-1, SeekOrigin.Current);
newLines += fs.ReadByte() == 13 ? 1 : 0; // look for \r
fs.Seek(-1, SeekOrigin.Current);
}
byte[] data = new byte[fs.Length - fs.Position];
fs.Read(data, 0, data.Length);
}
Take note that this assumes \r\n.
The code below uses a random-access FileStream to seed a StreamReader at an offset near the end of the file, discarding the first read line since it is most likely only partial.
FileStream stream = new FileStream(#"c:\temp\build.txt",
FileMode.Open, FileAccess.Read);
stream.Seek(-1024, SeekOrigin.End); // rewind enough for > 1 line
StreamReader reader = new StreamReader(stream);
reader.ReadLine(); // discard partial line
while (!reader.EndOfStream)
{
string nextLine = reader.ReadLine();
Console.WriteLine(nextLine);
}
Take a look at this related question's answer to read a text file in reverse. There is a lot of complexity to reading a file backward correctly because of stuff like encoding.
I'm currently trying to read a file, modify a few placeholders within and then write the file into an output stream. As its the output stream for a page response in aspx.net I'm using the OutputStream.Write method there (the file is an attachment in the end).
Originally I had:
using (FileStream fs = new FileStream(filename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
while (readBytes < fs.Length)
{
tmpReadBytes = fs.Read(bytes, 0, bytes.Length);
if (tmpReadBytes > 0)
{
readBytes += tmpReadBytes;
page.Response.OutputStream.Write(bytes, 0, tmpReadBytes);
}
}
}
After thinking things over I came up with the following:
foreach(string line in File.ReadLines(filename))
{
string modifiedLine = line.Replace("#PlaceHolder#", "NewValue");
byte[] modifiedByteArray = System.Text.Encoding.UTF8.GetBytes(modifiedLine);
page.Response.OutputStream.Write(modifiedByteArray, 0, modifiedByteArray.length);
}
But it looks inefficient especially with the conversions. So my question is: Is there any better way of doing this?
As note the file itself is not very big, it's an about 3-4 KB sized textfile.
You don't need to handle the bytes your self.
If you know the file is and always will be small,
this.Response.Write(File.ReadAllText("path").Replace("old", "new"));
otherwise
using (var stream = new FileStream("path", FileMode.Open))
{
using (var streamReader = new StreamReader(stream))
{
while (streamReader.Peek() != -1)
{
this.Response.Write(streamReader.ReadLine().Replace("old", "new"));
}
}
}
To get the lines in a string array:
string[] lines = File.ReadAllLines(file);
To alter the lines, use a loop.
for (int i = 0; i < lines.Length; i++)
{
lines[i] = lines[i].Replace("#PlaceHolder#", "NewValue");
}
And to save the new text, first create a string with all the lines.
string output = "";
foreach(string line in lines)
{
output+="\n"+line;
}
And then save the string to the file.
File.WriteAllText(file,output);
I'm attempting to use National Weather Service (U.S.) data, but something has changed recently and the GZip file no longer opens.
.NET 4.5 complains that...
Message=The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.
Source=System
StackTrace:
at System.IO.Compression.GZipDecoder.ReadHeader(InputBuffer input)
at System.IO.Compression.Inflater.Decode()
at System.IO.Compression.Inflater.Inflate(Byte[] bytes, Int32 offset, Int32 length)
at System.IO.Compression.DeflateStream.Read(Byte[] array, Int32 offset, Int32 count)
I don't understand what has changed, but this is becoming a real show-stopper. Can anyone with GZip format experience tell me what has changed to make this stop working?
A file that works:
http://www.srh.noaa.gov/ridge2/Precip/qpehourlyshape/2015/201504/20150404/nws_precip_2015040420.tar.gz
A file that doesn't work:
http://www.srh.noaa.gov/ridge2/Precip/qpehourlyshape/2015/201505/20150505/nws_precip_2015050505.tar.gz
Update with sample code
const string url = "http://www.srh.noaa.gov/ridge2/Precip/qpehourlyshape/2015/201505/20150505/nws_precip_2015050505.tar.gz";
string appPath = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
string downloadPath = Path.Combine(appPath, Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location), "nws_precip_2015050505.tar.gz");
using (var wc = new WebClient())
{
wc.DownloadFile(url, downloadPath);
}
string extractDirPath = Path.Combine(appPath, "Extracted");
if (!Directory.Exists(extractDirPath))
{
Directory.CreateDirectory(extractDirPath);
}
string extractFilePath = Path.Combine(extractDirPath, "nws_precip_2015050505.tar");
using (var fsIn = new FileStream(downloadPath, FileMode.Open, FileAccess.Read))
using (var fsOut = new FileStream(extractFilePath, FileMode.Create, FileAccess.Write))
using (var gz = new GZipStream(fsIn, CompressionMode.Decompress, true))
{
gz.CopyTo(fsOut);
}
It appears that this service SOMETIMES returns tar format files disguised as .tar.gz. This is very confusing, but if you check that the first two bytes are 0x1F and 0x8B, you can detect if the file is a GZip by checking its magic numbers manually.
using (FileStream fs = new FileStream(downloadPath, FileMode.Open, FileAccess.Read))
{
byte[] buffer = new byte[2];
fs.Read(buffer, 0, buffer.Length);
if (buffer[0] == 0x1F
&& buffer[1] == 0x8B)
{
// It's probably a GZip file
}
else
{
// It's probably not a GZip file
}
}
[Resolved] GZipStream complains magic number in header is not correct
//Exception magic number tar.gz file
Migcal error cause
File is not compress into tar.gz properly
File size is too big , above 1+GB
Solution over it
use .net framework 4.5.1 to over the this exception //OR//
manupulate the exsiting solution without change .net framework.
Please follow the step for implementation.
Remane abc.tar.gz as abc (remove extension).
pass this file and directly name to compress function
*public static void Compress(DirectoryInfo directorySelected, string directoryPath)
{
foreach (FileInfo fileToCompress in directorySelected.GetFiles())
{
using (FileStream originalFileStream = fileToCompress.OpenRead())
{
if ((File.GetAttributes(fileToCompress.FullName) &
FileAttributes.Hidden) != FileAttributes.Hidden & fileToCompress.Extension != ".tar.gz")
{
using (FileStream compressedFileStream = File.Create(fileToCompress.FullName + ".tar.gz"))
{
using (System.IO.Compression.GZipStream compressionStream = new System.IO.Compression.GZipStream(compressedFileStream,
System.IO.Compression.CompressionMode.Compress))
{
originalFileStream.CopyTo(compressionStream);
}
}
FileInfo info = new FileInfo(directoryPath + "\\" + fileToCompress.Name + ".tar.gz");
}
}
}
}
3. implement this code in following exception handler try catch block
try
{
TarGzFilePath=#"c:\temp\abc.tar.gz";
FileStream streams = File.OpenRead(TarGzFilePath);
string FileName=string.Empty;
GZipInputStream tarGz = new GZipInputStream(streams);
TarInputStream tar = new TarInputStream(tarGz);
// exception will occured in below lines should apply try catch
TarEntry ze;
try
{
ze = tar.GetNextEntry();// exception occured here "magical number"
}
catch (Exception extra)
{
tar.Close();
tarGz.Close();
streams.Close();
//please close all above , other wise it will come with exception "tihs process use by another process"
//rename your file *for better accuracy you can copy file to other location
File.Move(#"c:\temp\abc.tar.gz", #"c:\temp\abc"); // rename file
DirectoryInfo directorySelected = new DirectoryInfo(Path.GetDirectoryName(#"c:\temp\abc"));
Compress(directorySelected, directoryPath); // directorySelected=c:\temp\abc , directoryPath=c:\temp\abc.tar.gz // process in step 2 function
streams = File.OpenRead(TarGzFilePath);
tarGz = new GZipInputStream(streams);
tar = new TarInputStream(tarGz);
ze = tar.GetNextEntry();
}
// do anything with extraction with your code
}
catch (exception ex)
{
tar.Close();
tarGz.Close();
streams.Close();
}
In native C#, how can I read from the end of a file?
This is pertinent because I need to read a log file, and it doesn't make sense to read 10k, to read the last 3 lines.
To read the last 1024 bytes:
using (var reader = new StreamReader("foo.txt"))
{
if (reader.BaseStream.Length > 1024)
{
reader.BaseStream.Seek(-1024, SeekOrigin.End);
}
string line;
while ((line = reader.ReadLine()) != null)
{
Console.WriteLine(line);
}
}
Maybe something like this will work for you:
using (var fs = File.OpenRead(filePath))
{
fs.Seek(0, SeekOrigin.End);
int newLines = 0;
while (newLines < 3)
{
fs.Seek(-1, SeekOrigin.Current);
newLines += fs.ReadByte() == 13 ? 1 : 0; // look for \r
fs.Seek(-1, SeekOrigin.Current);
}
byte[] data = new byte[fs.Length - fs.Position];
fs.Read(data, 0, data.Length);
}
Take note that this assumes \r\n.
The code below uses a random-access FileStream to seed a StreamReader at an offset near the end of the file, discarding the first read line since it is most likely only partial.
FileStream stream = new FileStream(#"c:\temp\build.txt",
FileMode.Open, FileAccess.Read);
stream.Seek(-1024, SeekOrigin.End); // rewind enough for > 1 line
StreamReader reader = new StreamReader(stream);
reader.ReadLine(); // discard partial line
while (!reader.EndOfStream)
{
string nextLine = reader.ReadLine();
Console.WriteLine(nextLine);
}
Take a look at this related question's answer to read a text file in reverse. There is a lot of complexity to reading a file backward correctly because of stuff like encoding.
In Linux, a lot of IPC is done by appending to a file in 1 process and reading the new content from another process.
I want to do the above in Windows/.NET (Too messy to use normal IPC such as pipes). I'm appending to a file from a Python process, and I want to read the changes and ONLY the changes each time FileSystemWatcher reports an event. I do not want to read the entire file content into memory each time I'm looking for changes (the file will be huge)
Each append operation appends a row of data that starts with a unique incrementing counter (timestamp+key) and ends with a newline.
using (FileStream fs = new FileStream
(fileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
using (StreamReader sr = new StreamReader(fs))
{
while (someCondition)
{
while (!sr.EndOfStream)
ProcessLinr(sr.ReadLine());
while (sr.EndOfStream)
Thread.Sleep(100);
ProcessLinr(sr.ReadLine());
}
}
}
this will help you read only appended lines
You can store the offset of the last read operation and seek the file to that offset when you get a changed file notification. An example follows:
Main method:
public static void Main(string[] args)
{
File.WriteAllLines("test.txt", new string[] { });
new Thread(() => ReadFromFile()).Start();
WriteToFile();
}
Read from file method:
private static void ReadFromFile()
{
long offset = 0;
FileSystemWatcher fsw = new FileSystemWatcher
{
Path = Environment.CurrentDirectory,
Filter = "test.txt"
};
FileStream file = File.Open(
"test.txt",
FileMode.Open,
FileAccess.Read,
FileShare.Write);
StreamReader reader = new StreamReader(file);
while (true)
{
fsw.WaitForChanged(WatcherChangeTypes.Changed);
file.Seek(offset, SeekOrigin.Begin);
if (!reader.EndOfStream)
{
do
{
Console.WriteLine(reader.ReadLine());
} while (!reader.EndOfStream);
offset = file.Position;
}
}
}
Write to file method:
private static void WriteToFile()
{
for (int i = 0; i < 100; i++)
{
FileStream writeFile = File.Open(
"test.txt",
FileMode.Append,
FileAccess.Write,
FileShare.Read);
using (FileStream file = writeFile)
{
using (StreamWriter sw = new StreamWriter(file))
{
sw.WriteLine(i);
Thread.Sleep(100);
}
}
}
}