When reading and following log files on a system where logrotate is installed, it happens at certain times that the existing log file is renamed and a new file with the same name is created. The application will then write new log entries into the new file. When I'm still reading the old file and waiting for new data to be appended there, I'll have to know when the file with that name was replaced so that I can stop reading at the end of the file and restart reading the new file.
My log reader is written in C# (.NET Core 3.1) and will run on Linux. I can use native functions through the Mono.Posix.NETStandard package. But I'm not sure how to do that properly.
Should I fetch the inode number from the file name before I start reading? Or should I compare on other data like the size or time? What's the most robust approach to detecting when the file I've currently opened for reading is replaced?
The solution should ideally still work if the log file was replaced exactly at or around the moment when I opened the file for reading, and also if the old file was very small and the new file will be big from the start (because suddenly a lot happens). I couldn't find any information about this topic at all.
Related
I am trying to make an app that make use of open data.
The data I try to read out is in a CSV format (and is about 40mb big).
I have 2 problems I can't solve.
First I having difficulties to read the file from the web.
I already read on MSDN how to read files asynchrome but it's all about local files. I want to make a list of objects. Each line (except the first line) contains all props for 1 object
Secondly when I finally managed to read the file, is there a way to save it's data and read it somehow the next time? Because 40mb is pretty big to re-download each time you open the app and it takes a lot of time.
I was wondering if it is possible that when I read the the file on the web again, it will only read and at the new lines.
I am a newbie in UWP (c#) applications, so my apologies for the questions.
Thanks in advance.
There are two APIs you can use to download a file. One is HttpClient, described here on MSDN Documentation and in a UWP sample here. This class is usually recommended for smaller files and smaller data, but can easily handler larger files as well. Its disadvantage is, that when the user closes the app, the file will stop downloading.
The alternative is BackgroundDownloader, again here on MSDN and here in UWP samples. This class is usually recommended for downloading larger files and data, as it automatically perfroms the download in the background so the download will continue even when the app is closed.
To store your files, you can use the ApplicationData.Current.LocalFolder. This is a special folder provided to you by the system for storage of application files. You have read/write access to this folder and you can not only store your files here, but even create subfolder structure using UWP StorageFile and StorageFolder APIs. More about this is on MSDN.
Suppose that I would like to add extra information about a file, without writing that information as content of that file. How would I do this? A couple of good examples are:
With Word documents, you can add Author tag to a document. And,
MP3 files have lots of info stored inside of them but when you play the file, you don't see that info (unless the program playing the file has been programmed to display that information).
How does Windows do this?
This information is stored in the file system (on windows - NTFS).
In NTFS, you can actually store another file, as part of this information, and it stores much more information about each file than you may expected.
NTFS file streams
Exapmle in C how to consume them
About MP3 and word - In these cases the information is stored inside the file, as part of its format.
I am developing a winForm application in which I have a text file for logging purpose. I want to limit its size to 3 MB so that if this limit is passed and new data is written then the oldest data in text file is deleted to make space for new data. any useful suggestion ?
I know we have some logging framework (e.g. NLog or log4net, both available from NuGet) which provides rolling log files feature. But i dont want to use it as I have just one text file in the whole project.
I wnat to read starting at the beginning of the file, delete some data and then append new data at the end.
Thanks
Rolling log files does not mean that the oldest data in a text file gets deleted. It means that after a certain time or a certain file size, a new log file is created.
Text files cannot be deleted from, they are sequential files. So if you want to remove a line from a text file, you must read the file, delete the line from memory and overwrite the file again, OR create a new file and copy line per line to it, skipping the one(s) that you want to "delete".
I have an application that stores an XML Encrypted Document, each time the application wants to access the XML file it have to (decrypt-->read-->write-->encrypt). if a sudden shutdown or restart even a end task from task manager will result on either corrupted data in the XML or a un-encrypted XML file, so in the next run it will produce an error.
What i though about is to store the whole content of the XML in a the database and check if there are any errors then replace the old content.
Any ideas, tips, or thoughts much appreciated.
You should do the decryption/encryption in memory. Never, ever store unencrypted data on disk. Load the encrypted data, decrypt it in memory. When saving, first encrypt in memory, then store.
Replacing the entire content of a file in a way that is safe even in the event of sudden shutdowns is hard. A workaround is to create a new file under a temporary name and when it is written completely to disk, delete the old file and rename the new one. There are other implications of this though, such as security rights specific to the file being lost.
A better, but more advanced option, is to move everything into a database with transaction support.
Is it possible to create a file on a disk which is full??
Does creation of the file take any space??
Basically I am seeing a case where C# has created but failed to write anything whhich I think points to a full disk.
Does anyone know whether creating a file on a full disk will fail or not??
This wa done using c# o Windw xSerevr- The log file was also written to the same drive
Creating (empty) files should still be possible in most cases. The MFT is a separate part of the volume which won't get used for file data.
It should even be possible to store small amounts of data without needing more than the file entry in the MFT. NTFS can store streams as "resident data" in the stream descriptor which doesn't need any additional space, but only works for very small files.
I think your issue is another problem, though. It may be that you have permissions to create a file but not to write anything to it. You might want to check the ACLs of the location where you're trying to write.