I have a StreamWriter with an AutoFlush = true property. However, I still see the file only partially written when I randomly open it. I'm writing a file that needs to be fully written (JSON) or not during any given time.
var sw = new StreamWriter("C:\file.txt", true /* append */, Encoding.ASCII) { AutoFlush = true };
sw.WriteLine("....");
// long running (think like a logging application) -- 1000s of seconds
sw.Close();
In between the sw.WriteLine() call and sw.Close() I want to open the file, and always have it be in the "correct data format", i.e. my line should be complete.
Current Idea:
Increase the internal buffer of FileStream (and/or StreamWriter) to let's say 128KB. Then every 128KB-1, call .Flush() on the FileStream object. This leads me to my next question, when I do call Flush(), should I right before calling it get the Stream.Position and do a File.Lock(Position, 128KB-1)? Or does Flush take care of that?
Basically I don't want the reader to be able to read the contents in between Flush(), because it'll maybe partially broken.
using (StreamWriter sw = new StreamWriter("FILEPATH"))
{
sw.WriteLine("contents");
// if you open the file now, you may see partially written lines
// since the sw is still working on it.
}
// access the file now, since the stream writer has been properly closed and disposed.
If you need a "log-like" file which is never half-written, the way to go is not keeping it open.
Every time, you want to write your file, you should instantiate a new FileWriter, which will flush the file contents upon releasing the file like this:
private void LogLikeWrite(string filePath, string contents)
{
using (StreamWriter streamWriter = new StreamWriter(filePath, true)) // the true will make you append to the file instead of overwriting its contents
{
streamWriter.Write(contents);
}
}
This way your write operations will be flushed immediately.
If you are sharing the file between processes, your going to have a race condition unless you produce a locking mechanism of some kind. See https://stackoverflow.com/a/29127380/892327. This does require that you are able to modify both processes.
An alternative is to have process A wait for a file at a specified location. Process B writes to a intermediate file and once B has flushed, the file is copied to the location process A is expecting a file to be so that it can consume the file.
Related
I have a web service that is generating random errors and I think I've found the fault.
Basically, it is reading a config file as follows, then loaded into an XmlDocument:
var config = File.ReadAllText(filename);
xmlDoc.Load(config);
However, at a later time (maybe a second later) one of the config values is updated and the file saved
xmlDoc.Save(filename);
So I'm now experiencing more errors (unfortunately the original developer added an empty try block so I can't check just now) during the first READ operation and I think it's because it's trying to read the file just as another process spawned from IIS is at the .Save part. I don't know how File.ReadAllText works and whether it will fail on a write locked file.
What's the best solution to handle this to ensure reading will always work? the value being written is just a counter and if it fails it is ignored as it's not that important but would prefer it was written. I guess I could put it into a separate config file and live with the error but I'd rather it was just one file.
Thanks.
You can use a lock to make sure that a read is completed before a write and vice verser. As in:
using System;
using System.Threading;
class Program
{
static readonly object _fileAccess = new object();
static void Write()
{
// Obtain lock and write
lock (_fileAccess)
{
// Write data to filename
xmlDoc.Save(filename);
}
}
static void Read()
{
// Obtain lock and read
lock (_fileAccess)
{
// Read some data from filename
xmlDoc.load(filename);
}
}
static void Main()
{
ThreadStart writeT = new ThreadStart(Write);
new Thread(writeT).Start();
ThreadStart readT = new ThreadStart(Read);
new Thread(readT).Start();
}
}
With the lock, the Read() must wait for the Write() to complete and Write() must wait for Read() to complete.
To answer your question about how File.ReadAllText() works, looking at the source, it uses a StreamReader internally which in turn uses a FileStream opened with FileAccess.Read and FileShare.Read, so that would prevent any other process from writing to the file (e.g. your XmlDocument.Save()) until the ReadAllText completed.
Meanwhile, your XmlDocument.Save() eventually uses FileStream opened with
FileAccess.Write and FileShare.Read, so it would allow the File.ReadAllText() as long as the Save started before the ReadAllText.
References: https://referencesource.microsoft.com/#mscorlib/system/io/streamreader.cs,a820588d8233a829
https://referencesource.microsoft.com/#System.Xml/System/Xml/Dom/XmlDocument.cs,1db4dba15523d588
I have a StreamWriter with an AutoFlush = true property. However, I still see the file only partially written when I randomly open it. I'm writing a file that needs to be fully written (JSON) or not during any given time.
var sw = new StreamWriter("C:\file.txt", true /* append */, Encoding.ASCII) { AutoFlush = true };
sw.WriteLine("....");
// long running (think like a logging application) -- 1000s of seconds
sw.Close();
In between the sw.WriteLine() call and sw.Close() I want to open the file, and always have it be in the "correct data format", i.e. my line should be complete.
Current Idea:
Increase the internal buffer of FileStream (and/or StreamWriter) to let's say 128KB. Then every 128KB-1, call .Flush() on the FileStream object. This leads me to my next question, when I do call Flush(), should I right before calling it get the Stream.Position and do a File.Lock(Position, 128KB-1)? Or does Flush take care of that?
Basically I don't want the reader to be able to read the contents in between Flush(), because it'll maybe partially broken.
Basically I don't want the reader to be able to read the contents in between Flush(), because it'll maybe partially broken.
I take it you have one open stream.
There is a writer and a reader.
The reader should not read until the flush is completed.
You'll need to use the manual lock mechanism that releases after your flush call.
You can implement in-line or if this is constantly used you can create a thread-safe stream.
I have a program that continuously writes its log to a text file.
I don't have the source code of it, so I can not modify it in any way and it is also protected with Themida.
I need to read the log file and execute some scripts depending on the content of the file.
I can not delete the file because the program that is continuously writing to it has locked the file.
So what will be the better way to read the file and only read the new lines of the file?
Saving the last line position? Or is there something that will be useful for solving it in C#?
Perhaps use the FileSystemWatcher along with opening the file with FileShare (as it is being used by another process). Hans Passant has provided a nice answer for this part here:
var fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
using (var sr = new StreamReader(fs)) {
// etc...
}
Have a look at this question and the accepted answer which may also help.
using (var fs = new FileStream("test.txt", FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete))
using (var reader = new StreamReader(fs))
{
while (true)
{
var line = reader.ReadLine();
if (!String.IsNullOrWhiteSpace(line))
Console.WriteLine("Line read: " + line);
}
}
I tested the above code and it works if you are trying to read one line at a time. The only issue is that if the line is flushed to the file before it is finished being written then you will read the line in multiple parts. As long as the logging system is writing each line all at once it should be okay.
If not then you may want to read into a buffer instead of using ReadLine, so you can parse the buffer yourself by detecting each Environment.NewLine substring.
You can just keep calling ReadToEnd() in a tight loop. Even after it reaches the end of the file it'll just return an empty string "". If some more data is written to the file it will pick it up on a subsequent call.
while (true)
{
string moreData = streamReader.ReadToEnd();
Thread.Sleep(100);
}
Bear in mind you might read partial lines this way. Also if you are dealing with very large files you will probably need another approach.
Use the filesystemwatcher to detect changes and get new lines using last read position and seek the file.
http://msdn.microsoft.com/en-us/library/system.io.filestream.seek.aspx
The log file is being "continuously" updated so you really shouldn't use FileSystemWatcher to raise an event each time the file changes. This would be triggering continuously, and you already know it will be very frequently changing.
I'd suggest using a timer event to periodically process the file. Read this SO answer for a good pattern to use System.Threading.Timer1. Keep a file stream open for reading or reopen each time and Seek to the end position of your last successful read. By "last successful read" I mean that you should encapsulate the reading and validating of a complete log line. Once you've successfully read and validated a log line, then you have a new position for the next Seek.
1 Note that System.Threading.Timer will execute on a system supplied thread that is kept in business by the ThreadPool. For short tasks this is more desirable that a dedicated thread.
Use this answer on another post c# continuously read file.
This one is quite efficient, and it checks once per second if the file size has changed. So the file is usually not read-locked as a result.
The other answers are quite valid and simple. A couple of them will read-lock the file continuously, but that's probably not a problem for most.
I have a function that is reading a file and adding some of the string in a list and returning this list. Because I wanted that nobody and nothing could change, delete or whatever the current file that I was reading I locked it. Everything was fine, I did it somehow like this:
public static List<string> Read(string myfile)
{
using (FileStream fs = File.Open(myfile, FileMode.Open, FileAccess.Read, FileShare.None))
{
//read lines, add string to a list
//return list
}
}
Thats fine. Now I have another function in another class that is doing stuff with the list and calling other functions and so on. Now sometimes I want to move the file that I was reading. And here is the problem: because Im now in a new function and the function Read(string myfile) is already processed, there is no more lock in the file.
//in another class
public static void DoStuff(/*somefile*/)
{
List<string> = Read(/*somefile*/);
//the file (somefile) is not more locked!
//do stuff
if (something)
Move(/*somefile*/) //could get an error, file maybe not more there or changed...
}
So another function/user could change the file, rename it, deleting it or whatever => Im not able to move this file. Or I will move the changed file, but I dont what that. If I would use threading, another thread with the same function could lock the file again and I could not move it.
Thats why I somehow need to lock this file for a longer time. Is there an easy way? Or do I have to replace my using (FileStream fs = File.Open(myfile, FileMode.Open, FileAccess.Read, FileShare.None) code? Any suggestions? thank you
If you want to keep the file locked for longer then you need to refactor your code so that the Stream object is kept around for longer - I would change the Read method to accept a FileStream, a little bit like this
using (FileStream fs = File.Open(myfile, FileMode.Open, FileAccess., FileShare.None))
{
List<string> = Read(fs);
if (something)
{
File.Move(/* somefile */)
}
}
The problem you are going to have is that File.Move method is going to fail as this file is already locked (by you, but File.Move doesn't know that).
Depending on what exactly it is you want to do it might be possible to work out a way of keeping the file locked while also "moving" the file, (for example if you know something in advance you could open the file specifying FileOptions.DeleteOnClose and write a new file with the same contents in the desired destination), however this isn't really the same as moving the file and so it all depends on what exactly it is you are trying to do.
In general such things are almost always going to be more trouble than they are worth - you are better off just unlocking the file just before you move it and catching/ handling any exception that is thrown as a result of the move.
The only way you could keep it locked is to keep it exclusively open, like you have done in your code.
Maybe you need to //do stuff within your using statement, and then straight after call Move
No amount of locking will prevent this. A lock only protects the data in the file. The user (or any other program) can still move or rename the file. The file's attributes, including name, time stamps and file attributes are stored separately and can be changed at will.
This is just something you'll have to deal with in any Windows program. It is rare enough that simply catching the exception is good enough to let you know that something happened to the file. The user will rarely be surprised. If you really need to know up front then you can use FileSystemWatcher to get a notification when it happens.
You are locking the file only when Read method is called.
If you want to keep it locked and release it only when you decide, make your methods OpenFile(string filename) and CloseFile(string filename). Then remove the using statement from Read method.
Open it when you start working (lock). Read it when you need it. When you have to move it, simply create a new file with the same name and copy the content. Close the original file (unlock) and delete it.
I'm writing an application that manipulates a text file. The first half of my function reads the textfile, while the second half writes to (optionally) the same file. Although I call .close() on the StreamReader object before opening the StreamWriter object, I still get a IOException: The process cannot access the file "file.txt" because it is being used by another process.
How do I force my program to release the file before continuing?
public static void manipulateFile(String fileIn, String fileOut,String obj)
{
StreamReader sr = new StreamReader(fileIn);
String line;
while ((line = sr.ReadLine()) != null)
{
//code to split up file into part1, part2, and part3[]
}
sr.Close();
//Write the file
if (fileOut != null)
{
StreamWriter sw = new StreamWriter(fileOut);
sw.Write(part1 + part2);
foreach (String s in part3)
{
sw.WriteLine(s);
}
sw.Close();
}
}
Your code as posted runs fine - I don't see the exception.
However calling Close() manually like that is a bad idea - if an exception is thrown your call to Close() might never be made. You should use a finally block, or better yet : a using statement.
using (StreamReader sr = new StreamReader(fileIn))
{
// ...
}
But the actual problem you are experiencing might not be specifically with this method, but a general problem with forgetting to close files properly in using blocks. I suggest you go through all your code base and look for all the places in your code where you use IDisposable objects and check that you dispose them correctly even when there could be exceptions.
Getting read access to a file that's already opened elsewhere isn't usually difficult. Most code would open a file for reading with FileShare.Read, allowing somebody else to read the file as well. StreamReader does so for example.
Getting write access is an entirely different ball of wax. That same FileShare.Read does not include FileShare.Write, allowing you to write the file while somebody else is reading it. That's very troublesome, you're jerking the mat out from under that somebody else, suddenly providing entirely different data.
All you have to do is find out who that 'somebody else' might be. SysInternals' Handles utility can tell you. Hopefully it is your own program, you could do something about that.
May sound like a stupid question, but are you sure you didn't edit the file with another application, which didn't release the file? I've had this situation before, mostly with Excel files where Excel didn't completely unloading from memory (or me just being dumb enough not to close the other application sometimes). Might happen with whatever application you use for .txt files, if any. Just a suggestion.