Right now I have a StreamWriter and a StreamReader, 1 file that holds the (text) data, at least 2 threads. 1 thread is a listener and reads the data. The other thread writes stuff into the stream.
Can I avoid using a file as the memory buffer ?
I thought it might be possible to connect the 2 streams from both ends. But dunno how. I create the writer that writes to the file. Then I start a thread that creates a reader that reads from this file and does his work. It works but I want to avoid the file thingy.
// writer
StreamWriter writer = new StreamWriter(new FileStream("text.txt", FileMode.Create, FileAccess.Write, FileShare.Read));
writer.AutoFlush = true;
// reader
StreamReader reader = new StreamReader(new FileStream("text.txt", FileMode.Open, FileAccess.Read, FileShare.ReadWrite));
I published something I called ProducerConsumerStream that will do this. It's an in-memory stream that allows one reader and one writer. It's a fixed-size circular buffer that allows a consumer to read as fast as the producer can write. See Building a new type of stream.
Related
This question already has answers here:
Cannot write to file after reading
(5 answers)
Closed 3 years ago.
This question stems from this other thread: How to lock a file with C#?
Basically let's say you want to lock a JSON file, read it, then write to it afterward, and finally unlock it. You can lock the file using the answers from the other question.
However I'm having trouble where this is allowing me to read the file, but not write to it afterward without first unlocking the file. That is, the recommended method, which seems to be fairly well-respected, is locking the same thread out of its own resource.
Example:
using (var fs = new FileStream(GetJsonPath(), FileMode.Open, FileAccess.ReadWrite,
FileShare.None))
{
SomeDtoType dto;
using (StreamReader reader = new StreamReader(fs))
{
dto = ((SomeDtoType)(new JsonSerializer()).Deserialize(reader,
typeof(SomeDtoType)));
}
// Make changes to the DTO.....
using (StreamWriter writer = new StreamWriter(fs))
{
new JsonSerializer().Serialize(writer, dto);
}
}
The using line that creates the StreamWriter throws the following exception:
Stream was not writable.
Now one thing that comes to mind is the value of FileShare.None. The problem here is that that particular enum is evidently setting lock permissions for more than just external processes.
How can you lock external threads/processes out of changing/deleting the file, yet allow your own to make these two subsequent read/write accesses?
EDIT:
Evidently moving everything into the using block for the StreamReader, then setting fs.Position to 0 between the read and the write kind of fixes the issue. The fs.Position part is fine, but having to move the write logic into the using block for the StreamReader, just so they can both use the same FileStream lock, seems a tad odd...
StreamReader closes the stream if you don't use ctor overload and instruct it don't do that:
using (StreamReader reader = new StreamReader(fs, Encoding.UTF8, true, 4096, leaveOpen:true))
There is no finalizer in StreamReader, you can move it out using block and keep as undisposed, however I'd recommend explicitly control the lifetime and behavior.
Another issue is that you'll append to the file. If you want to override a content you need to reset it before you write to it:
fs.SetLength(0);
If I have this:
StreamWriter cout = new StreamWriter("test.txt");
cout.WriteLine("XXX");
// here IOException...
StreamReader cin = new StreamReader("test.txt");
string text = cin.ReadLine();
the clr throws an IOException because I haven't close the cout.
In fact If I do this:
StreamWriter cout = new StreamWriter("test.txt");
cout.WriteLine("XXX");
cout.Close();
StreamReader cin = new StreamReader("test.txt");
string text = cin.ReadLine();
I have no exception.
But If I do this and then exit from the application:
StreamReader cin = new StreamReader("test.txt");
string text = cin.ReadLine();
without closing cin the file can from the OS opened and written.
However reading the source code of StreamReader.cs I didn't' find a destructor method (i.e. ~StreamReader(...)). So who does free that file if the garbage collector doesn't invoke Dispose and there is no finalization method?
Internally, the StreamReader uses a FileStream:
Stream stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read, DefaultFileStreamBufferSize, FileOptions.SequentialScan, Path.GetFileName(path), false, false, checkHost);
The FileStream class, being the class that ultimately accesses the file and therefore needs to guarantee cleanup, does have a finalizer, which closes the actual underlying stream. The Dispose method on the StreamReader just calls the Close on the underlying FileStream.
StreamReader and StreamWriter uses a FileStream to access the file. FileStream uses SafeFileHandle to store the underlying OS file handle. As the SafeFileHandle class controls an unmanaged resource it correctly has a finalizer (what you call a destructor) that closes the file handle.
But If I do this and then exit from the application: [...] without closing cin the file can from the OS opened and written
When a process terminates all resources used by that process is released to the operating system. It doesn't matter if your application forgot to close a file handle (even though SafeFileHandle will not "forget"). You will always observe the described behavior no matter how badly written your application is.
I just want to point out that the best way to work with StreamReader and StreamWriter and similar classes is using:
using (StreamWriter cout = new StreamWriter("test.txt")) {
cout.WriteLine("XXX");
}
using (StreamReader cin = new StreamReader("test.txt")) {
string text = cin.ReadLine();
}
This deterministically closes the files when the using block ends even if an exception is thrown while processing the file.
The operating system has a list which process (application) has what file opened. If your process terminates without explicitly closing the file, the operating system still knows that it is not any longer accessing the file and so can allow other requests to access the file.
Operation sytem frees handles and memory owned by application, if the were not freed by application if it closes. Anyway, I'm sure Stream has finalizer.
I am working in a project involving data acquisition. One very important requisite is described like this:
At the beginning of the recording, a file must be created, and its headers must be written;
As soon as the data-acquisition starts, the application should keep saving collected data to file periodically (typically once per second);
Writing consist on appending data blocks to the file, atomically if possible;
Should any error occur (program error, power failure), the file must be kept valid untill the last write before the error.
So I plan to use some Thread to watch for data received and write this data do file, but I don't know which practice is best (code below is not real, just to get the feeling):
First option: Single open
using (var fs = new FileStream(filename, FileMode.CreateNew, FileAccess.Write))
fs.Write(headers, 0, headers.Length);
using (var fs = new FileStream(filename, FileMode.Append, FileAccess.Write))
{
while (acquisitionRunning)
{
Thread.Sleep(100);
if (getNewData(out _someData;))
{
fs.Write(_someData, 0, _someData.Length);
}
}
}
Second option: multiple open:
using (var fs = new FileStream(filename, FileMode.CreateNew, FileAccess.Write))
fs.Write(headers, 0, headers.Length);
while (acquisitionRunning)
{
Thread.Sleep(100);
if (getNewData(out _someData;))
{
using (var fs = new FileStream(filename, FileMode.Append, FileAccess.Write))
{
fs.Write(_someData, 0, _someData.Length);
}
}
}
The application is supposed to work in a client machine, and file access by other processes should not be a concern. What I am most concerned about is:
Do multiple open/close impact performance (mind that typical write interval is once per second);
Which one is best to keep file integrity safe in the event of failure (including explicitly power failure)?
Is any of this forms considered a particularly good or bad practice, or either can be used depending on specifics of the problem at hand?
A good way to preserve file content in the event of a power outage/etc, is to flush the filestream after each write. This will make sure the contents you just wrote to the stream get immediately written to disk.
As you've mentioned, other processes won't be accessing the file, so keeping it open wouldn't complicate things, and also it would be faster. But, keep in mind that if the app crashes the lock will remain on the file and you might probably need to handle this accordingly to your scenario.
I have a file stream opened in writeshare and append mode from multiple processes.
Does anybody know if a single unbuffered write operation can be considered atomic?
Or have i to develop a way to synchronize different writes to ensure my data are safe?
I found my way.
You can open a filestream using this constructor.
new FileStream(FileName,
FileMode.Append,
System.Security.AccessControl.FileSystemRights.AppendData,
FileShare.ReadWrite, 4096, FileOptions.None);
using System.Security.AccessControl.FileSystemRights.AppendData parameter to open the stream, with FileMode.Append, the OS will try to write the buffer in atomic way.
If your write is bigger than buffer size, the operation will not be atomic, so you have to check your buffer size.
I'm using the Console.SetOut method to write all my Console.Out.WriteLines to a file, and this works. The only problem is that it only writes everything to the textfile when I close my application instead of it writing whenever a Console.Out.WriteLine happens.
Any ideas on how I can realise this?
How I do it:
Before Application.Run();
FileStream writerOutput = new FileStream("Logging_Admin.txt", FileMode.Append, FileAccess.Write);
StreamWriter writer = new StreamWriter(writerOutput);
Console.SetOut(writer);
After Application.Run():
writer.Dispose();
Thanks.
StreamWriter has an AutoFlush property. When set to true, you should get the result you need.
The StreamWriter will buffer its contents by default. If you want to flush the buffer you must call the Flush method:
Clears all buffers for the current writer and causes any buffered data to be written to the underlying stream.
If you don't want to call flush every time manually, you might want to consider implementing your own TextWriter-derived object to do this for you.
To flush the buffer, you can do
writer.Close();