I am working in a project involving data acquisition. One very important requisite is described like this:
At the beginning of the recording, a file must be created, and its headers must be written;
As soon as the data-acquisition starts, the application should keep saving collected data to file periodically (typically once per second);
Writing consist on appending data blocks to the file, atomically if possible;
Should any error occur (program error, power failure), the file must be kept valid untill the last write before the error.
So I plan to use some Thread to watch for data received and write this data do file, but I don't know which practice is best (code below is not real, just to get the feeling):
First option: Single open
using (var fs = new FileStream(filename, FileMode.CreateNew, FileAccess.Write))
fs.Write(headers, 0, headers.Length);
using (var fs = new FileStream(filename, FileMode.Append, FileAccess.Write))
{
while (acquisitionRunning)
{
Thread.Sleep(100);
if (getNewData(out _someData;))
{
fs.Write(_someData, 0, _someData.Length);
}
}
}
Second option: multiple open:
using (var fs = new FileStream(filename, FileMode.CreateNew, FileAccess.Write))
fs.Write(headers, 0, headers.Length);
while (acquisitionRunning)
{
Thread.Sleep(100);
if (getNewData(out _someData;))
{
using (var fs = new FileStream(filename, FileMode.Append, FileAccess.Write))
{
fs.Write(_someData, 0, _someData.Length);
}
}
}
The application is supposed to work in a client machine, and file access by other processes should not be a concern. What I am most concerned about is:
Do multiple open/close impact performance (mind that typical write interval is once per second);
Which one is best to keep file integrity safe in the event of failure (including explicitly power failure)?
Is any of this forms considered a particularly good or bad practice, or either can be used depending on specifics of the problem at hand?
A good way to preserve file content in the event of a power outage/etc, is to flush the filestream after each write. This will make sure the contents you just wrote to the stream get immediately written to disk.
As you've mentioned, other processes won't be accessing the file, so keeping it open wouldn't complicate things, and also it would be faster. But, keep in mind that if the app crashes the lock will remain on the file and you might probably need to handle this accordingly to your scenario.
Related
I receive a fair amount of binary data from an external device, about 30-40 MB/s. I need to save it to a file. On the external source device side, I have a very small buffer that I can't enlarge and as soon as the transmission stutters on the C# application side, it quickly gets clogged and I lose data.
In the application, I tried writing using FileStream, but unfortunately it is not fast enough.
_FileStream = new FileStream(FileName, FileMode.Create, FileAccess.Write);
...
void Handler_OnFtdiBytesReceived(object sender, FtdiBytesReceivedEventArgs e)
{
...
Array.Copy(e.Bytes, 0, _ReceivedDataBuffer, _ReceivedDataBufferPosition, e.NumBytesAvailable);
_ReceivedDataBufferPosition += (int)e.NumBytesAvailable;
if (_ReceivedDataBufferPosition > 0)
{
_FileStream.Write(_ReceivedDataBuffer, 0, (int)e.NumBytesAvailable);
_ReceivedDataBufferPosition = 0;
}
if (_IsOperationFinished == true)
{
_FileStream.Flush();
_FileStream.Close();
}
...
}
I also tried adding a BinaryWriter:
_FileStream = new FileStream(FileName, FileMode.Create, FileAccess.Write);
_BinaryWriter = new BinaryWriter(_FileStream);
and then:
_BinaryWriter.Write(_ReceivedDataBuffer, 0, (int)e.NumBytesAvailable);
instead of previous _FileStream.Write(...), but also the buffer on the transmit side gets clogged.
Is there any way to deal with this?
I wonder if it might help to somehow buffer the data in the computer's RAM when receiving it and, for example, when it reaches some sizable amount (say, 512 MB), start writing to the file in a separate Task, so that in the meantime, new data can be collected into the buffer continuously. Perhaps I would need to use two buffers and use them alternately, one to receive data continuously and the other from which to write to the file, and swap??
This seems quite complex code for someone with little experience, which I don't know if it will help, so I'd like to ask you for a hint first.
I'll just add at the end that I have the ability to watch the buffer fill up in this external device and by commenting out this one line regarding writing:
_FileStream.Write(_ReceivedDataBuffer, 0, (int)e.NumBytesAvailable);
I see that the problem with its clogging disappears. Earlier I also analyzed whether other code fragments might be inefficient, such as Array.Copy(...) or passing parameters via Event, but it had no effect.
I am reading source file in chunks and pass it to WCf service to write down on some remote SMB. I am keeping open the FileStream until all data is written.
Opening and closing file handle multiple time decrease performance so I am following this approach.
Once all data is written, I call CloseHandle(). Then I may need to perform some other operation on same file by calling DoSomeOperation(). As I have closed the file handle in CloseHandle() function, but I get the error "file is in use with some other process" in DoSomeOperation(). If I call DoSomeOperation() after some delay then issue is not there.
Please help us to close the file handle instantly as I call FileStream.Close().
This code snippet is part of a big program, so I can't mention all the code here.
//In WCF service
FileStream fs = null;
public void AppendBytes(string fileName, byte[] data, long position)
{
try
{
if (fs==null)//In first call, open the file handle
fs = System.IO.File.Open(fileName, System.IO.FileMode.Append, System.IO.FileAccess.Write, System.IO.FileShare.None);
fs.Write(data, 0, data.Length);
}
catch (Exception ex)
{
//Close handle in case of error
if (fs != null)
fs.Close();
}
}
public void CloseHandle()
{
//Close handle explicitly
if (fs != null)
fs.Close();
}
public void DoSomeOperation(string fileName)
{
using (FileStream fsO = System.IO.File.Open(fileName, System.IO.FileMode.Append, System.IO.FileAccess.Write, System.IO.FileShare.None))
{
//Do something with file here, this is atomic operation so I am opening FileStream with 'using' to dispose at operation is over
}
}
//In client
public void CallerFunction()
{
//Read Data from sourceFile in chunk and copy to target file using WCF.AppendBytes on another machine
WCF.AppendBytes(filename, data, pos);
WCF.CloseHandle();
WCF.DoSomeOperation(filename); //I get error here that file is in use with some other process. if I put a thread.sleep(1000) just before this statement then all works fine.
}
I have written a small test code to reproduce the same scenario on console application: Just Call TestHandleClose() from Main(), it will give error after some cycles.
static void TestHandleClose()
{
int i = 0;
try
{
if (File.Exists(#"d:\destination\file2.exe"))
File.Delete(#"d:\destination\file2.exe");
byte[] data = null;
int blocksize = 10 * 1024 * 1024;
for( i=0;i<100;i++)
{
using (FileStream fr = File.Open(#"d:\destination\File1.zip", FileMode.Open, FileAccess.Read, FileShare.None))
{
data = new byte[blocksize];
fr.Read(data, 0, blocksize); //We are reading the file single time but appending same data to target file multiple time.
using (FileStream f = File.Open(#"d:\destination\file2.exe", FileMode.Append, FileAccess.Write, FileShare.None))
{
f.Write(data, 0, data.Length); //We are writing same data multiple times.
f.Flush();
f.Close();
}
}
}
if (File.Exists(#"d:\destination\file2.exe"))
File.Delete(#"d:\destination\file2.exe");
}
catch (Exception ex)
{
throw;
}
}
The final sample code helped, I'm pretty sure I have your problem.
The thing is, even the latest sample code fails to reproduce. However, it shows another thing that you probably missed - the file you're writing is .exe. Why is this a problem? Well, there's a couple of reasons, but one of those is that when you list the directory where the .exe file is using explorer, explorer goes ahead and tries to read it (to get the icon). In this short time, the file cannot be opened with FileShare.None (and in fact, FileShare.Read probably will not help either, since it's quite likely whoever opened it didn't specify FileShare.ReadWrite).
So yet again, FileStream is closed just fine, and works well (get rid of the Flush and Close calls, though - they're wasting performance, and useless). The problem is that another process tries to read the file in the meantime. It might be some file manager like in my case (Explorer, Total Commander, ...), it might be some FileSystemWatcher you have somewhere, it might be a virus scanner (though most virus scanners nowadays use shadow copies), it might be something that automatically creates thumbnails for images, etc.. But the code you posted yourself simply doesn't cause the problem - it's someone else grabbing your file.
There's basically two options you have - either keep the file opened the whole time you need it, or treat the IOException as temporary and retry a few times in a given interval. That's what you should be doing anyway, instead of relying on the path being happy - most readers only allow concurrent reads, not writes.
In my case it was the FileShare mode that was missing. I used to read/write from multiples sources in the same file in a short period of time . Sometimes it passed good and sometimes it blocked (another process was holding the file) even though the filestream was released and i was using a lock and a using(). Everything solved when i added the FileShare mode ReadWrite.
using (FileStream fileStream = new FileStream(file, mode, access, FileShare.ReadWrite))
{...
I was searching in StackOverflow about try-finally and using blocks and what are the best practices on using them.
I read in a comment here that if your application is terminated abruptly by killing the process, a finally block will not get executed.
I was wondering, does the same apply to the using blocks? Will for example a stream get closed if a Environment.exit() call occurs inside the using block?:
//....
using (FileStream fsSource1 = new FileStream(pathSource,
FileMode.Open, FileAccess.Read))
{
//Use the stream here
Environment.exit();
}
As a second thought on it, and knowing that CLR Garbage Collector will probably take care of the stream objects if not closed properly in the program calls, is it considered necessary to close a stream in code, if the program gets terminated for sure after the completion of the stream usage?
For example, is there any practical difference between:
//....
using (FileStream fsSource1 = new FileStream(pathSource,
FileMode.Open, FileAccess.Read))
{
//Use the stream here
}
Environment.exit();
And:
//....
FileStream fsSource1 = new FileStream(pathSource, FileMode.Open, FileAccess.Read);
//Use the stream here
Environment.exit();
Or even the example mentioned before?
It shouldn't make a difference in the specific case of FileStream, modulo a tricky corner case when you used its BeginWrite() method. Its finalizer attempts to complete writing any unwritten data still present in its internal buffer. This is however not generally true, it will make a difference if you use StreamWriter for example.
What you are leaving up to the .NET Framework to decide is whether you truly meant to jerk the floor mat and seize writing the file. Or whether it should make a last-gasp attempt to flush any unwritten data. The outcome in the case of StreamWriter will tend to be an unpleasant one, non-zero odds that something is going to fall over when it tries to read a half-written file.
Always be explicit, if you want to make sure this does not happen then it is up to you to ensure that you properly called the Close() or Dispose() method. Or delete the file.
I have this code that saves a pdf file.
FileStream fs = new FileStream(SaveLocation, FileMode.Create);
fs.Write(result.DocumentBytes, 0, result.DocumentBytes.Length);
fs.Flush();
fs.Close();
It works fine. However sometimes it does not release the lock right away and that causes file locking exceptions with functions run after this one run.
Is there a ideal way to release the file lock right after the fs.Close()
Here's the ideal:
using (var fs = new FileStream(SaveLocation, FileMode.Create))
{
fs.Write(result.DocumentBytes, 0, result.DocumentBytes.Length);
}
which is roughly equivalent to:
FileStream fs = null;
try
{
fs = new FileStream(SaveLocation, FileMode.Create);
fs.Write(result.DocumentBytes, 0, result.DocumentBytes.Length);
}
finally
{
if (fs != null)
{
((IDisposable)fs).Dispose();
}
}
the using being more readable.
UPDATE:
#aron, now that I'm thinking
File.WriteAllBytes(SaveLocation, result.DocumentBytes);
looks even prettier to the eye than the ideal :-)
We have seen this same issue in production with a using() statement wrapping it.
One of the top culprits here is anti-virus software which can sneak in after the file is closed, grab it to check it doesn't contain a virus before releasing it.
But even with all anti-virus software out of the mix, in very high load systems with files stored on network shares we still saw the problem occasionally. A, cough, short Thread.Sleep(), cough, after the close seemed to cure it. If someone has a better solution I'd love to hear it!
I can't imagine why the lock would be maintained after the file is closed. But you should consider wrapping this in a using statment to ensure that the file is closed even if an exception is raised
using (FileStream fs = new FileStream(SaveLocation, FileMode.Create))
{
fs.Write(result.DocumentBytes, 0, result.DocumentBytes.Length);
}
If the functions that run after this one are part of the same application, then a better approach might be to open the file for read/write at the beginning of the entire process, and then pass the file to each function without closing it until the end of the process. Then it will be unnecessary for the application to block waiting for the IO operation to complete.
This worked for me when using .Flush() I had to add a close inside the using statement.
using (var imageFile = new FileStream(filePath, FileMode.Create, FileAccess.ReadWrite,FileShare.ReadWrite))
{
imageFile.Write(bytes, 0, bytes.Length);
imageFile.Flush();
imageFile.Close();
}
Just had the same issue when I closed a FileStream and opened the file immediately in another class. The using statement was not a solution since the FileStream had been created at another place and stored in a list. Clearing the list was not enough.
It looks like the stream needs to be freed by the garbage collector before the file can be reused. If the time between closing and opening is too short, you can use
GC.Collect();
right after you closed the stream. This worked for me.
I guess the solutions of Ian Mercer to put the thread to sleep might have the same effect, giving the GC time to free the resources.
I have an application where i sometimes need to read from file being written to and as a result being locked. As I have understood from other questions i should catch the IOException and retry until i can read.
But my question is how do i know for certain that the file is locked and that it is not another IOExcetpion that occurs.
When you open a file for reading in .NET it will at some point try to create a file handle using the CreateFile API function which sets the error code which can be used to see why it failed:
const int ERROR_SHARING_VIOLATION = 32;
try
{
using (var stream = new FileStream("test.dat", FileMode.Open, FileAccess.Read, FileShare.Read))
{
}
}
catch (IOException ex)
{
if (Marshal.GetLastWin32Error() == ERROR_SHARING_VIOLATION)
{
Console.WriteLine("The process cannot access the file because it is being used by another process.");
}
}
There's a useful discussion on google groups which you really should read. One of the options is close to darin's; however, to guarantee you get the right win32 error, you really should call the win32 OpenFile() API yourself (otherwise, you really don't know which error you are retrieving).
Another is to parse the error message: that will fail if your application is run on another language version.
A third option is to hack inside the exception class with reflection to fish out the actual HRESULT.
None of the alternatives are really that attractive: the IOException hierarchy would benefit from a few more subclasses IMHO.
To read data you can do:
using (FileStream fs = new
FileStream(fileName, FileMode.Open,
FileAccess.Read, FileShare.ReadWrite |
FileShare.Delete)) { .... }
and to save into file:
using (FileStream fs = new
FileStream(fileName, FileMode.Append,
FileAccess.Write, FileShare.Read |
FileShare.Delete)) { ... }
Flags at the end of constructors describes what other process can do with the file. It is fine, of course, if you control both write and read...
You may open it (as described by bezieur) then try to lock sections (or whole file) :
http://www.java2s.com/Code/VB/File-Directory/Lockandunlockafile.htm
Do you mean you are both reading and writing to the file? Or that an external application is writing to it.
If you are doing the reading and writing then I assume you're doing it on different threads in which case take a look at the ReaderWriteLock class which will do this the management for you, and allow you to provide timeouts.
http://msdn.microsoft.com/en-us/library/system.threading.readerwriterlock.aspx
Otherwise all you need to do is open the file in a read only mode. Then you shouldn't have any problems:
fileStream = new FileStream(fileName, FileMode.Open, FileAccess.Read));
You can compare against the type IOException to check and see if it is not something else
Such as
if (ex is FileNotFoundException)
You might want to look up the help on System.IO. A lot of the exceptions in that class inherit from IOException. Outside of checking to see if it is another type of exception you may have to look in the message of the description or you might look into making a Win32 API call into shell32.dll. There may be a function in there to check if a file is locked.
Also, if you absolutely need to wait you can use loop, but if you want to do other actions while waiting use an asynchronous thread.