Develop Reference

Is there a way to disable the write behind caching policy on a disk drive programmatically using C# in windows?

I need to remove write behind caching on the disk drives of our servers. Doing so in windows ==> Device Manager ==> Disk Drives ==> (right click) Properties ==> Policies ==> (CheckBox) Enable write behind caching | is easy but windows re-enables it automatically. I believe this happens with updates.
I want to know if I can access these policies using C# to disable them on a daily basis.
Anyone know of a way?
I have looked at DriveInfo but found no methods to access the policies.

I got 2 possible solutions, which both aim at solving the issue on the per-file level rather then OS settings level.
If you wanted to change a Policy for Policies sake, then the Windows Policy Management maay have an option for this.
WriteThrough
Now there is a whole host of caches in play here - Application/Class, OS, the RAM in the Disk Eletronics. And I am unsure if it can affect the specific cache(s) you are worried about. But FileStream has a option called "WriteThrough":
Indicates that the system should write through any intermediate cache and go directly to disk.
https://learn.microsoft.com/en-us/dotnet/api/system.io.fileoptions
The Application/Class cache is trivial and to 99% disabeled by this.
The mention of "System" indicates to me that it will get to the OS cache (wich you seem worried about). But this is only a single line in the Documentation, so I can not say with certain it goes down to OS Level. But if there is one Option I would define for logging to a file and asume to be enabeled on all existing loggers, it this is it.
Asuming this is not for logging or similar streaming work and the core issue is the danger of corrupting file with partial writes, a minor redesign can fix that issue.
Writing to Tempfile -> Move
One surefire way to avoid corrupting a file when (re)writing it, is to write to a different temporary file - and then just swap those files out with a move (wich ideally will only change the File System Name entries). I did write some example code for this not to long ago after seeing this behavior in anything down to Word Processors:
string sourcepath; //containts the source file path, set by other code
string temppath; //containts the path of the tempfile. Should be in the same folder, and thus same partiion
//Open both Streams, can use a single using for this
//The supression of any Buffering on the output should be optional and will be detrimental to performance
using(var sourceStream = File.OpenRead(sourcepath),
outStream = File.Create(temppath, 0, FileOptions.WriteThrough )){
string line = "";
//itterte over the input
while((line = streamReader.ReadLine()) != null){
//do processing on line here
outStream.Write(line);
}
}
//replace the files. Pretty sure it will just overwrite without asking
File.Move(temppath, sourcepath);
The downside is of course you will temporarily need twice the Disk Space and it will cause a lot of extra write work on some modifications.

Okay, So I found a solution for this issue.
As OlivierRogier also mentioned. This setting seems to be stored in the Registry under:
HKLM\SYSTEM\CurrentControlSet\Enum\(IDE OR SCSI)\Diskxxxxxxxxxxxx\DeviceParameters\Disk
Key : UserWriteCacheSetting
where xxxxxxx is manufacturer information.
Link HERE
I set up some code that checks this setting on a daily basis => changes it => restarts the computer. Will share the code if requested.
Thank you everyone for your input.
Blixem

How to tell a file is *completely* written

I am familiar with the FileSystemWatcher class, and have tested using this, alternatively I have tested using a fast loop and doing a directory listing of files of type in a directory. In this particular case they are zip compressed SDF files, I need to decompress, open, and query.
The problem is that when a large file is put in a directory, sometimes that takes time, such as it being downloaded, or copied from a network location, etc...
When the FileSystemWatcher raises an OnChange event, I have a handle to the ChangeType and on these types of operations the Create is immediate, while the file is still not completely copied to the location.
Likewise using the loop, I see a file is there, before the whole file is there.
The FileSystemWatcher raises several change events, one after create, and then one or more during the copy, nothing that says This file is now complete
So if I am expecting files of a type, to be placed in a directory ultimately to read and processed, with no knowledge of their transport mechanism, and no knowledge of their final size...
How do I know when the file is ready to actually be processed other than with using error control as a workflow control (albeit the error control is there anyway as it should be)? This just seems like a bad way to have to handle this, as sometimes the error control may actually be representing a legitimate issue, sometimes it may just be that the file is not completely written, and I do not see any real safe way to differentiate.
I despise anticipated error, but realize that is has its place like sockets, nothing guarantees a check for open does not change before an attempt to read/write. But I do avoid it at all costs.
This particular one troubles me mostly because of the ambiguity of the message that will be produced. There is a conflict queue for files that legitimately error because they did not come across entirely or are otherwise corrupt, I do not want otherwise good files going there. Getting more granular to detect this specific case will be almost impossible.
edit:
I know I can do this... And I have read the other SA articles concerning others doing the same thing. (And I know this method is both crude and blocking, it is just an example.)
private static void OnChanged(object source, FileSystemEventArgs e)
{
if (e.ChangeType == WatcherChangeTypes.Created)
{
bool ready = false;
while (!ready)
{
try
{
using (FileStream fs = new FileStream(e.FullPath, FileMode.Open))
{
Console.WriteLine(String.Format("{0} - {1}", e.FullPath, fs.Length));
}
ready = true;
}
catch (IOException)
{
ready = false;
}
}
}
}
What I am trying to find out is this definitively the only way, is there no other component, or some hook to the file system that will actually do this with a proper event?

The only way to tell is to open the file with FileShare.Read. That will always fail if the process is still writing to the file and hasn't closed it yet. There is otherwise no mechanism to know anything at all about which particular process is doing the writing, FSW operates at the file system device driver level and doesn't know anything about what process is performing the operation. Could be more than one.
That will very often fail the first time you try, FSW is very efficient. In general you have no idea how much time the process will take, it of course depends on how it is written and might leave the file opened for a while. Could be hours or days, a log file would be an example.
So you need a re-try mechanism, it should have an exponential back-off algorithm to increase the re-try delays between attempts. Start it off at, say, a half second delay and keep increasing that delay when it fails. This needs to be done in a worker thread, not the FSW callback. Use a thread-safe queue to pass the path of the file from the FSW callback to the worker thread. Also in general a good strategy to deal with the multiple FSW notifications you get.
Watch out for startup effects, you of course missed any notification before you started running so there might be a load of files that are waiting for work. And watch out for Heisenbugs, whatever you do with the file might cause another process to fall over. Much like this process did to yours :)
Consider that a batch-style program that you periodically run with the task scheduler could be an easier alternative.

For the one extreme, you could use a file system mini filter driver which analyzes all activities for a file at the lowest level (and communicates with a user mode application).
I wrote a proof-of-concept mini filter some time ago to detect MS Office file conversions. See below. This way, you can reliably check for every open handle to the file.
But: even this would be no universal solution for you problem:
Consider:
A tool (e.g. FTP file transfer) could in theory write part of the file, close it, and re-open it again for appending new data. This seems very curious, but you cannot reliably just check for “no more open file handles” ==> “file is ready now”
Alex K. provided a good link in his comment, and I myself would use a solution similar to the answer from Jon (https://stackoverflow.com/a/4278034/4547223)
If time is not critical (you can waste a few seconds for the decision):
Periodic timer (1 second seems reasonable)
Check file size in every timer tick
If file size did not increment for e.g. 10 seconds and there are no more FSWatcher change events too, try to open it. If you realize that the size increments take place uneven or very slowly, you could adjust the “wait time” on the fly.
Your big advantage is that you are processing ZIP files only, where you have a chance of detecting invalid (incomplete) files due to “checksum not valid”
I do not expect official ways to detect this, since there is no universal notion of “file written completely”.
File System mini filter
This may be like a sledgehammer solution for the problem.
Some time ago, I had the requirement of working around a weird bug in Office 2010, where it does not copy ADS meta data during office file conversion (ADS needed for File Classification). We discussed this with Microsoft engineers (MS was not willing to fix the bug), they complied with our filter driver solution (in the end, this was stopped since business preferred a manual workaround.)
Nevertheless, if someony really want to check if this could be a possible solution:
I have written an explanation of the steps:
https://stackoverflow.com/a/29252665/4547223

Win API NTFS OpenFileByID throwing AccessViolationException

The OpenFileByID line in test() is giving me System.AccessViolationException Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
I am trying to replicate this code example (see the answer), which I'm running in Visual Studio Express 2013 for Windows Desktop. But this example doesn't seem to work for me. It is breaking on the OpenFileByID line in test().
In a nutshell, I am getting a file's ID, then trying to create a file handle from that ID. Later on I plan to use that handle to get information about the file. The reason I'm using IDs is so that I can repair broken links, since a target file's GUID is far more reliable than its presumed location. Help appreciated!
Edit: The file I'm trying to open is an ordinary text file on my Desktop, nothing special.

You're not checking to see if you got a valid volume handle - which you might not be. Could be the source of your a/v.
When you're opening the root dir, the doc says you shouldn't use FILE_ATTRIBUTE_NORMAL with any other flags - but you're using it with FILE_FLAG_BACKUP_SEMANTICS. To use FILE_FLAG_BACKUP_SEMANTICS, you have to get privileges for SE_BACKUP_NAME. You'll have to be an admin or a backup operator to do so. I can't imagine that you need that flag.
You can get the volume handle by opening "\\.\C:" (for example)...which is different than that handle to the root folder. I usually open it with GenericRead, but if all you need it for is for OpenFileById, you can specify 0 for access.
Also - adding object IDs to files isn't necessary - the file reference number (FRN) is the master file table identifier for the file - it's the "other" kind of ID you can pass in a FILE_ID_DESCRIPTOR. You can get it from an open file handle calling GetFileInformationByHandle - it's the nFileindexHigh and nFileIndexLow made into a long int. When you move a file, the FRN stays (only it's parent FRN changes). Also, when you rename a file, the FRN doesn't change. The benefit of using this over ObjectID is that you're not altering the volume in order to track a file...and you don't have to use DeviceIOControl - which is a bit of an interop bad dream.
One more thought - OpenFileByID didn't show up until Vista and Windows Server 2008. You're there, right?

Directory.Exists sensible to time?

I have the following piece of code in my application:
if (!Directory.Exists(myPath))
Directory.CreateDirectory(myPath);
If I run it in a regular unit test sometimes it passes, sometimes not. The directory is always there (I made sure of it, so technically it will never be "created" by code). But every once in a while Directory.Exists(myPath) returned false, which makes the code try to create the folder and then I get an UnauthorizedAccessException!
The funny thing here is if I put a breakpoint on the CreateDirectory, and then move the yellow arrow up back to test, the test returns true!
What's going on?
myPath is \\nameOfLocalMachine\sharedFolder. The share is reliable and constantly used... .NET 4.0
I just made a fiddler simulate 3000 sequentials requests. 175 failed... All with the same message:
Access to the path '\nameOfLocalMachine\sharedFolder\randomFileName.json' is denied

This mishap is pretty normal on Windows. Programs open a handle on a directory like this and specify delete sharing. Which permits anybody to delete the directory, even though the program is using it. The directory won't actually disappear from the file system until that handle is closed. What follows is that trying to recreate that directory cannot work, it still exists. Windows generates an "access denied" error, reported in your C# program with the UnauthorizedAccessException.
While that sounds like an obscure feature, every program in Windows does this. Every process has a default working directory, the value of Environment.CurrentDirectory. Creating a handle on such a directory ensures that it cannot disappear while the program is using it. There are other cases, FileSystemWatcher would be another example. Or a program busy iterating the directory. Anti-malware and search indexers are notorious for hard to diagnose sources of such errors.
Otherwise a standard hazard of a multi-tasking operating system. You are not the only one using the file system. Not repeatedly deleting and creating the same directory ought to be very high on your list. If this is absolutely necessary then rename the directory first before you delete it. You'd still fail to delete the renamed directory but you won't fail recreating it. You can delete it later, next time you need to do this. Much lower odds for trouble then. Because more time passed.

IsolatedStorageSettings.ApplicationSettings are dismissed after App-Restart (WP8)

This happens with the Emulator as well as with the real device, in Debug-Mode as well as in Release-Mode.
In the app I store several application settings successfully - from simple value types to more complex objects and lists of objects.
With "WP POWER TOOLS" I can track the file "__ApplicationSettings" in the root of the IsolatedStorage. It is "well filled" - in the first lines I find some classes and assemblies, that define the complex type definitions, and below the XML starts with the <ArrayOfKeyValueOfstringanyType...>
So, everything looks normal to me so far.
When I close my app, the last piece of running code is the "Application_Closing"-Handler in App.xaml.cs. In this moment I can check the ApplicationSettings the last time - everything is okay.
For example: I check the count of the entries:
var count = System.IO.IsolatedStorage.IsolatedStorageSettings.ApplicationSettings.Count;
...and the count is right and the keys/values are right.
Then - I restart the app at once (Visual-Studio-Debugging is not interrupted) and the first piece of running code is the ctor App() in App.xaml.cs.
In the first line I check the count of ApplicationSettings-Entries again, and: it is 0 !!!
But: WP POWER TOOLS still show me, that the "__ApplicationSettings"-File is existing and is still filled like before.
(The consequence of this error is afterwards, that with the first attempt to save any setting again, the whole __ApplicationSettings-File is overwritten and contains just the one new setting.)
So what could be preventing the App from "using" the existing "__ApplicationSettings"-File???
Thanks in advance!
(PS 1: I already experienced, that all ApplicationSettings are destroyed, when there happens an Exception while saving the settings. I investigated all of that already and are 99.9% sure, that there is no Exception anymore.)
(PS 2: Just to make it clear: It is NOT the case, that the complete IsolatedStorage is gone. I have also another file for logging purposes, that I write to the root of the IsolatedStorage. This file is always there. Also the __ApplicationSettings file is not "deleted", it just seems, that the app doesn´t "read" it when launching.)

I tried the repro scenario with an app of mine and confirmed what I expected, that IsolatedStorageSettings.ApplicationSettings.Count was nonzero on entry to the App() ctor upon running the app for a second time in the same emulator process. So there is hope for you to get to this desired state too!
Since you report that the _ApplicationSettings file is not empty, I'll guess 2 possibilities: Maybe an app (or some other process?) keeps the _ApplicationSettings file open when the 2nd run of the app is trying to open the file for reading? MSFT doesn't document how the read is done, so maybe the file is opened with FileShare.None, or with FileShare.Read but some other process still has the file open for writing? I have no idea how to test this on the emulator, but on the real device you might try this scenario:
Run the app for the first time, verify it saves a non-empty _ApplicationSettings.
Restart the phone device (debugger will disconnect)
Run the app for the second time, with a breakpoint in App() ctor.
After 2) I would be confident no other process could have the file open, so the app should be able to read its contents without interference. But if you discover that it still has zero count in 3), then another possibility exists:
Maybe the restarted app encounters an error trying to deserialize the settings from the file into your data structure(s)? The error might not prevent the data from being serialized when the first run of the app is exiting.
To check this possibility, first look for error messages in the Output, Debug window. Do you see any errors when restarting the app the a second time?
If you don't see any helpful error messages, the next thing to try is to simplify the data structures being saved as settings. Try cutting down to just one setting that is a simple type like an int or string. See whether that can be restored correctly, then add more of your settings back into the file until you home in on the one which causes a problem.

Do you call Save on settings? Does it throw any error?