I would like to write some stuff to file, like
using( var fs = File.OpenWrite( file ) )
{
fs.Write( bytes, 0, bytes.Length );
}
However, this changes the "last write time". I can reset it later, by using
File.SetLastWriteTime( file, <old last write time> );
But in the meantime, a FileSystemWatcher already triggers.
Now my question: Is is possible to write a file without altering the "last write time"?
You can achieve it by using P/Invoke calls in Kernel32.dll.
This Powershell script from MS TechNet achieves it, and explicitly states that a FileSystemWatcher's events are not triggered.
I have briefly looked into the script and the code is pretty straightforward and can easily be copied to your C# project.
Declaration:
[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool SetFileTime(IntPtr hFile, ref long lpCreationTime, ref long lpLastAccessTime, ref long lpLastWriteTime);
The script uses SetFileTime to lock the file times before writing.
private const int64 fileTimeUnchanged = 0xFFFFFFFF;
This constant is passed as reference to the method for lpCreationTime, lpLastAccessTime and lpLastWriteTime:
// assuming fileStreamHandle is an IntPtr with the handle of the opened filestream
SetFileTime(fileStreamHandle, ref fileTimeUnchanged, ref fileTimeUnchanged, ref fileTimeUnchanged);
// Write to the file and close the stream
Don't think it's possible,nor that I'm aware of.
Also consider that "last write time" Is Not Always Updated, which leads to some wired results if you're going to pick (say) some files from the folder based on that parameter or rely on that property in some way. So it's not a parameter you can rely on in your development, it's just not reliable by architecture of OS.
Simply create a flag: boolean, if this you write and watch into the same application,
or a flag, like some specific named file, if you write form one and watch from another application.
Related
I am looking for the difference between
IntPtr handle_1 = process.Handle;
Gets the native handle of the associated process.
[DllImport("kernel32.dll", SetLastError = true)]
public static extern IntPtr OpenProcess(
uint processAccess,
bool bInheritHandle,
uint processId
);
IntPtr handle_2 = OpenProcess(0x0010,false,process.Id);
If the function succeeds, the return value is an open handle to the specified process.
Both got different values. But i can still read the memory with those. I would like to understand the difference between those two, to prevent making mistakes.
I am using them in the context:
ReadProcessMemory(handle_x, addr, data, data.Length, IntPtr.Zero);
Both are process handles, and as such can be used in the same manner.
A Process object contains a process handle through its Handle property. When you call Dispose on that object, you close that handle.
When you call OpenProcess on the process' ID, you get a different handle (so it has a different value) that refers to the same process. You must separately close that handle when you're done with it (using the Win32 function CloseHandle): disposing the Process object won't magically close the handle you got from OpenProcess.
So why would you call OpenProcess when you already have a perfectly functional handle in Process? Well, access rights, really. The handle obtained by Process has PROCESS_ALL_ACCESS (i.e. full access rights). If you want an handle with fewer rights, you can use OpenProcess for that.
But really, for most purposes, there isn't really a need to mess around with native APIs when you need a process handle.
We recently hit some issues that may be related to the GC behavior of CLR.
The problem I encountered is as follows:
We have a long running stress testing application written in C# that keeps opening file handles on a remote SMB file share (which is Azure Files Service), and uses those handles to perform file system operations like read/write, etc.
Typically we’ll keep those handle open for quite a long time, as we’ll use them repeatedly. But sometimes when we try to access some of those opened handles, we found that these handles were closed already. And from the trace logs captured by Process Monitor (one sample below):
fltmgr.sys!FltpPerformPreCallbacks+0x324
fltmgr.sys!FltpPassThroughInternal+0x8c
fltmgr.sys!FltpPassThrough+0x169
fltmgr.sys!FltpDispatch+0x9e
ntoskrnl.exeIopCloseFile+0x146
ntoskrnl.exeObpDecrementHandleCount+0x9a
ntoskrnl.exeNtClose+0x3d9
ntoskrnl.exeKiSystemServiceCopyEnd+0x13
ntdll.dll!ZwClose+0xa
KERNELBASE.dll!CloseHandle+0x17
mscorlib.ni.dll!mscorlib.ni.dll!+0x566038
clr.dll!CallDescrWorkerInternal+0x83
clr.dll!CallDescrWorkerWithHandler+0x4a
clr.dll!DispatchCallSimple+0x60
clr.dll!SafeHandle::RunReleaseMethod+0x69
clr.dll!SafeHandle::Release+0x152
clr.dll!SafeHandle::Dispose+0x5a
clr.dll!SafeHandle::DisposeNative+0x9b
mscorlib.ni.dll!mscorlib.ni.dll!+0x48d9d1
mscorlib.ni.dll!mscorlib.ni.dll!+0x504b83
clr.dll!FastCallFinalizeWorker+0x6
clr.dll!FastCallFinalize+0x55
clr.dll!MethodTable::CallFinalizer+0xac
clr.dll!WKS::CallFinalizer+0x61
clr.dll!WKS::DoOneFinalization+0x92
clr.dll!WKS::FinalizeAllObjects+0x8f
clr.dll!WKS::FinalizeAllObjects_Wrapper+0x18
clr.dll!ManagedThreadBase_DispatchInner+0x2d
clr.dll!ManagedThreadBase_DispatchMiddle+0x6c
clr.dll!ManagedThreadBase_DispatchOuter+0x75
clr.dll!ManagedThreadBase_DispatchInCorrectAD+0x15
clr.dll!Thread::DoADCallBack+0xff
clr.dll!ManagedThreadBase_DispatchInner+0x1d822c
clr.dll!WKS::DoOneFinalization+0x145
clr.dll!WKS::FinalizeAllObjects+0x8f
clr.dll!WKS::GCHeap::FinalizerThreadWorker+0xa1
clr.dll!ManagedThreadBase_DispatchInner+0x2d
clr.dll!ManagedThreadBase_DispatchMiddle+0x6c
clr.dll!ManagedThreadBase_DispatchOuter+0x75
clr.dll!WKS::GCHeap::FinalizerThreadStart+0xd7
clr.dll!Thread::intermediateThreadProc+0x7d
KERNEL32.dll!BaseThreadInitThunk+0x1a
ntdll.dll!RtlUserThreadStart+0x1d
It seems that the handles were closed in CLR GC Finalizer thread. However, our handles are opened in the following pattern which should not be GC’ed:
We use P/Invoke to open a file handle and obtain a SafeFileHandle and use that SafeFileHandle to construct a FileStream, and we’ll save the FileStream object in another object defined as follows:
public class ScteFileHandle
{
/// <summary>
/// local file handle
/// </summary>
[NonSerialized]
public FileStream FileStreamHandle;
/*
* Some other fields
*/
}
P/Invoke we use:
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Auto)]
public static extern SafeFileHandle CreateFile(
string lpFileName,
Win32FileAccess dwDesiredAccess,
Win32FileShare dwShareMode,
IntPtr lpSecurityAttributes,
Win32FileMode dwCreationDisposition,
Win32FileAttributes dwFlagsAndAttributes,
IntPtr hTemplateFile);
SafeFileHandle fileHandle = Win32FileIO.CreateFile(fullFilePath, win32FileAccess, win32FileShare, IntPtr.Zero, win32FileMode, win32FileAttr, IntPtr.Zero);
FileStream fileStream = new FileStream(fileHandle, fileAccess, Constants.XSMBFileSectorSize);
One thing we’re sure of is that during the whole lifetime of our stress testing application, we definitely keep a reference to the ScteFileHandle object, so it will never be cleaned up by GC. However, we do have observed the SafeHandle referenced within the ScteFileHandle ‘s FileStream got finalized in CLR GC thread, as pasted in above trace log.
So I’m wondering what caused the SafeFileHandle to be GC’ed and if there’s any approach to avoid this ? I’m not familiar with the CLR GC behavior but from my perspective, the SafeFileHandle is not supposed to be GC’ed.
Any pointer or insight is greatly appreciated ! Please let me know if any other detail you need to diagnostic this issue : )
I'm working on a project, which requires to copy a lot of files and directories, while preserving their original timestamps. So I need to make many calls to the target's SetCreationTime(), SetLastWriteTime() and SetLastAccessTime() methods in order to copy the original values from source to target. As the screenshot below shows, these simple operations take up to 42% of the total computation time.
Since this is limiting my whole application's performance tremendously, I'd like to speed things up. I assume, that each of these calls requires to open and close a new stream to the file/directory. If that's the reason, I'd like to leave this stream open until I finished writing all attributes. How do I accomplish this? I guess this would require the use of some P/Invoke.
Update:
I followed Lukas' advice to use the WinAPI method CreateFile(..) with the FILE_WRITE_ATTRIBUTES. In order to P/Invoke the mentioned method I created following wrapper:
public class Win32ApiWrapper
{
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Auto)]
private static extern SafeFileHandle CreateFile(string lpFileName,
[MarshalAs(UnmanagedType.U4)] FileAccess dwDesiredAccess,
[MarshalAs(UnmanagedType.U4)] FileShare dwShareMode,
IntPtr lpSecurityAttributes,
[MarshalAs(UnmanagedType.U4)] FileMode dwCreationDisposition,
[MarshalAs(UnmanagedType.U4)] FileAttributes dwFlagsAndAttributes,
IntPtr hTemplateFile);
public static SafeFileHandle CreateFileGetHandle(string path, int fileAttributes)
{
return CreateFile(path,
(FileAccess)(EFileAccess.FILE_WRITE_ATTRIBUTES | EFileAccess.FILE_WRITE_DATA),
0,
IntPtr.Zero,
FileMode.Create,
(FileAttributes)fileAttributes,
IntPtr.Zero);
}
}
The enums I used can be found here.This allowed me to do all all things with only opening the file once: Create the file, apply all attributes, set the timestamps and copy the actual content from the original file.
FileInfo targetFile;
int fileAttributes;
IDictionary<string, long> timeStamps;
using (var hFile = Win32ApiWrapper.CreateFileGetHandle(targetFile.FullName, attributeFlags))
using (var targetStream = new FileStream(hFile, FileAccess.Write))
{
// copy file
Win32ApiWrapper.SetFileTime(hFile, timeStamps);
}
Was it worth the effort? YES. It reduced computation time by ~40% from 86s to 51s.
Results before optimization:
Results after optimization:
I'm not a C# programmer and I don't know how those System.IO.FileSystemInfo methods are implemented. But I've made a few tests with the WIN32 API function SetFileTime(..) which will be called by C# at some point.
Here is the code snippet of my benchmark-loop:
#define NO_OF_ITERATIONS 100000
int iteration;
DWORD tStart;
SYSTEMTIME tSys;
FILETIME tFile;
HANDLE hFile;
DWORD tEllapsed;
iteration = NO_OF_ITERATIONS;
GetLocalTime(&tSys);
tStart = GetTickCount();
while (iteration)
{
tSys.wYear++;
if (tSys.wYear > 2020)
{
tSys.wYear = 2000;
}
SystemTimeToFileTime(&tSys, &tFile);
hFile = CreateFile("test.dat",
GENERIC_WRITE, // FILE_WRITE_ATTRIBUTES
0,
NULL,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,
NULL);
if (hFile == INVALID_HANDLE_VALUE)
{
printf("CreateFile(..) failed (error: %d)\n", GetLastError());
break;
}
SetFileTime(hFile, &tFile, &tFile, &tFile);
CloseHandle(hFile);
iteration--;
}
tEllapsed = GetTickCount() - tStart;
I've seen that the expensive part of setting the file times is the opening/closing of the file. About 60% of the time is used to open the file and about 40% to close it (which needs to flush the modifications to disc). The above loop took about 9s for 10000 iterations.
A little research showed that calling CreateFile(..) with FILE_WRITE_ATTRIBUTES (instead of GENERIC_WRITE) is sufficient to change the time attributes of a file.
This modification speed things up significantly! Now the same loop finishes within 2s for 10000 iterations. Since the number of iterations is quite small I've made a second run with 100000 iterations to get a more reliable time measurement:
FILE_WRITE_ATTRIBUTES: 5 runs with 100000 iterations: 12.7-13.2s
GENERIC_WRITE: 5 runs with 100000 iterations: 63.2-72.5s
Based on the above numbers my guess is that the C# methods use the wrong access mode when opening the file to change to file time. Or some other C# behavior slows things down...
So maybe a solution to your speed issue is to implement a DLL which exports a C function which changes the file times using SetFileTime(..)? Or maybe you can even import the functions CreateFile(..), SetFileTime(..) and CloseHandle(..) directly to avoid calling the C# methods?
Good luck!
everyone,i have a lot of files write to disk per seconds,i want to disable disk cache to improve performance,i google search find a solution:win32 CreateFile method with FILE_FLAG_NO_BUFFERING and How to empty/flush Windows READ disk cache in C#?.
i write a little of code to test whether can worked:
const int FILE_FLAG_NO_BUFFERING = unchecked((int)0x20000000);
[DllImport("KERNEL32", SetLastError = true, CharSet = CharSet.Auto, BestFitMapping = false)]
static extern SafeFileHandle CreateFile(
String fileName,
int desiredAccess,
System.IO.FileShare shareMode,
IntPtr securityAttrs,
System.IO.FileMode creationDisposition,
int flagsAndAttributes,
IntPtr templateFile);
static void Main(string[] args)
{
var handler = CreateFile(#"d:\temp.bin", (int)FileAccess.Write, FileShare.None,IntPtr.Zero, FileMode.Create, FILE_FLAG_NO_BUFFERING, IntPtr.Zero);
var stream = new FileStream(handler, FileAccess.Write, BlockSize);//BlockSize=4096
byte[] array = Encoding.UTF8.GetBytes("hello,world");
stream.Write(array, 0, array.Length);
stream.Close();
}
when running this program,the application get exception:IO operation will not work. Most likely the file will become too long or the handle was not opened to support synchronous IO operations
later,i found this article When you create an object with constraints, you have to make sure everybody who uses the object understands those constraints,but i can't fully understand,so i change my code to test:
var stream = new FileStream(handler, FileAccess.Write, 4096);
byte[] ioBuffer = new byte[4096];
byte[] array = Encoding.UTF8.GetBytes("hello,world");
Array.Copy(array, ioBuffer, array.Length);
stream.Write(ioBuffer, 0, ioBuffer.Length);
stream.Close();
it's running ok,but i just want "hello,world" bytes not all.i trying change blocksize to 1 or other integer(not 512 multiple) get same error.i also try win32 WriteFile api also get same error.someone can help me?
CreateFile() function in No Buffering mode imposes strict requirements on what may and what may not be done. Having a buffer of certain size (multiple of device sector size) is one of them.
Now, you can improve file writes in this way only if you use buffering in your code. If you want to write 10 bytes without buffering, then No Buffering mode won't help you.
If I understood your requirements correctly, this is what I'd try first:
Create a queue with objects that have the data in memory and the target file on the disk.
You start writing the files first just into memory, and then on another thread start going through the queue, opening io-completion port based filestream handles (isAsync=True) - just don't open too many of them as at some point you'll probably start losing perf due to cache trashing etc. You need to profile to see what is optimal amount for your system and ssd's.
After each open, you can use the async filestream methods Begin... to start writing data from memory to the files. the isAsync puts some requirements so this may not be as easy to get working in every corner case as using filestream normally.
Whether there will be any improvement to using another thread to create the files and another to write to them using the async api, that might only be the case if there is a possibility that creating/opening the files would block. SSD's perform various things internally to keep the access to data fast, so when you start doing this sort of extreme performance stuff, there may be pronounced differences between SSD controllers. It's also possible that if the controller drivers aren't well implemented, OS/Windows may start to feel sluggish or freeze. The hardware benchmarks sites do not really stress this particular kind of scenario (eg. create and write x KB into million files asap) and no doubt there's some drivers out there that are slower than others.
I'm reading a file from a SQL server and writing it to disk temporarily so that another program can access it. Pretty easy with Path.GetTempFileName().
My problem is that I need these temp files deleted once the process has been completed. I'm doing this with the FileStream class, such as:
using (FileStream fs = new FileStream(filename, FileMode.Create, FileAccess.Write, FileShare.None, 8, FileOptions.DeleteOnClose))
{
//code here ultimately accessing the temp file
}
With the fileOptions set, the file is deleted once the using is finished and FileStream is disposed. The file is definitely creating and deleting on cue, but any process accessing the file responds with "The process cannot access the file because it is being used..."
Understandable if my FileStream still has access to the file (I've tried modifying the FileShare without success; this other process is not a filestream.)
I'm hoping to avoid using delete methods (which will require more error trapping). Is there a simple way to (utilizing the FileStream) create my temp file and remove it once the other process is complete?
I don't see a possibility here with the using statement, but you can use the Process.WaitForExit() method to wait for the other process to end, after which you can safely delete the file.
If you running on the Windows OS, DeleteOnClose is actually backed up by the FILE_FLAG_DELETE_ON_CLOSE flags passed to the CreateFile Function
Documentation says: "The file is to be deleted immediately after all of its handles are closed, which includes the specified handle and any other open or duplicated handles."
So you need to call DuplicateHandle (doc here) on FileStream's Handle or SafeFileHandle property. You should be able to do this in the using. Here is one possibility of DuplicateHandle declaration in C#:
[DllImport("kernel32.dll", SetLastError = true)]
public static extern bool DuplicateHandle(
IntPtr hSourceProcess,
SafeFileHandle hSourceHandle,
IntPtr hTargetProcessHandle,
out IntPtr hTargetHandle,
uint dwDesiredAccess,
bool fInheritHandle,
uint dwOptions);
I have not tested this, but It has good chances to work :-)
Close th fileStream using Close() method of the stream before the closing of using clause.