We recently hit some issues that may be related to the GC behavior of CLR.
The problem I encountered is as follows:
We have a long running stress testing application written in C# that keeps opening file handles on a remote SMB file share (which is Azure Files Service), and uses those handles to perform file system operations like read/write, etc.
Typically we’ll keep those handle open for quite a long time, as we’ll use them repeatedly. But sometimes when we try to access some of those opened handles, we found that these handles were closed already. And from the trace logs captured by Process Monitor (one sample below):
fltmgr.sys!FltpPerformPreCallbacks+0x324
fltmgr.sys!FltpPassThroughInternal+0x8c
fltmgr.sys!FltpPassThrough+0x169
fltmgr.sys!FltpDispatch+0x9e
ntoskrnl.exeIopCloseFile+0x146
ntoskrnl.exeObpDecrementHandleCount+0x9a
ntoskrnl.exeNtClose+0x3d9
ntoskrnl.exeKiSystemServiceCopyEnd+0x13
ntdll.dll!ZwClose+0xa
KERNELBASE.dll!CloseHandle+0x17
mscorlib.ni.dll!mscorlib.ni.dll!+0x566038
clr.dll!CallDescrWorkerInternal+0x83
clr.dll!CallDescrWorkerWithHandler+0x4a
clr.dll!DispatchCallSimple+0x60
clr.dll!SafeHandle::RunReleaseMethod+0x69
clr.dll!SafeHandle::Release+0x152
clr.dll!SafeHandle::Dispose+0x5a
clr.dll!SafeHandle::DisposeNative+0x9b
mscorlib.ni.dll!mscorlib.ni.dll!+0x48d9d1
mscorlib.ni.dll!mscorlib.ni.dll!+0x504b83
clr.dll!FastCallFinalizeWorker+0x6
clr.dll!FastCallFinalize+0x55
clr.dll!MethodTable::CallFinalizer+0xac
clr.dll!WKS::CallFinalizer+0x61
clr.dll!WKS::DoOneFinalization+0x92
clr.dll!WKS::FinalizeAllObjects+0x8f
clr.dll!WKS::FinalizeAllObjects_Wrapper+0x18
clr.dll!ManagedThreadBase_DispatchInner+0x2d
clr.dll!ManagedThreadBase_DispatchMiddle+0x6c
clr.dll!ManagedThreadBase_DispatchOuter+0x75
clr.dll!ManagedThreadBase_DispatchInCorrectAD+0x15
clr.dll!Thread::DoADCallBack+0xff
clr.dll!ManagedThreadBase_DispatchInner+0x1d822c
clr.dll!WKS::DoOneFinalization+0x145
clr.dll!WKS::FinalizeAllObjects+0x8f
clr.dll!WKS::GCHeap::FinalizerThreadWorker+0xa1
clr.dll!ManagedThreadBase_DispatchInner+0x2d
clr.dll!ManagedThreadBase_DispatchMiddle+0x6c
clr.dll!ManagedThreadBase_DispatchOuter+0x75
clr.dll!WKS::GCHeap::FinalizerThreadStart+0xd7
clr.dll!Thread::intermediateThreadProc+0x7d
KERNEL32.dll!BaseThreadInitThunk+0x1a
ntdll.dll!RtlUserThreadStart+0x1d
It seems that the handles were closed in CLR GC Finalizer thread. However, our handles are opened in the following pattern which should not be GC’ed:
We use P/Invoke to open a file handle and obtain a SafeFileHandle and use that SafeFileHandle to construct a FileStream, and we’ll save the FileStream object in another object defined as follows:
public class ScteFileHandle
{
/// <summary>
/// local file handle
/// </summary>
[NonSerialized]
public FileStream FileStreamHandle;
/*
* Some other fields
*/
}
P/Invoke we use:
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Auto)]
public static extern SafeFileHandle CreateFile(
string lpFileName,
Win32FileAccess dwDesiredAccess,
Win32FileShare dwShareMode,
IntPtr lpSecurityAttributes,
Win32FileMode dwCreationDisposition,
Win32FileAttributes dwFlagsAndAttributes,
IntPtr hTemplateFile);
SafeFileHandle fileHandle = Win32FileIO.CreateFile(fullFilePath, win32FileAccess, win32FileShare, IntPtr.Zero, win32FileMode, win32FileAttr, IntPtr.Zero);
FileStream fileStream = new FileStream(fileHandle, fileAccess, Constants.XSMBFileSectorSize);
One thing we’re sure of is that during the whole lifetime of our stress testing application, we definitely keep a reference to the ScteFileHandle object, so it will never be cleaned up by GC. However, we do have observed the SafeHandle referenced within the ScteFileHandle ‘s FileStream got finalized in CLR GC thread, as pasted in above trace log.
So I’m wondering what caused the SafeFileHandle to be GC’ed and if there’s any approach to avoid this ? I’m not familiar with the CLR GC behavior but from my perspective, the SafeFileHandle is not supposed to be GC’ed.
Any pointer or insight is greatly appreciated ! Please let me know if any other detail you need to diagnostic this issue : )
Related
I am looking for the difference between
IntPtr handle_1 = process.Handle;
Gets the native handle of the associated process.
[DllImport("kernel32.dll", SetLastError = true)]
public static extern IntPtr OpenProcess(
uint processAccess,
bool bInheritHandle,
uint processId
);
IntPtr handle_2 = OpenProcess(0x0010,false,process.Id);
If the function succeeds, the return value is an open handle to the specified process.
Both got different values. But i can still read the memory with those. I would like to understand the difference between those two, to prevent making mistakes.
I am using them in the context:
ReadProcessMemory(handle_x, addr, data, data.Length, IntPtr.Zero);
Both are process handles, and as such can be used in the same manner.
A Process object contains a process handle through its Handle property. When you call Dispose on that object, you close that handle.
When you call OpenProcess on the process' ID, you get a different handle (so it has a different value) that refers to the same process. You must separately close that handle when you're done with it (using the Win32 function CloseHandle): disposing the Process object won't magically close the handle you got from OpenProcess.
So why would you call OpenProcess when you already have a perfectly functional handle in Process? Well, access rights, really. The handle obtained by Process has PROCESS_ALL_ACCESS (i.e. full access rights). If you want an handle with fewer rights, you can use OpenProcess for that.
But really, for most purposes, there isn't really a need to mess around with native APIs when you need a process handle.
I am currently trying to write a console application in C# with two screen buffers, which should be swapped back and forth (much like VSync on a modern GPU). Since the System.Console class does not provide a way to switch buffers, I had to P/Invoke several methods from kernel32.dll.
This is my current code, grossly simplified:
static void Main(string[] args)
{
IntPtr oldBuffer = GetStdHandle(-11); //Gets the handle for the default console buffer
IntPtr newBuffer = CreateConsoleScreenBuffer(0, 0x00000001, IntPtr.Zero, 1, 0); //Creates a new console buffer
/* Write data to newBuffer */
SetConsoleActiveScreenBuffer(newBuffer);
}
The following things occured:
The screen remains empty, even though it should be displaying newBuffer
When written to oldBuffer instead of newBuffer, the data appears immediately. Thus, my way of writing into the buffer should be correct.
Upon calling SetConsoleActiveScreenBuffer(newBuffer), the error code is now 6, which means invalid handle. This is strange, as the handle is not -1, which the documentation discribes as invalid.
I should note that I very rarely worked with the Win32 API directly and have very little understanding of common Win32-related problems. I would appreciate any sort of help.
As IInspectable points out in the comments, you're setting dwDesiredAccess to zero. That gives you a handle with no access permissions. There are some edge cases where such a handle is useful, but this isn't one of them.
The only slight oddity is that you're getting "invalid handle" rather than "access denied". I'm guessing you're running Windows 7, so the handle is a user-mode object (a "pseudohandle") rather than a kernel handle.
At any rate, you need to set dwDesiredAccess to GENERIC_READ | GENERIC_WRITE as shown in the sample code.
Also, as Hans pointed out in the comments, the declaration on pinvoke.net was incorrect, specifying the last argument as a four-byte integer rather than a pointer-sized integer. I believe the correct declaration is
[DllImport("kernel32.dll", SetLastError = true)]
static extern IntPtr CreateConsoleScreenBuffer(
uint dwDesiredAccess,
uint dwShareMode,
IntPtr lpSecurityAttributes,
uint dwFlags,
IntPtr lpScreenBufferData
);
I would like to write some stuff to file, like
using( var fs = File.OpenWrite( file ) )
{
fs.Write( bytes, 0, bytes.Length );
}
However, this changes the "last write time". I can reset it later, by using
File.SetLastWriteTime( file, <old last write time> );
But in the meantime, a FileSystemWatcher already triggers.
Now my question: Is is possible to write a file without altering the "last write time"?
You can achieve it by using P/Invoke calls in Kernel32.dll.
This Powershell script from MS TechNet achieves it, and explicitly states that a FileSystemWatcher's events are not triggered.
I have briefly looked into the script and the code is pretty straightforward and can easily be copied to your C# project.
Declaration:
[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool SetFileTime(IntPtr hFile, ref long lpCreationTime, ref long lpLastAccessTime, ref long lpLastWriteTime);
The script uses SetFileTime to lock the file times before writing.
private const int64 fileTimeUnchanged = 0xFFFFFFFF;
This constant is passed as reference to the method for lpCreationTime, lpLastAccessTime and lpLastWriteTime:
// assuming fileStreamHandle is an IntPtr with the handle of the opened filestream
SetFileTime(fileStreamHandle, ref fileTimeUnchanged, ref fileTimeUnchanged, ref fileTimeUnchanged);
// Write to the file and close the stream
Don't think it's possible,nor that I'm aware of.
Also consider that "last write time" Is Not Always Updated, which leads to some wired results if you're going to pick (say) some files from the folder based on that parameter or rely on that property in some way. So it's not a parameter you can rely on in your development, it's just not reliable by architecture of OS.
Simply create a flag: boolean, if this you write and watch into the same application,
or a flag, like some specific named file, if you write form one and watch from another application.
I'm reading a file from a SQL server and writing it to disk temporarily so that another program can access it. Pretty easy with Path.GetTempFileName().
My problem is that I need these temp files deleted once the process has been completed. I'm doing this with the FileStream class, such as:
using (FileStream fs = new FileStream(filename, FileMode.Create, FileAccess.Write, FileShare.None, 8, FileOptions.DeleteOnClose))
{
//code here ultimately accessing the temp file
}
With the fileOptions set, the file is deleted once the using is finished and FileStream is disposed. The file is definitely creating and deleting on cue, but any process accessing the file responds with "The process cannot access the file because it is being used..."
Understandable if my FileStream still has access to the file (I've tried modifying the FileShare without success; this other process is not a filestream.)
I'm hoping to avoid using delete methods (which will require more error trapping). Is there a simple way to (utilizing the FileStream) create my temp file and remove it once the other process is complete?
I don't see a possibility here with the using statement, but you can use the Process.WaitForExit() method to wait for the other process to end, after which you can safely delete the file.
If you running on the Windows OS, DeleteOnClose is actually backed up by the FILE_FLAG_DELETE_ON_CLOSE flags passed to the CreateFile Function
Documentation says: "The file is to be deleted immediately after all of its handles are closed, which includes the specified handle and any other open or duplicated handles."
So you need to call DuplicateHandle (doc here) on FileStream's Handle or SafeFileHandle property. You should be able to do this in the using. Here is one possibility of DuplicateHandle declaration in C#:
[DllImport("kernel32.dll", SetLastError = true)]
public static extern bool DuplicateHandle(
IntPtr hSourceProcess,
SafeFileHandle hSourceHandle,
IntPtr hTargetProcessHandle,
out IntPtr hTargetHandle,
uint dwDesiredAccess,
bool fInheritHandle,
uint dwOptions);
I have not tested this, but It has good chances to work :-)
Close th fileStream using Close() method of the stream before the closing of using clause.
UPDATED QUESTION
Since the ctor is not supported by .NETCF (public FileStream(IntPtr handle, FileAccess access). Could you please suggest other ways of sharing large file in memory between managed and unmanaged code on a limited resource (RAM) platform. Basically I want to map the file in the upper region of 2GB user space (Win CE 5.0) outside of process space / heap. How can I do that in C#.
Also, do MemoryStream objects allocate space in heap or in memory mapped region on Win CE 5.0 ?
thanks...
ORIGINAL QUESTION
I am instantiating a FileStream Object (.NETCF , C#) using a file handle returned by native CreateFile() as below:
//P/Invoke
[DllImport("coredll.dll", SetLastError = true)]
public static extern IntPtr CreateFile(string lpFileName,
uint dwDesiredAccess,
uint dwShareMode,
IntPtr lpSecurityAttributes,
uint dwCreationDisposition,
uint dwFlagsAndAttributes,
IntPtr hTemplateFile);
// File handle received from native Win32 API
IntPtr ptr= CreateFile("myfile.txt",
0,
0,
0,
FileMode.Create,
0,
IntPtr.Zero);
//Instantiate a FileStream object using handle (returned above) as parameter.
FileStream fs = new FileStream(ptr,FileAccess.ReadWrite);
The file will grow to large size > 500 KB or more. So, my questions are:
*1) Is there anything wrong with this way of doing things given that SafeFileHandle / Handle properties are not supported in .NETCF version ? Is there any better way of doing it (I am planning to use native memory mapped file handle with FileStream / MemoryStream) ?
2) Is the memory allocated by FileStream object fall under .NETCF garbage collector ? Or given that handle is of a memory mapped file created using native API, it (managed FileStream object and its resources) is out of purview of garbage collector ?*
Thanks in advance.
Overall there is nothing wrong with this approach of using a native Create file and wrapping it in a FileStream object. This is a supported feature of FileStream.
n terms of garbage collection though there are really 2 things at play here.
The memory associated with the FileStream object. Yes this will be garbage collected
The handle which is a resource created with CreateFile. The FileStream object will take ownership of this handle and will free it when it is disposed (passively or actively).
According to the documentation, the constructor you're planning on using isn't available in .NET CF.