Is there a way, in C# on Windows, to write a file and flag it as temporary so that the operating system won't bother physically writing it to disk? The file in question is small, and will be read by another program in the very near future, and deleted and rewritten very soon after that, so that keeping it purely in RAM would make sense.
I'm currently writing the file with File.WriteAllText but could use a different API if necessary.
That's simple and could be done without any P/Invoke:
var file = File.Create(path);
File.SetAttributes(path, File.GetAttributes(path) | FileAttributes.Temporary);
The FileAttributes.Temporary enum value is the same as FILE_ATTRIBUTE_TEMPORARY constant (0x100):
Specifying the FILE_ATTRIBUTE_TEMPORARY attribute causes file systems
to avoid writing data back to mass storage if sufficient cache memory
is available, because an application deletes a temporary file after a
handle is closed. In that case, the system can entirely avoid writing
the data. Although it does not directly control data caching in the
same way as the previously mentioned flags, the
FILE_ATTRIBUTE_TEMPORARY attribute does tell the system to hold as
much as possible in the system cache without writing and therefore may
be of concern for certain applications.
source
Related
Lets assume that exactly 1 byte after the File-1-EOF another file (file2) starts.
If I open up file 1 and use FileStream Filemode.Append, does it overwrite file2 or does it make another copy at a place where there is enough memory?
Thanks, in regards!
Edit:
For everyone after me: I forgot that you have a file system, which is split into chunks. Making this question nonsense!
You appear to be laboring under the misapprehension that files are stored sequentially on disk, and that extending one file might overwrite parts of another file. This doesn't happen when you go via a filestream append in c#. The operating system will write the bytes you add however it likes, wherever it likes (and it likes to not overwrite other files) which is how files end up broken into smaller chunks (and why defragging is thing) scattered all over the disk. None of this is of any concern to you, because the OS presents those scattered file fragments as a single contiguous stream of bytes to any program that wants to read them
Of course, if you wrote a program that bypassed the OS and performed low level disk access, located the end of the file and then blindly write more bytes into the locations after it then you would end up damaging other files, and even the OS's carefully curated filesystem .. but a .net file stream won't make that possible
TLDR; add your bytes and don't worry about it. Keeping the filesystem in order is not your job
If I open up file 1 and use FileStream Filemode.Append, does it overwrite file2 or does it make another copy at a place where there is enough memory?
Thankfully no.
Here's a brief overview why:
Your .NET C# code does not have direct OS level interaction.
Your code is compiled into byte-code and is interpreted at runtime by the .NET runtime.
During runtime your byte-code is executed by the .NET Runtime which is built mostly in a combination of C#/C/C++.
The runtime secures what it calls SafeHandles, which are wrappers around the file handles provided by what I can assume is window.h(for WIN32 applications at least), or whatever OS level provider for file handles you're architecture is running on.
The runtime uses these handles to read and write data using the OS level API.
It is the OS's job to ensure changes to yourfile.txt, using the handle it's provided to the runtime, only affects that file.
Files are not generally stored in memory, and as such are not subject to buffer overflows.
The runtime may use a buffer in memory to.. buffer your reads and writes but that is implemented by the runtime, and has no affect on the file and operating system.
Any attempt to overflow this buffer is safe-guarded by the runtime itself and the execution of your code will stop. Regardless, if a buffer overflow happened on this buffer successfully - no extra bytes will be written to the underlying handle. Rather the runtime would likely stop executing with a memory access violation, or general unspecified behavior.
The handle you're given is little more than a token that the OS uses to keep track which file you want to read or write bytes to.
If you attempt to write more bytes to a file than an architecture allows - most operating systems will have safe guards in place to end your process, close the file, or straight up send an interrupt to crash the system.
I using WriteFile function for writing sectors on disk. How WriteFile function interacts with another data on drive or disk? How I can write file without accidentally removing another file? And is it possible that the operating system can accidentally remove my file?
When you write directly to the disk you are bypassing the file system completely. Unless you re-implement the functionality required to read and respect the file system then you can expect to destroy the disk. You will likely not only write over other files, but it is likely that you will overwrite the meta data – that is the information that describes the structure and location of the directories and files, their attributes and so on.
If the disk already contains a functioning file system and you don't want to disturb that then there is practically no scenario that I can imagine where it makes sense to write to the disk directly. If you want to write files to the disk, do just that. I suspect that you have made a mistake in your reasoning somewhere that has led you to believe that you should be writing directly to the disk.
Do you really write sectors on disk and not to a file on disk? Some background information would have been great, because if you are really writing into the raw disk surface instead of writing to a file on the disk, through the operating system, using higher level functions such as fopen(), fwrite(), or even higher that that, then you should be doing it for a good reason. Might we inquire as to what that reason is?
If you are writing disk sectors with no regards as to what filesystem the disk has, then all bets are off. Supposing that the operating system allows that, there's nothing to protect you from overwriting critical disk data or from the OS overwriting your files.
I've done that to access numbered sectors on an embedded system whose only contact to the "outside world" (the PC) was a custom hacked USB mass storage interface. And even then I broke cold sweat every time I had to do it - If my code would have accidentally written to the hard disk of the PC, it would have probably been good-bye to the OS installation on the hard disk and all the files written to it.
I was wondering, that what file extension do those memory mapped files have. Are they .dll's or something like that. Another thing is that can I use such a file, if I don't know its contents.
What file extension do memory mapped files have?
Memory mapped files can have any file extension. You can create a file mapping for any file.
Can I use such a file, if I don't know its contents?
Yes, you can create a file mapping for any file without knowing its contents.
These answers are so trivial that I suspect that you don't fully understand what a memory mapped file is and why they are useful. I suspect that the question you should have asked is: What is a memory mapped file?
Quote from MSDN
A memory-mapped file contains the contents of a file in virtual
memory. This mapping between a file and memory space enables an
application, including multiple processes, to modify the file by
reading and writing directly to the memory. Starting with the .NET
Framework 4, you can use managed code to access memory-mapped files in
the same way that native Windows functions access memory-mapped files,
as described in Managing Memory-Mapped Files in Win32 in the MSDN
Library.
There are two types of memory-mapped files:
Persisted memory-mapped files Persisted files are memory-mapped files that are
associated with a source file on a disk. When the last process has
finished working with the file, the data is saved to the source file
on the disk. These memory-mapped files are suitable for working with
extremely large source files.
Non-persisted memory-mapped files
Non-persisted files are memory-mapped files that are not associated
with a file on a disk. When the last process has finished working with
the file, the data is lost and the file is reclaimed by garbage
collection. These files are suitable for creating shared memory for
inter-process communications (IPC).
A memory-mapped file is a technique provided by the operating system that allows you to access any given file as if it were a piece of memory. The OS just maps it to a portion of the operating memory available to your process.
Nothing more, nothing less. Hence concerns regading the file's extensions and knowledge of its contents are irrelevant. However, one would expect you know what's in a file you are trying to work with.
The origin of a "memory mapped file" is from OS/2 (the predecessor to Windows NT) where it was called, "global shared memory segment" which in fact is a more accurate term for it.
These are used to share DATA IN MEMORY across applications. Such data can be saved to disk upon ALL apps that have hooks to it, have also exited (persistence)...sometimes needed, sometimes not.
For those that talk about reading in a file into memory, yes, you could do that, but WHY would you? Do you ever need to re-read the same file? If so, get what you need and load up some variables (e.g. configuration file).
This feature is really used for sharing DATA that is continually modified (by one or more apps) and read by one or more apps. Much easier and quicker to do than using a database, reading/writing to disk files, etc.
Is it possible to create a large (physically) contiguous file on a disk using C#? Preferably using only managed code, but if it's not possible then an unmanaged solution would be appreciated.
Any file you create will be logically contiguous.
If you want physically contiguous you are on OS and FS territory. Really (far) beyond the control of normal I/O API's.
But what will probably come close is to claim the space up-front: create an Empty stream and set the Length or Position property to what you need.
Writing a defragger?
It sounds like you're after the defragmentation API anyway:-
http://msdn.microsoft.com/en-us/library/aa363911%28v=vs.85%29.aspx
The link from the bottom, cos it seems you've missed the C# wrapper that someone has kindly produced.
http://blogs.msdn.com/b/jeffrey_wall/archive/2004/09/13/229137.aspx
With modern file systems it is hard to ensure a contiguous file on the hard disk. Logically the file is always contiguous, but the physical blocks that keep the data vary from file system to file system.
The best bet for this would be to use an old file system (ext2, FAT32, etc.) and just ask for a large file using seek to the file size you want and then flushing this file. More up-to-date file systems will probably mark a large file size, but won't actually write anything to the hard disk, instead returning zeros on a future read without actually reading.
int fileSize = 1024 * 1024 * 512;
FileStream file = new FileStream("C:\\MyFile", FileMode.Create, FileAccess.Write);
file.Seek(fileSize, SeekOrigin.Begin);
file.Close();
To build a database, you will need to use the scatter-gather I/O functions provided by the Windows API. This is a special type of file I/O that allows you to either "scatter" data from a file into memory or "gather" data from memory and write it to a contiguous region of a file. While the buffers into which the data is scattered or from which it is gathered need not be contiguous, the source or destination file region is always contiguous.
This functionality consists of two primary functions, both of which work asynchronously. The ReadFileScatter function reads contiguous data from a file on disk and writes it into an array of non-contiguous memory buffers. The WriteFileGather function reads non-contiguous data from memory buffers and writes it to a contiguous file on disk. Of course, you'll also need the OVERLAPPED structure that is used by both of these functions.
This is exactly what SQL Server uses when it reads and writes to the database and/or its log files, and in fact this functionality was added to an early service pack for NT 4.0 specifically for SQL Server's use.
Of course, this is pretty advanced level stuff, and hardly for the faint of heart. Surprisingly, you can actually find the P/Invoke definitions on pinvoke.net, but I have an intensely skeptical mistrust of the site. Since you'll need to spend a lot of quality time with the documentation just to understand how these functions work, you might as well write the declarations yourself. And doing it from C# will create a whole host of additional problems for you, such that I don't even recommend it. If this kind of I/O performance is important to you, I think you're using the wrong tool for the job.
The poor man's solution is contig.exe, a single-file defragmenter available for free download here.
In short no then
The OS will do this in the background, what i would do is make the file as big as you expect it to be, that way the OS will place it contigously. And if you need to grow the file, you again grow it by like 10% each time.
This is simular to how a SQL server keeps it database files.
Open opening the FileStream you open it with append.
Example:
FileStream fwriter = new FileStream("C:\\test.txt", FileMode.Append, FileAccess.Write, FileShare.Read);
I have a mixed application that has data both in a database and in a physical file store maintained by my application. As I have been developing my application I have, on occasion, run into scenarios where I am moving or deleting a file from the hard drive through my application and for whatever reason something will go wrong and an exception is thrown. Currently I simply log this and move on.
Assuming the delete or move scenario, when I throw and log the exception I now have a rouge or possibly missing file taking up space and also possibly causing presentation errors within the application. Beyond manual maintenance of the file system, what are some reliable techniques for maintaining a file system from an application?
I am particularly interested in how to make sure, no matter what, a file I call Delete() on in my application is in fact deleted.
If you are using Vista or later, you can use the Transactional File System to ensure your operations are atomic. You can find some examples at Transactional File System Operations with some wrappers and the like.
Since you are already using a database in your application, you could consider creating a table to track the file system operations. For instance you could create a row containing the details of the file system operation you are about to perform, then perform the file system operation, and upon success either delete the row or mark it completed in the database. If your application fails and/or needs to be restarted, this would provide an easy mechanism to determine which file system operations did not complete successfully and need to be retried.