I've got a program that will be running over a long duration (hours), and regularly writing output to a text file.
I'm looking to use a TextWriter implementation to write to the file, and I am concerned that keeping the file locked open during the entire length of operations may be problematic.
First question: Will there be performance problems (or other kinds) for keeping a stream open to a file for an extended duration?
If so, will a StreamWriter (opened with File Name constructor) manage opening and closing the file on a regular buffered basis for me, or will it hold the file open for the duration of its existence?
Lastly, is there a built in option for handling these more long-duration writes? Or will I need a custom Writer/Stream implementation?
Personally I would use one of the File.Appendxxx routines, which open the file, append the data and then close it again.
If I'm writing at such a rate that the cost of all this opening and closing is too high, then I add some kind of memory-based queue and flush it periodically.
If you're doing text-based logging, you might look at one of the umpteen logging frameworks for .NET, which can do this sort of stuff for you, along with things like rotating filenames, etc.
StreamWriter / FileStream, etc, will generally hold the file open until you dispose them.
Having a file opened indefinitely for it to be written to is not advised because this can cause issues when you need to back-up the file, edit the file, read the file, or even if the system crashes while it is in the process of writing to the file inevitably corrupting the data.
Will's answer does provide a solution of opening and appending the file when needed. The point here is if the file is not being written to 24/7 then there is no reason why you shouldn't close it.
To directly answer your question in the sense of the file needing to be open 24/7. I would use asynchronous methods to write to the file using tasks. This way you can then invoke another task to write a back-up of the file, say everyday. The back-up file will allow you to view the information of the production file.
Related
I'm looking for an archive file type that I can use for processing large archive files in AWS lambda. The entries in the archive are not so large by themselves, the largest maybe 100mb, but there could be a lot of them. My strategy is to create a lambda for processing each entry, where the parameters to my the lambda are a path to the file in s3, as well as a byte range for the entry inside the archive. This would allow for processing each entry without needing to load the entire file. I can write a format to handle this, but I figure something like this probably already exists.
Not required, but hoping to work with these files in C#.
As long as your files are not that big, I can suggest the following approach.
Function invoked
If there is a file in /tmp GoTo Step 4.
If there is no file in /tmp download a new file from S3.
Pop data from the file in chunks making sure that the remaining file shrinks while you process it.
Process the popped chunks of data.
If the function is about to timeout, stop processing file and invoke yourself again (call sibling). It may spawn in the same container or in a different one and will either start processing another file (remaining from some other run) or continue the same one.
When file is completely processed - mark it in some way (Tag) in S3.
There are some limitations here:
- You should not care about the order of processing the files and the rows inside files.
- Occasional multiple processing of same chunks of data should not cause any problem.
- You probably want to keep track of processed files also somewhere externally
A pretty similar approach is used in the Scheduler class of the sosw package. This is a Python package not C#, but idea could help you.
I have a program that logs it's progress and other data to an XML file. I want to be able to open this XML file without blocking out the writer program (not a .NET program, and I have no control over it), and to read the XML as it comes, waiting for more when it is all processed, until the EOF is received.
How can this be achieved in C#?
Note that there are 2 problems:
Keeping a reading stream open without blocking the other process.
Knowing when there is more input and waiting when there isn't.
If I needed to do this I would do something like the following:
Use a FileSystemWatcher to get notified when the file changes. Then just read the file and parse the XML as you require.
I would go down this route as it will be difficult to read the stream as and when the external application writes to the file.
I did soemthing similar in past yielding in an OS program called Tailf.
Just check the code if you want to do it yourself, or grab all from it, it should almost work for you as well, a part the fact I just care about text files.
You can open a file stream without locking it by passing in the following flags:
new FileStream(logFilePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
As far as waiting for the "EOF", if the other program is only writing data intermittently, you may have to put some sort of heuristics into your progress (ie. stop peeking for new data only if there's nothing new for X minutes).
I have an application that stores an XML Encrypted Document, each time the application wants to access the XML file it have to (decrypt-->read-->write-->encrypt). if a sudden shutdown or restart even a end task from task manager will result on either corrupted data in the XML or a un-encrypted XML file, so in the next run it will produce an error.
What i though about is to store the whole content of the XML in a the database and check if there are any errors then replace the old content.
Any ideas, tips, or thoughts much appreciated.
You should do the decryption/encryption in memory. Never, ever store unencrypted data on disk. Load the encrypted data, decrypt it in memory. When saving, first encrypt in memory, then store.
Replacing the entire content of a file in a way that is safe even in the event of sudden shutdowns is hard. A workaround is to create a new file under a temporary name and when it is written completely to disk, delete the old file and rename the new one. There are other implications of this though, such as security rights specific to the file being lost.
A better, but more advanced option, is to move everything into a database with transaction support.
In my web application I am working with files. Some files are very large. I use Response.Write() to write the file to the browser. This goes well for the smaller files, but for large files this can take a while and the bandwidth is fully used.
Is it possible to split large documents and send it piece by piece to the browser? Are there other ways to send the document quicker to the browser?
I hold the document as a property of an object.
Why don't you compress the file and store it in the DB and decompress it will extracting it?
You can do a lot of things depending on this questions:
How often does the file change?
Do I really need the files in the DB?
Why not store the File path in the
DB and the File on disk?
Anyhow, since your files are extremely high bandwidth and you would want your app to respond appropriately you might want to use AJAX load the files Asynchronously. You can have a WebHandler .ashx for this.
Here's a few examples:
http://www.dotnetcurry.com/ShowArticle.aspx?ID=193&AspxAutoDetectCookieSupport=1
http://www.viawindowslive.com/Articles/VirtualEarth/InvokingserversidecodeusingAJAX.aspx
My question is, is it possible to
split large documents and send it
piece by piece to the browser?
It depends on the file type, but in general no. If you are sending something like an excel file or a word doc etc. the receiving application will need all of the information (bytes) to fully form the document. You could physically separate the document into multiple ones, and that would allow you to do so.
If the bandwidth is fully used, then there is nothing you can do to "speed it up" short of compressing the document prior to send. In other words, zip it up.
Depending on the document (I know you said .mht, but we're talking content here) you will see the size go down by some amount. Maybe it's enough, maybe not.
Either way, this is entirely a function of the amount of content you want to send versus the size of the pipe available to send it. One of those is more difficult to change than the other.
Try setting IIS's dynamic compression. By default, it's set fairly low, but you can try setting it for a higher compression level and see how much that helps.
I'm not up to speed with ASP.NET but you might be able to buffer from a FileStream to some sort of output stream.
You can use the Flush method to send the currently buffered data to the client (the browser).
Note that this has some implications, as is described aptly here.
I've considered using it myself, a project sent documents that became fairly large and I was cautious about storing the whole data in memory. In the end I decided the data was not large enough to be a problem though.
Sadly the MSDN documentation is very, very vague on what Flush implies and you will probably have to use Google to troubleshoot.
Is it possible to create a file on a disk which is full??
Does creation of the file take any space??
Basically I am seeing a case where C# has created but failed to write anything whhich I think points to a full disk.
Does anyone know whether creating a file on a full disk will fail or not??
This wa done using c# o Windw xSerevr- The log file was also written to the same drive
Creating (empty) files should still be possible in most cases. The MFT is a separate part of the volume which won't get used for file data.
It should even be possible to store small amounts of data without needing more than the file entry in the MFT. NTFS can store streams as "resident data" in the stream descriptor which doesn't need any additional space, but only works for very small files.
I think your issue is another problem, though. It may be that you have permissions to create a file but not to write anything to it. You might want to check the ACLs of the location where you're trying to write.