Develop Reference

C# Exception, file being use by another process [duplicate]

Writing Stringbuilder to file asynchronously. This code takes control of a file, writes a stream to it and releases it. It deals with requests from asynchronous operations, which may come in at any time.
The FilePath is set per class instance (so the lock Object is per instance), but there is potential for conflict since these classes may share FilePaths. That sort of conflict, as well as all other types from outside the class instance, would be dealt with retries.
Is this code suitable for its purpose? Is there a better way to handle this that means less (or no) reliance on the catch and retry mechanic?
Also how do I avoid catching exceptions that have occurred for other reasons.
public string Filepath { get; set; }
private Object locker = new Object();
public async Task WriteToFile(StringBuilder text)
{
int timeOut = 100;
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
while (true)
{
try
{
//Wait for resource to be free
lock (locker)
{
using (FileStream file = new FileStream(Filepath, FileMode.Append, FileAccess.Write, FileShare.Read))
using (StreamWriter writer = new StreamWriter(file, Encoding.Unicode))
{
writer.Write(text.ToString());
}
}
break;
}
catch
{
//File not available, conflict with other class instances or application
}
if (stopwatch.ElapsedMilliseconds > timeOut)
{
//Give up.
break;
}
//Wait and Retry
await Task.Delay(5);
}
stopwatch.Stop();
}

How you approach this is going to depend a lot on how frequently you're writing. If you're writing a relatively small amount of text fairly infrequently, then just use a static lock and be done with it. That might be your best bet in any case because the disk drive can only satisfy one request at a time. Assuming that all of your output files are on the same drive (perhaps not a fair assumption, but bear with me), there's not going to be much difference between locking at the application level and the lock that's done at the OS level.
So if you declare locker as:
static object locker = new object();
You'll be assured that there are no conflicts with other threads in your program.
If you want this thing to be bulletproof (or at least reasonably so), you can't get away from catching exceptions. Bad things can happen. You must handle exceptions in some way. What you do in the face of error is something else entirely. You'll probably want to retry a few times if the file is locked. If you get a bad path or filename error or disk full or any of a number of other errors, you probably want to kill the program. Again, that's up to you. But you can't avoid exception handling unless you're okay with the program crashing on error.
By the way, you can replace all of this code:
using (FileStream file = new FileStream(Filepath, FileMode.Append, FileAccess.Write, FileShare.Read))
using (StreamWriter writer = new StreamWriter(file, Encoding.Unicode))
{
writer.Write(text.ToString());
}
With a single call:
File.AppendAllText(Filepath, text.ToString());
Assuming you're using .NET 4.0 or later. See File.AppendAllText.
One other way you could handle this is to have the threads write their messages to a queue, and have a dedicated thread that services that queue. You'd have a BlockingCollection of messages and associated file paths. For example:
class LogMessage
{
public string Filepath { get; set; }
public string Text { get; set; }
}
BlockingCollection<LogMessage> _logMessages = new BlockingCollection<LogMessage>();
Your threads write data to that queue:
_logMessages.Add(new LogMessage("foo.log", "this is a test"));
You start a long-running background task that does nothing but service that queue:
foreach (var msg in _logMessages.GetConsumingEnumerable())
{
// of course you'll want your exception handling in here
File.AppendAllText(msg.Filepath, msg.Text);
}
Your potential risk here is that threads create messages too fast, causing the queue to grow without bound because the consumer can't keep up. Whether that's a real risk in your application is something only you can say. If you think it might be a risk, you can put a maximum size (number of entries) on the queue so that if the queue size exceeds that value, producers will wait until there is room in the queue before they can add.

You could also use ReaderWriterLock, it is considered to be more 'appropriate' way to control thread safety when dealing with read write operations...
To debug my web apps (when remote debug fails) I use following ('debug.txt' end up in \bin folder on the server):
public static class LoggingExtensions
{
static ReaderWriterLock locker = new ReaderWriterLock();
public static void WriteDebug(string text)
{
try
{
locker.AcquireWriterLock(int.MaxValue);
System.IO.File.AppendAllLines(Path.Combine(Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase).Replace("file:\\", ""), "debug.txt"), new[] { text });
}
finally
{
locker.ReleaseWriterLock();
}
}
}
Hope this saves you some time.

Unable to implement data parsing in a multi-threaded context using lock

I've built a program that
takes in a list of record data from a file
parses and cleans up each record in a parsing object
outputs it to an output file
So far this has worked on a single thread, but considering the fact that records can exceed 1 million in some cases, we want to implement this in a multi threading context. Multi threading is new to me in .Net, and I've given it a shot but its not working. Below I will provide more details and code:
Main Class (simplified):
public class MainClass
{
parseObject[] parseObjects;
Thread[] threads;
List<InputLineItem> inputList = new List<InputLineItem>();
FileUtils fileUtils = new FileUtils();
public GenParseUtilsThreaded(int threadCount)
{
this.threadCount = threadCount;
Init();
}
public void Init()
{
inputList = fileUtils.GetInputList();
parseObjects = new parseObject[threadCount - 1];
threads = new Thread[threadCount - 1];
InitParseObjects();
Parse();
}
private void InitParseObjects()
{
//using a ref of fileUtils to use as my lock expression
parseObjects[0] = new ParseObject(ref fileUtils);
parseObjects[0].InitValues();
for (int i = 1; i < threadCount - 1; i++)
{
parseObjects[i] = new parseObject(ref fileUtils);
parseObjects[i].InitValues();
}
}
private void InitThreads()
{
for (int i = 0; i < threadCount - 1; i++)
{
Thread t = new Thread(new ThreadStart(parseObjects[0].CleanupAndParseInput));
threads[i] = t;
}
}
public void Parse()
{
try
{
InitThreads();
int objectIndex = 0;
foreach (InputLineItem inputLineItem in inputList)
{
parseObjects[0].inputLineItem = inputLineItem;
threads[objectIndex].Start();
objectIndex++;
if (objectIndex == threadCount)
{
objectIndex = 0;
InitThreads(); //do i need to re-init the threads after I've already used them all once?
}
}
}
catch (Exception e)
{
Console.WriteLine("(286) The following error occured: " + e);
}
}
}
}
And my Parse object class (also simplified):
public class ParseObject
{
public ParserLibrary parser { get; set; }
public FileUtils fileUtils { get; set; }
public InputLineItem inputLineItem { get; set; }
public ParseObject( ref FileUtils fileUtils)
{
this.fileUtils = fileUtils;
}
public void InitValues()
{
//relevant config of parser library object occurs here
}
public void CleanupFields()
{
parser.Clean(inputLineItem.nameValue);
inputLineItem.nameValue = GetCleanupUpValueFromParser();
}
private string GetCleanupFieldValue()
{
//code to extract cleanup up value from parses
}
public void CleanupAndParseInput()
{
CleanupFields();
ParseInput();
}
public void ParseInput()
{
try
{
parser.Parse(InputLineItem.NameValue);
}
catch (Exception e)
{
}
try
{
lock (fileUtils)
{
WriteOutputToFile(inputLineItem);
}
}
catch (Exception e)
{
Console.WriteLine("(414) Failed to write to output: " + e);
}
}
public void WriteOutputToFile(InputLineItem inputLineItem)
{
//writes updated value to output file
}
}
The error I get is when trying to run the Parse function, I get this message:
An unhandled exception of type 'System.AccessViolationException' occurred in GenParse.NET.dll
Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
That being said, I feel like there's a whole lot more that I'm doing wrong here aside from what is causing that error.
I also have further questions:
Do I create multiple parse objects and iteratively feed them to each thread as I'm attempting to do, or should I use one Parse object that gets shared or cloned across each thread?
If, outside the thread, I change a value in the object that I'm passing to the thread, will that change reflect in the object passed to the thread? i.e, is the object passed by value or reference?
Is there a more efficient way for each record to be assigned to a thread and its parse object than I am currently doing with the objectIndex iterator?
THANKS!

Do I create multiple parse objects and iteratively feed them to each thread as I'm attempting to do, or should I use one Parse object that gets shared or cloned across each thread?
You initialize each thread with new ThreadStart(parseObjects[0].CleanupAndParseInput) so all threads will share the same parse object. It is a fairly safe bet that the parse objects are not threadsafe. So each thread should have a separate object. Note that this might not be sufficient, if the parse library uses any global fields it might be non-threadsafe even when using separate objects.
If, outside the thread, I change a value in the object that I'm passing to the thread, will that change reflect in the object passed to the thread? i.e, is the object passed by value or reference?
Objects (i.e. classes) are passed by reference. But any changes to an object are not guaranteed to be visible in other threads unless a memoryBarrier is issued. Most synchronization code (like lock) will issue memory barriers. Keep in mind that any non-atomic operation is unsafe if a field is written an read concurrently.
Is there a more efficient way for each record to be assigned to a thread and its parse object than I am currently doing with the objectIndex iterator?
Using manual threads in this way is very old-school. The modern, easier, and probably faster way is to use a parallel-for loop. This will try to be smart about how many threads it will use and try to adapt chunk sizes to keep the synchronization overhead low.
var items = new List<int>();
ParseObject LocalInit()
{
// Do initalization, This is run once for each thread used
return new ParseObject();
}
ParseObject ThreadMain(int value, ParallelLoopState state, ParseObject threadLocalObject)
{
// Do whatever you need to do
// This is run on multiple threads
return threadLocalObject;
}
void LocalFinally(ParseObject obj)
{
// Do Cleanup for each thread
}
Parallel.ForEach(items, LocalInit, ThreadMain, LocalFinally);
As a final note, I would advice against using multithreading unless you are familiar with the potential dangers and pitfalls it involves, at least for any project where the result is important. There are many ways to screw up and make a program that will work 99.9% of the time, and silently corrupt data the remaining 0.1% of the time.

Throttling file operations

I have a byte array that I want to persist in a file. But, I don't want to write to file each time it is updated because it can be updated very frequently. Currently I am planning to use an approach similar to following;
class ThrottleTest
{
private byte[] _internal_data = new byte[256];
CancellationTokenSource _cancel_saving = new CancellationTokenSource();
public void write_to_file()
{
Task.Delay(1000).ContinueWith((task) =>
{
File.WriteAllBytes("path/to/file.data", _internal_data);
}, _cancel_saving.Token);
}
public void operation_that_update_internal_data()
{
// cancel writing previous update
_cancel_saving.Cancel();
/*
* operate on _internal_data
*/
write_to_file();
}
public void another_operation_that_update_internal_data()
{
// cancel writing previous update
_cancel_saving.Cancel();
/*
* operate on _internal_data
*/
write_to_file();
}
}
I don't think this approach would work, because, when I cancel the token once, it will be canceled forever, so it will never write to the file.
First of all, I was wondering if I am on the right track here, and above code can be made to work. If not, what would be the best approach to achieve this behaviour. Moreover, is there a practical way to generalize it to Dictionary<string,byte[]>, where any byte[] can be modified independently?

I would start writing to file by cancelling first the previous operation.
I would also include the cancellation token in the Delay task.
CancellationTokenSource _cancel_saving;
public void write_to_file()
{
_cancel_saving?.Cancel();
_cancel_saving = new CancellationTokenSource();
Task.Delay(1000, _cancel_saving.Token).ContinueWith((task) =>
{
File.WriteAllBytes("path/to/file.data", _internal_data);
}, _cancel_saving.Token);
}

You should use Microsoft's Reactive Framework (aka Rx) - NuGet System.Reactive and add using System.Reactive.Linq; - then you can do this:
public class ThrottleTest
{
private byte[] _internal_data = new byte[256];
private Subject<Unit> _write_to_file = new Subject<Unit>();
public ThrottleTest()
{
_write_to_file
.Throttle(TimeSpan.FromSeconds(1.0))
.Subscribe(_ => File.WriteAllBytes("path/to/file.data", _internal_data));
}
public void write_to_file()
{
_write_to_file.OnNext(Unit.Default);
}
public void operation_that_update_internal_data()
{
/*
* operate on _internal_data
*/
write_to_file();
}
public void another_operation_that_update_internal_data()
{
/*
* operate on _internal_data
*/
write_to_file();
}
}

Your context seems a little odd to me. You are writing all of the bytes, and not using a stream. Putting aside your issue with the cancellation token, delaying a write by 1 second won't reduce the overall load or overall throughput to disk.
This answer has the following assumptions:
You are using an SSD and are concerned about hardware lifetime
This is a low priority activity, where some data loss will be tolerated
This is not a logging activity (otherwise an append to file would work better with a BufferedStream)
This is likely the saving of a serialized C# object tree to disk in case the power goes out
You don't want every change made to the object tree to result in a write to disk.
You don't want to write to disk every second if there has been no change to the object tree.
It should write to disk right away if there hasn't been a write for N seconds
It should wait if there has been a write recently.
Having the WriteAllBytes step as the throttle point is not ideal.
Usage:
rootObject.subObject.value = 9;
rootObject.Save(token);
Support code:
TimeSpan minimumDiskInterval = TimeSpan.FromSeconds(60);
DateTime lastSaveAt = DateTime.MinValue;
bool alreadyQueued = false;
public void Save(CancellationToken token)
{
if (alreadyQueued) //TODO: Interlocked with long instead for atomic memory barrier
return;
alreadyQueued = true; //TODO: Interlocked with long instead for atomic memory barrier
var thisSaveAt = DateTime.UtcNow;
var sinceLastSave = thisSaveAt.Subtract(lastSaveAt);
var difference = TimeSpan.TotalSeconds - sinceLastSave.TotalSeconds;
if (difference < 0)
{
//It has been a while - no need to delay
SaveNow();
}
else
{
//It was done recently
T Task.Delay(TimeSpan.FromSeconds(difference).ContinueWith((task) =>
{
SaveNow();
}, _cancel_saving.Token);
}
}
object fileAccessSync = new object();
public void SaveNow()
{
alreadyQueued = false; //TODO: Interlocked with long instead for atomic memory barrier
byte[] serializedBytes = Serialise(this)
lock (fileAccessSync)
{
File.WriteAllBytes("path/to/file.data", serializedBytes);
}
}

lock(obj) may not be working c#

I came across an issue, and I'm not sure if it's me or if there's an issue with thread locking.
I have a class I use for basic utilities. In that class is method to create or append a text file. And because I use it debug, I have the method using lock() to keep the access singular. Except, it appears to be failing and allowing multiple threads into the blocked code.
When running my test threads it doesn't throw an error every time. It's a little weird. There are 50 threads/tasks being created. Each thread is writing a line to a singe file using the class below. It cycles through about 3100 individual tasks. But a maximum of 50 tasks are created to handle each batch. As each thread completes its task, a new one is created to take its place. The last batch processed 3188 commands and threw 16 errors.
I have tried using Monitor.Enter and Exit, but I get the same results. I have also tried making the StdLibLockObj readonly. All with the same results.
Error: The process cannot access the file 'ThreadExe.txt' because it is being used by another process.
static class StdLib
{
private static object StdLibLockObj = new object();
public static void WriteLogFile(string #AFileName, string FileData, bool AppendIfExists = true, bool AddAppPath = true)
{
lock (StdLibLockObj)
{
StreamWriter sw = null;
try
{
if (AddAppPath)
{
AFileName = #Path.Combine(#ApplicationPath(), #AFileName);
}
if ((AppendIfExists) && File.Exists(AFileName))
{
sw = File.AppendText(AFileName);
}
else
{
sw = File.CreateText(AFileName);
}
sw.Write(FileData);
}
finally
{
if (sw != null)
{
sw.Flush();
sw.Close();
sw.Dispose();
}
sw = null;
}
}
}
}
My background is mostly in Delphi, where threading is a bit more granular.
Any help would be appreciated.

Wrap your StreamWriter entries in a "using" block. That will get rid of locking. Sort of like this:
public static void ErrorMessage(string logMessage)
{
using (StreamWriter sw_errors = new StreamWriter(m_errors, true))
{
sw_errors.Write("\r\nLog Entry : ");
sw_errors.WriteLine("{0} {1}", DateTime.Now.ToLongTimeString(),
DateTime.Now.ToLongDateString());
sw_errors.WriteLine(" :");
sw_errors.WriteLine(" :{0}", logMessage);
sw_errors.WriteLine("-------------------------------");
}
}

Out of memory exceptions when using MemoryStream in cache

We are dealing with a lot of files which need to be opened and close for data reads mostly.
Is it a good idea or not to cache the memorystream of each file in a temp hashtable or some other object?
We have noticed when opening files over 100MB we are running into out of memory exceptions.
We are using a wpf app.
We could successfully open the files 1 or 2 time sometimes 3 to 4 times but after that we are running into out of memory exceptions.

If you are currently caching these files, then you would expect to run out of memory quite quickly.
If you aren't caching them yet, don't, because you'll just make it worse. Perhaps you have a memory leak? Are you disposing of the memorystream once you've used it?
The best way to deal with large files is to stream data in and out (using FileStreams), so that you don't have to have the whole file in memory at once...

One issue with the MemoryStream is the internal buffer doubles in size each time the capacity is forced to increase. Even if your MemoryStream is 100MB and your file is 101MB, as soon as you try to write that last 1MB to the MemoryStream the internal buffer on MemoryStream is doubled to 200MB. You may reduce this if you give the Memory Buffer a starting capacity equal to that of you files. But this will still allow the files to use all of the memory and stop any new allocations after the some of the files are loaded. If create a cache object that is help inside of a WeakReference object you would be able to allow the garbage collector to toss a few of your cached files as needed. But don't forget you will need to add code to recreate the lost cache on demand.
public class CacheStore<TKey, TCache>
{
private static object _lockStore = new object();
private static CacheStore<TKey, TCache> _store;
private static object _lockCache = new object();
private static Dictionary<TKey, TCache> _cache =
new Dictionary<TKey, TCache>();
public TCache this[TKey index]
{
get
{
lock (_lockCache)
{
if (_cache.ContainsKey(index))
return _cache[index];
return default(TCache);
}
}
set
{
lock (_lockCache)
{
if (_cache.ContainsKey(index))
_cache.Remove(index);
_cache.Add(index, value);
}
}
}
public static CacheStore<TKey, TCache> Instance
{
get
{
lock (_lockStore)
{
if (_store == null)
_store = new CacheStore<TKey, TCache>();
return _store;
}
}
}
}
public class FileCache
{
private WeakReference _cache;
public FileCache(string fileLocation)
{
if (!File.Exists(fileLocation))
throw new FileNotFoundException("fileLocation", fileLocation);
this.FileLocation = fileLocation;
}
private MemoryStream GetStream()
{
if (!File.Exists(this.FileLocation))
throw new FileNotFoundException("fileLocation", FileLocation);
return new MemoryStream(File.ReadAllBytes(this.FileLocation));
}
public string FileLocation { get; private set; }
public MemoryStream Data
{
get
{
if (_cache == null)
_cache = new WeakReference(GetStream(), false);
var ret = _cache.Target as MemoryStream;
if (ret == null)
{
Recreated++;
ret = GetStream();
_cache.Target = ret;
}
return ret;
}
}
public int Recreated { get; private set; }
}
class Program
{
static void Main(string[] args)
{
var cache = CacheStore<string, FileCache>.Instance;
var fileName = #"c:\boot.ini";
cache[fileName] = new FileCache(fileName);
var ret = cache[fileName].Data.ToArray();
Console.WriteLine("Recreated {0}", cache[fileName].Recreated);
Console.WriteLine(Encoding.ASCII.GetString(ret));
GC.Collect();
var ret2 = cache[fileName].Data.ToArray();
Console.WriteLine("Recreated {0}", cache[fileName].Recreated);
Console.WriteLine(Encoding.ASCII.GetString(ret2));
GC.Collect();
var ret3 = cache[fileName].Data.ToArray();
Console.WriteLine("Recreated {0}", cache[fileName].Recreated);
Console.WriteLine(Encoding.ASCII.GetString(ret3));
Console.Read();
}
}

It's very dificutl say "yes" or "no", if is file content caching right in the common case and/or with so little informations. However - finited resources are real state of world, and you (as developer) must count with it. If you want cache something, you should use some mechanism for auto unloading data. In .NET framework you can use a WeakReference class, which unloads the target object (byte array and memory stream are objects too).
If you have the HW in you control, and you can use 64bit and have funds for very big RAM, you can cache big files.
However, you should be humble to resources (cpu,ram) and use the "cheap" way of implementation.

I think that the problem is that after you are done, the file is not disposed immediatly, it is waiting to the next GC cycle.
Streams are IDisposable, whice means you can and should use the using block. then the stream will dispose immidiatly when your are done dealing with it.

I don't think that caching such amount of data is a good solution, even if you don't get ever memroy overflow. Check out Memory Mapped files solution, which means that file lays on file system but speed of reading is almost equal to the in memory ones (there is an overhead for sure). Check out this link. MemoryMappedFiles
P.S. Ther are pretty good articles and examples on this topic arround in internet.
Good Luck.