Develop Reference

Multiple users writing at the same file

I have a project which is a Web API project, my project is accessed by multiple users (i mean a really-really lot of users). When my project being accessed from frontend (web page using HTML 5), and user doing something like updating or retrieving data, the backend app (web API) will write a single log file (a .log file but the content is JSON).
The problem is, when being accessed by multiple users, the frontend became unresponsive (always loading). The problem is in writing process of the log file (single log file being accessed by a really-really lot of users). I heard that using a multi threading technique can solve the problem, but i don't know which method. So, maybe anyone can help me please.
Here is my code (sorry if typo, i use my smartphone and mobile version of stack overflow):
public static void JsonInputLogging<T>(T m, string methodName)
{
MemoryStream ms = new MemoryStream();
DataContractJsonSerializer ser = new
DataContractJsonSerializer(typeof(T));
ser.WriteObject(ms, m);
string jsonString = Encoding.UTF8.GetString(ms.ToArray());
ms.Close();
logging("MethodName: " + methodName + Environment.NewLine + jsonString.ToString());
}
public static void logging (string message)
{
string pathLogFile = "D:\jsoninput.log";
FileInfo jsonInputFile = new FileInfo(pathLogFile);
if (File.Exists(jsonInputFile.ToString()))
{
long fileLength = jsonInputFile.Length;
if (fileLength > 1000000)
{
File.Move(pathLogFile, pathLogFile.Replace(*some new path*);
}
}
File.AppendAllText(pathLogFile, *some text*);
}

You have to understand some internals here first. For each [x] users, ASP.Net will use a single worker process. One worker process holds multiple threads. If you're using multiple instances on the cloud, it's even worse because then you also have multiple server instances (I assume this ain't the case).
A few problems here:
You have multiple users and therefore multiple threads.
Multiple threads can deadlock each other writing the files.
You have multiple appdomains and therefore multiple processes.
Multiple processes can lock out each other
Opening and locking files
File.Open has a few flags for locking. You can basically lock files exclusively per process, which is a good idea in this case. A two-step approach with Exists and Open won't help, because in between another worker process might do something. Bascially the idea is to call Open with write-exclusive access and if it fails, try again with another filename.
This basically solves the issue with multiple processes.
Writing from multiple threads
File access is single threaded. Instead of writing your stuff to a file, you might want to use a separate thread to do the file access, and multiple threads that tell the thing to write.
If you have more log requests than you can handle, you're in the wrong zone either way. In that case, the best way to handle it for logging IMO is to simply drop the data. In other words, make the logger somewhat lossy to make life better for your users. You can use the queue for that as well.
I usually use a ConcurrentQueue for this and a separate thread that works away all the logged data.
This is basically how to do this:
// Starts the worker thread that gets rid of the queue:
internal void Start()
{
loggingWorker = new Thread(LogHandler)
{
Name = "Logging worker thread",
IsBackground = true,
Priority = ThreadPriority.BelowNormal
};
loggingWorker.Start();
}
We also need something to do the actual work and some variables that are shared:
private Thread loggingWorker = null;
private int loggingWorkerState = 0;
private ManualResetEventSlim waiter = new ManualResetEventSlim();
private ConcurrentQueue<Tuple<LogMessageHandler, string>> queue =
new ConcurrentQueue<Tuple<LogMessageHandler, string>>();
private void LogHandler(object o)
{
Interlocked.Exchange(ref loggingWorkerState, 1);
while (Interlocked.CompareExchange(ref loggingWorkerState, 1, 1) == 1)
{
waiter.Wait(TimeSpan.FromSeconds(10.0));
waiter.Reset();
Tuple<LogMessageHandler, string> item;
while (queue.TryDequeue(out item))
{
writeToFile(item.Item1, item.Item2);
}
}
}
Basically this code enables you to work away all the items from a single thread using a queue that's shared across threads. Note that ConcurrentQueue doesn't use locks for TryDequeue, so clients won't feel any pain because of this.
Last thing that's needed is to add stuff to the queue. That's the easy part:
public void Add(LogMessageHandler l, string msg)
{
if (queue.Count < MaxLogQueueSize)
{
queue.Enqueue(new Tuple<LogMessageHandler, string>(l, msg));
waiter.Set();
}
}
This code will be called from multiple threads. It's not 100% correct because Count and Enqueue don't necessarily have to be called in a consistent way - but for our intents and purposes it's good enough. It also doesn't lock in the Enqueue and the waiter will ensure that the stuff is removed by the other thread.
Wrap all this in a singleton pattern, add some more logic to it, and your problem should be solved.

That can be problematic, since every client request handled by new thread by default anyway. You need some "root" object that is known across the project (don't think you can achieve this in static class), so you can lock on it before you access the log file. However, note that it will basically serialize the requests, and probably will have a very bad effect on performance.

No multi-threading does not solve your problem. How are multiple threads supposed to write to the same file at the same time? You would need to care about data consistency and I don't think that's the actual problem here.
What you search is asynchronous programming. The reason your GUI becomes unresponsive is, that it waits for the tasks to complete. If you know, the logger is your bottleneck then use async to your advantage. Fire the log method and forget about the outcome, just write the file.
Actually I don't really think your logger is the problem. Are you sure there is no other logic which blocks you?

Prevent multiple threads writing the same file

i am designing and developing an api where multiple threads are downloading files from the net and then write it to disk.
if it is used incorrectly it could happen that the same file is downloaded and written by more than one threads, which will lead to an exception at the moment of writing to disk.
i would like to avoid this problem with a lock() { ... } around the part that writes the file, but obviously i dont want to lock with a global object, just something that is related to that specific file so that not all threads are locked when a file is written.
i hope this question is understandable.

So what you want to be able to do is synchronize a bunch of actions based no a given key. In this case, that key can be an absolute file name. We can implement this as a dictionary that maps a key to some synchronization object. This could be either an object to lock on, if we want to implement a blocking synchronization mechanism, or a Task if we want to represent an asynchronous method of running the code when appropriate; I went with the later. I also went with a ConcurrentDictionary to let it handle the synchronization, rather than handling it manually, and used Lazy to ensure that each task was created exactly once:
public class KeyedSynchronizer<TKey>
{
private ConcurrentDictionary<TKey, Lazy<Task>> dictionary;
public KeyedSynchronizer(IEqualityComparer<TKey> comparer = null)
{
dictionary = new ConcurrentDictionary<TKey, Lazy<Task>>(
comparer ?? EqualityComparer<TKey>.Default);
}
public Task ActOnKey(TKey key, Action action)
{
var dictionaryValue = dictionary.AddOrUpdate(key,
new Lazy<Task>(() => Task.Run(action)),
(_, task) => new Lazy<Task>(() =>
task.Value.ContinueWith(t => action())));
return dictionaryValue.Value;
}
public static readonly KeyedSynchronizer<TKey> Default =
new KeyedSynchronizer<TKey>();
}
You can now create an instance of this synchronizer, and then specify actions along with the keys (files) that they correspond to. You can be confident that the actions won't be executed until any previous actions on that file have completed. If you want to wait until that action completes, then you can Wait on the task, if you don't have any need to wait, then you can just not. This also allows you to do your processing asynchronously by awaiting the task.

You may consider using ReaderWriterLockSlim
http://msdn.microsoft.com/en-us/library/system.threading.readerwriterlockslim.aspx
private ReaderWriterLockSlim fileLock = new ReaderWriterLockSlim();
fileLock.EnterWriteLock();
try
{
//write your file here
}
finally
{
fileLock.ExitWriteLock();
}

I had a similar situation, and resolved it by lock()ing on the StreamWriter object in question:
private Dictionary<string, StreamWriter> _writers; // Consider using a thread-safe dictionary
void WriteContent(string file, string content)
{
StreamWriter writer;
if (_writers.TryGetValue(file, out writer))
lock (writer)
writer.Write(content);
// Else handle missing writer
}
That's from memory, it may not compile. I'd read up on Andrew's solution (I will be), as it may be more exactly what you need... but this is super-simple, if you just want a quick-and-dirty.

I'll make it an answer with some explanation.
Windows already have something like you want, idea behind is simple: to allow multiple processes access same file and to carry on all writing/reading operations, so that: 1) all processes operates with the most recent data of that file 2) multiple writing or reading occurs without waiting (if possible).
It's called Memory-Mapped Files. I was using it for IPC mostly (without file), so can't provide an example, but there should be some.
You could mimic MMF behavior by using some buffer and sort of layer on top of it, which will redirect all reading/writing operations to that buffer and periodically flush updated content into physical file.
P.S: try to look also for file-sharing (open file for shared reading/writing).

Multi processes read&write one file

I have a txt file ABC.txt which will be read and wrote by multi processes. So when one process is reading from or writing to file ABC.txt, file ABC.txt must be locked so that any other processes can not reading from or writing to it. I know the enum System.IO.FileShare may be the right way to handle this problem. But I used another way which I'm not sure if it is right. The following is my solution.
I added another file Lock.txt to the folder. Before I can read from or write to file ABC.txt, I must have the capability to read from file Lock.txt. And after I have read from or written to file ABC.txt, I have to release that capability. The following is the code.
#region Enter the lock
FileStream lockFileStream = null;
bool lockEntered = false;
while (lockEntered == false)
{
try
{
lockFileStream = File.Open("Lock.txt", FileMode.Open, FileAccess.Read, FileShare.None);
lockEntered = true;
}
catch (Exception)
{
Thread.Sleep(500);
}
}
#endregion
#region Do the work
// Read from or write to File ABC.txt
// Read from or write to other files
#endregion
#region Release the lock
try
{
if (lockFileStream != null)
{
lockFileStream.Dispose();
}
}
catch
{
}
#endregion
On my computer, it seems that this solution works well, but I still can not make sure if it is appropriate..
Edit: Multi processes, not multi threads in the same process.

C#'s named EventWaitHandle is the way to go here. Create an instance of wait handle in every process which wants to use that file and give it a name which is shared by all such processes.
EventWaitHandle waitHandle = new EventWaitHandle(true, EventResetMode.AutoReset, "SHARED_BY_ALL_PROCESSES");
Then when accessing the file wait on waitHandle and when finished processing file, set it so the next process in the queue may access it.
waitHandle.WaitOne();
/* process file*/
waitHandle.Set();
When you name an event wait handle then that name is shared across all processes in the operating system. Therefore in order to avoid possibility of collisions, use a guid for name ("SHARED_BY_ALL_PROCESSES" above).

A mutex in C# may be shared across multiple processes. Here is an example for multiple processes writing to a single file:
using (var mutex = new Mutex(false, "Strand www.jakemdrew.com"))
{
mutex.WaitOne();
File.AppendAllText(outputFilePath,theFileText);
mutex.ReleaseMutex();
}
You need to make sure that the mutex is given a unique name that will be shared across the entire system.
Additional reading here:
http://www.albahari.com/threading/part2.aspx#_Mutex

Your solution is error prone. You've basically implemented double-checked locking (http://en.wikipedia.org/wiki/Double-checked_locking) which can be very unsafe.
A better solution would be to either introduce thread isolation, whereby only one thread ever accesses the file and does so by reading from a queue upon which requests to read or write are placed by other threads (and of course the queue is protected by mutually exclusive access by threads) or where the threads synchronize themselves either by synchronization devices (lock sections, mutices, whatever) or by using some other file access logic (for example, System.IO.FileShare came up in a few reponses here.)

If it was me, I would install something like SQL Server Compact Edition for reading/writing this data.
However, if you want to be able to lock access to a resource that is shared between multiple processes, you need to use a Mutex or a Semaphore.
The Mutex class is a .Net wrapper around an OS Level locking mechanism.
Overview of Synchronization Primitives

Best solution to solve File can not access error

I have a multi thread application. My problem is that sometimes I get an error like the program cannot access to a file as it is been used by something else (other part of the application).
Right now what I did is catch the exception and repeat the task after a 100ms delay.
It is working as the application does not open the files so often and it is open the files just for a short period.
Though I am not happy with that solution as it is ugly, and it still can go wrong.
Are there any good ways to sort out that problem? If there are not any easy answer and you need more information about the application(what files, why I might access in the same time) I can answer, I just try to keep the question simple, and generic.
Regards
Daniel
EDIT:
I made up something fast, what do you guys think about that solution, it is not the most generic one, but had no better idea just now:
When the program try to write into the file check if the it is used or not. If it is used collect a data in the memory, as soon as the application want to write in to the file again and the file is free insert in all of the data.
It still not the best, but it might give some more ideas.

Since your application is both reading and writing the file, all you need to do is keep track of which files your application is using. Keep a dictionary of R/W locks keyed by file name. When you go to access a file, take out the appropriate lock.
Here's how you might code it:
Dictionary<string, ReaderWriterLockSlim> lockDict = new Dictionary<string, ReaderWriterLockSlim>();
ReaderWriterLockSlim LockFile(string filename, bool readOnly)
{
var fullPath = System.IO.Path.GetFullPath(filename);
ReaderWriterLockSlim myLock;
// lock the dictionary while we're accessing it
lock (lockDict)
{
if (!lockDict.TryGetValue(fullPath, out myLock))
{
myLock = new ReaderWriterLockSlim();
lockDict[fullPath] = myLock;
}
}
// only block on the file lock once the dictionary is unlocked
if (readOnly)
myLock.EnterReadLock();
else
myLock.EnterWriteLock();
// file is now "locked", so caller can proceed with read/write, then unlock
return myLock;
}
And you might use it like this:
// block until we're not using the file anymore
var myLock = LockFile(filename, readOnly: false);
try
{
using (var file = new StreamWriter(filename))
{
...
}
}
finally
{
myLock.ExitWriteLock();
}

Correct way to read\write to a file: heavily used application

We have an heavily used .Net 3.5 application that reads "expensive to create" data and caches it. The application reads\writes files based on it not "being used by another process". If some other process is reading and writing to the file, then the application goes to sleep(for some time) and retries. Is this a correct way of reading and writing files? Please advise.
public void Add<T>(string key, CacheItem<T> item)
{
bool fileInUse = false;
while (!fileInUse)
{
try
{
using (Stream stream = new FileStream(Path.Combine(cachePath, key+".bin"), FileMode.Create, FileAccess.Write, FileShare.None))
{
Serializer.NonGeneric.Serialize(stream, item);
}
fileInUse = true;
}
catch (IOException ex)
{
if (ex.Message.Contains("being used by another process"))
{
//Poll till the file is free to be used by this process
Thread.Sleep(100);
fileInUse = false;
}
}
}
}
public CacheItem<T> Get<T>(string key, Type type)
{
CacheItem<T> item = null;
FileInfo fileInfo = new FileInfo(Path.Combine(cachePath, key+".bin"));
fileInfo.Refresh();
if (fileInfo.Exists)
{
bool fileInUse = false;
while (!fileInUse)
{
try
{
using (Stream stream = new FileStream(fileInfo.FullName, FileMode.Open, FileAccess.Read, FileShare.None))
{
object objectTemp = Serializer.NonGeneric.Deserialize(type, stream);
item = (CacheItem<T>)objectTemp;
}
fileInUse = true;
}
catch(IOException ex)
{
if (ex.Message.Contains("being used by another process"))
{
//Poll till the file is free to be used by this process
Thread.Sleep(100);
fileInUse = false;
}
}
}
}
return item;
}

You could add a global mutex on top of it, to avoid waiting more than absolutely necessary.
Global mutex is created by passing a non-empty name to Mutex Constructor.
Benefits:
Mutex will allow you to wake up immediately after the file becomes available, instead of waiting 50 ms on average.
Mutex allows you to fall asleep once and wake up once, instead of repeatedly sleeping/waking. Threads that are sleeping are handled very efficiently by the OS and consume almost no resources at all.
Once you acquire the mutex, the file open is almost 100% likely to succeed, instead of failing potentially many times before succeeding.
All in all, you are not just going to be faster, you'll likely consume less CPU cycles in the process.

I would suggest a different design altogether if this is something you're doing a lot of (and therefore performance is an issue).
You should have a public static method (or a method in a singleton) that takes a string (and a filename if this works with multiple files). In that method it should place that string into a BlockingCollection<string>. (You can have just one, or one per file through a Dictionary<string, BlockingCollection<string>> that maps filenames to the queue for that file.)
In the majority of cases (i.e. the buffer isn't full) the tasks that want to write something to a file just add it to a queue and then get right back to working.
You then need to have a separate thread/task that just sits there reading from the blocking collection(s) (you should only need one, even if you have lots of blocking collections) and writing the data out to a file. Since there is only one thread writing to the files there is no locking that needs to go around the file IO, and BlockingCollection is designed to work in this producer/consumer model and takes care of all the needed locking for you.
I would suggest using just a single BlockingColleciton for all files unless you find a compelling performance reason to try to deal with multiple queues. It'll be quite a bit easier to manage.