this kind of follows on from another question of mine.
Basically, once I have the code to access the file (will review the answers there in a minute) what would be the best way to test it?
I am thinking of creating a method which just spawns lots of BackgroundWorker's or something and tells them all load/save the file, and test with varying file/object sizes. Then, get a response back from the threads to see if it failed/succeeded/made the world implode etc.
Can you guys offer any suggestions on the best way to approach this? As I said before, this is all kinda new to me :)
Edit
Following ajmastrean's post:
I am using a console app to test with Debug.Asserts :)
Update
I originally rolled with using BackgroundWorker to deal with the threading (since I am used to that from Windows dev) I soon realised that when I was performing tests where multiple ops (threads) needed to complete before continuing, I realised it was going to be a bit of a hack to get it to do this.
I then followed up on ajmastrean's post and realised I should really be using the Thread class for working with concurrent operations. I will now refactor using this method (albeit a different approach).
In .NET, ThreadPool threads won't return without setting up ManualResetEvents or AutoResetEvents. I find these overkill for a quick test method (not to mention kind of complicated to create, set, and manage). Background worker is a also a bit complex with the callbacks and such.
Something I have found that works is
Create an array of threads.
Setup the ThreadStart method of each thread.
Start each thread.
Join on all threads (blocks the current thread until all other threads complete or abort)
public static void MultiThreadedTest()
{
Thread[] threads = new Thread[count];
for (int i = 0; i < threads.Length; i++)
{
threads[i] = new Thread(DoSomeWork());
}
foreach(Thread thread in threads)
{
thread.Start();
}
foreach(Thread thread in threads)
{
thread.Join();
}
}
#ajmastrean, since unit test result must be predictable we need to synchronize threads somehow. I can't see a simple way to do it without using events.
I found that ThreadPool.QueueUserWorkItem gives me an easy way to test such use cases
ThreadPool.QueueUserWorkItem(x => {
File.Open(fileName, FileMode.Open);
event1.Set(); // Start 2nd tread;
event2.WaitOne(); // Blocking the file;
});
ThreadPool.QueueUserWorkItem(x => {
try
{
event1.WaitOne(); // Waiting until 1st thread open file
File.Delete(fileName); // Simulating conflict
}
catch (IOException e)
{
Debug.Write("File access denied");
}
});
Your idea should work fine. Basically you just want to spawn a bunch of threads, and make sure the ones writing the file take long enough to do it to actually make the readers wait. If all of your threads return without error, and without blocking forever, then the test succeeds.
Related
I have created a semaphore instance on top of this class
public static SemaphoreSlim _zReportSemaphore = new SemaphoreSlim(1, 500);
And somewhere in my code i need to retrieve and send some data.
while (_isRunning)
{
try
{
xBsonDocument = null;
//I think its very clear in this line...
MongoDBDAO.xGetInstance().GetZReportData(ref xBsonDocument);
foreach (BsonDocument item in xBsonDocument)
{
try
{
ThreadObject xThreadObject = new ThreadObject();
xThreadObject.m_strTerminalId = item.GetValue("_id")["TERMINAL_ID"].ToString();
xThreadObject.m_strZNo = item.GetValue("_id")["Z_NO"].ToString();
m_xBuildAndSendZReportThread =
new Thread(new ParameterizedThreadStart(vBuildAndSendZReport));
m_xBuildAndSendZReportThread.Start(xThreadObject);
}
catch (Exception xException)
{
xException.TraceError();
continue;
}
Thread.Sleep(m_litleStepQTime);
}
}
catch (Exception xException)
{
Thread.Sleep(m_bigStepQTime);
Trace.vInsertError(xException);
continue;
}
Thread.Sleep(m_iSleepTime);
}
This thread targeting to send files to ftp
private void vBuildAndSendZReport(object prm_objParameters)
{
_zReportSemaphore.Wait();
RetriveDataFromMongoAndSend();
_zReportSemaphore.Release();
}
In this structure; if i don't use a semaphore it has working great but sometimes thread count overloading the CPU or Memory usage and machine has been crushing.
1- How can i provide control over data usage (ballancing, isolating threads etc.) with this slim semaphore?
2- Can I use SemaphoreSlim for this type of job in production? What can be the advantages and disadvantages of using such a workflow organization like this?
Does it improve performance? in my special case
3- Is there another alternative that will provide system resource management and will wrap up the technical exception management
Update:
I asked this question during a job I did a long time ago. After solving the problem, I realized that I did not return.
In the above example, the report sending job was happening in the file sharing environment. Other solutions are possible, such as using a CDN.
The question was: Why should I use a thread if it can't keep me informed about what it's doing, if it doesn't tell me if it has had successful results? Why should I use SemaphoreSlim for example!?
yes, of course it can be done with async programming. but I didn't want to include this library in related environment. It had to be. I'm sure this situation is needed in many codes.
my solution was this: I eliminated the possibility of the exception in the code that was throwing the exception. so i synced the conflict with the thread outside the app. I made something like a threadpool. It worked flawlessly as a consumer. I did this by setting up a custom timing mechanism.
Regardless, I still agree. A thread should be set up to carry information about the job it is doing. I'm not talking about writing a Mutex object in between. Thread itself can carry this information.
By the way, I gave points to those who answered. Because they made the right comments according the question.
This is the first hit on Google for "Semaphore and SemaphoreSlim usage Best Practices", so I would like to add 1 remark:
At least, this code
semaphore.Wait();
DoSomeThing();
semaphore.Release();
should be at the minimum
semaphore.Wait();
try
{
DoSomeThing();
}
finally
{
semaphore.Release();
}
Or else you might end up in NEVER releasing the semaphore again if an exceptions occurs in DoSomeThing...
And in async programming, consider using
await semaphore.WaitAsync();
Is there any event when semaphore ends its all threads
No. It's not even clear what that might mean. For example, what do you want to happen if, due to thread-scheduling issues, you have just one running thread in the semaphore at the moment, and that thread completes, releasing the semaphore, before one or more other threads even get to try to acquire the semaphore?
The semaphore has no way to detect this condition as different from every thread being done.
If you want to know when some collection of asynchronous operations has completed, you'll need to wait on that specifically. You have a number of options in .NET, including:
Call Thread.Join() on all of the thread objects you've started.
Use Task to run your asynchronous tasks instead of Thread, and use Task.WhenAll() (or less preferably, Task.WaitAll()) to wait for them to complete.
Use CountdownEvent. Call AddCount() for each task you start, have each task call Signal() when it's done, and then wait on the CountdownEvent.
By the way, the code you posted is suspect in other ways:
Why are you specifying a maximum count for the SemaphoreSlim, and why is this maximum not the same as your initial count? Do you actually expect to call Release() more often than you call Wait()?
Code that calls Thread.Sleep() is often incorrect. It's not clear why you are doing that, but it's likely there are better ways to solve whatever issue you're trying to address with those calls.
Without a good Minimal, Complete, and Verifiable example, I can't say for sure that those things are wrong. But there's a low likelihood of them being right. :)
I have a multithreaded application which is used to extract data from a website. I wanted to be able to pause and resume multiple threads from the UI. After searching on the web I came to know about two approaches that I can use to control (pause/resume) my threads.
Using Monitor class.
Using EventWaitHandle and ManualResetEvent class.
What I did:
I have a function named GetHtml that simply returns the html of the website. I am just showing the fraction part of this function for brevity.
public string GetHtml(string url, bool isProxy = false)
{
string result = "";
ExecutionGateway();
//->> EXTRA CODE FOR FETCHING HTML
return result;
}
I have a function ControlTasks used to control threads from UI, below I have explained the ControlTasks function using both thread control approaches using the Monitor class as well as the EventWaitHandle class (I will also briefly explain the working of the function ExecutionGateway).
1. Using the Monitor class
private object taskStopper = new object();
public bool ControlTasks(bool isPause)
{
try
{
if (isPause)
{
Monitor.Enter(taskStopper);
}
else
{
Monitor.Exit(taskStopper);
}
return true;
}
catch (Exception ex)
{
Logger.Instance.WriteLog("ControlTasks:", ex, Logger.LogTypes.Error);
return false;
}
}
ControlTasks is called from the UI where if isPause is true the exclusive lock is used on object taskStopper else releases the lock, Now here comes the function ExecutionGateway which is used to acquire lock on object taskStopper but it does nothing as the code below shows.
private void ExecutionGateway()
{
lock(taskStopper){ }
}
In this way all running threads enters waiting state when isPause is true in ControlTasks as taskStopper is exclusively locked and if isPause is false all threads resumes their processing.
2. Using the EventWaitHandle class
private EventWaitHandle handle = new ManualResetEvent(true);
public bool ControlTasks(bool isPause)
{
try
{
if (isPause)
{
handle.Reset();
}
else
{
handle.Set();
}
return true;
}
catch (Exception ex)
{
Logger.Instance.WriteLog("ControlTasks:", ex, Logger.LogTypes.Error);
return false;
}
}
This code also fundamentally does the same job, where the event state is signaled/non-signaled depending on the isPause parameter. Now, the corresponding ExecutionGateway method.
private void ExecutionGateway()
{
handle.WaitOne(Timeout.Infinite);
}
Problem:
What is the difference between these two approaches, is one better than the other? Are there any other ways to do this?
The main problem I have faced many times is if I use either of the above methods and I have 100 threads; when I pause them, then resume them after 5 or more minutes, the UI starts hanging. The UI is terrifically unresponsive. It gets updated but keeps on hanging and I keep getting the message "Not Responding" at each interval. One thing I want to mention each thread extracts data and notifies the UI about the data fetched through event handling. What could be the reason of this unresponsiveness? Is it a problem with my approach(es)?
I think it's always desirable to use a construct that communicates your intent clearly. You want a signal to other threads that they should wait (i.e. stop doing what they're doing) until you signal to them that they can start again. You have one controlling thread (your UI) and potentially many threads doing work and marshalling results back to the UI.
Approach 1 isn't ideal because locks (at least in my experience) are most often used to protect a resource that isn't suitable for use in multi threaded code. For example, writing to a shared field.
Approach 2 makes much more sense, a manual reset event functions like a gate: open the gate and things can pass through, close it and they can't. That's exactly the behaviour you're looking for and I think most developers would understand quite quickly that that's your intent.
As for your second problem, it sounds like you're getting waves of messages clogging the UI. If you stop all 100 of your threads then start them at the same time, there's a good chance they're going to finish their work quite close together and all be trying to send the result of their work to the UI thread. To solve that you could try staggering the work when you restart or use fewer threads. Another option would be to aggregate results and only dispatch the the UI every x seconds - but that's a bit more work.
In Option 1, using the Monitor class means that only one thread owns the exclusive lock of the monitor object at a time. This means that of your 100 threads, only 1 is processing at a time, which kind of defeats the purpose of using threads. It also means that your GUI thread has to wait until the current worker thread has finished before it can obtain the lock.
The ManualResetEvent is a much better choice as it is used to signal between threads, rather than protect against multiple thread access.
I do not know why your GUI is so unresponsive using the second option, but I do not think it is related to your manual reset event. More likely you have a different problem where the GUI thread is getting swamped. You suggest you have 100 threads all firing notification events to the GUI which would seem a likely culprit.
What happens if you debug your app, and just randomly break when your GUI is unresponsive? Doing this many times should show what your GUI thread is up to and where the bottleneck is.
I have a function where I want to execute in a separate thread avoiding two threads to access the same resources. Also I want to make sure that if the thread is currently executing then stop that thread and start executing the new thread. This is what I have:
volatile int threadCount = 0; // use it to know the number of threads being executed
private void DoWork(string text, Action OncallbackDone)
{
threadCount++;
var t = new Thread(new ThreadStart(() =>
{
lock (_lock) // make sure that this code is only accessed by one thread
{
if (threadCount > 1) // if a new thread got in here return and let the last one execute
{
threadCount--;
return;
}
// do some work in here
Thread.Sleep(1000);
OncallbackDone();
threadCount--;
}
}));
t.Start();
}
if I fire that method 5 times then all the threads will be waiting for the lock until the lock is released. I want to make sure that I execute the last thread though. when the threads are waiting to be the owner of the lock how can I determine which will be the next one owning the lock. I want them to own the resource in the order that I created the threads...
EDIT
I am not creating this application with .net 4.0 . Sorry for not mentioning what I was trying to accomplish. I am creating an autocomplete control where I am filtering a lot of data. I don't want the main window to freeze eveytime I want to filter results. also I want to filter results as the user types. If the user types 5 letters at once I want to stop all threads and I will just be interested in the last one. because the lock blocks all the threads sometimes the last thread that I created may own the lock first.
I think you are overcomplicating this. If you are able to use 4.0, then just use the Task Parallel Library. With it, you can just set up a ContinueWith function so that threads that must happen in a certain order are done in the order you dictate. If this is NOT what you are looking for, then I actually would suggest that you not use threading, as this sounds like a synchronous action that you are trying to force into parallelism.
If you are just looking to cancel tasks: then here is a SO question on how to cancel TPL tasks. Why waste the resources if you are just going to dump them all except for the last one.
If you are not using 4.0, then you can accomplish the same thing with a Background Worker. It just takes more boilerplate code to accomplish the same thing :)
I agree with Justin in that you should use the .NET 4 Task Parallel Library. But if you want complete control you should not use the default Task Scheduler, which favors LIFO, but create your own Task Scheduler (http://msdn.microsoft.com/en-us/library/system.threading.tasks.taskscheduler.aspx) and implement the logic that you want to determine which task gets preference.
Using Threads directly is not recommended unless you have deep knowledge of .NET Threading. If you are on .NET 4.0; Tasks and TPL are preferred.
This is what I came up with after reading the links that you guys posted. I guess I needed a Queue therefore I implemented:
volatile int threadCount = 0;
private void GetPredicateAsync(string text, Action<object> DoneCallback)
{
threadCount++;
ThreadPool.QueueUserWorkItem((x) =>
{
lock (_lock)
{
if (threadCount > 1) // disable executing threads at same time
{
threadCount--;
return; // if a new thread is created exit.
// let the newer task do work!
}
// do work in here
Application.Current.Dispatcher.BeginInvoke(new Action(() =>
{
threadCount--;
DoneCallback(Foo);
}));
}
},text);
}
I am using the BackgroundWorker to do some heavy stuff in the background so that the UI does not become unresponsive.
But today I noticed that when I run my program, only one of the two CPUs is being used.
Is there any way to use all CPUs with the BackgroundWorker?
Here is my simplified code, just if you are curious!
private System.ComponentModel.BackgroundWorker bwPatchApplier;
this.bwPatchApplier.WorkerReportsProgress = true;
this.bwPatchApplier.DoWork += new System.ComponentModel.DoWorkEventHandler(this.bwPatchApplier_DoWork);
this.bwPatchApplier.ProgressChanged += new System.ComponentModel.ProgressChangedEventHandler(this.bwPatchApplier_ProgressChanged);
this.bwPatchApplier.RunWorkerCompleted += new System.ComponentModel.RunWorkerCompletedEventHandler(this.bwPatchApplier_RunWorkerCompleted);
private void bwPatchApplier_DoWork(object sender, DoWorkEventArgs e)
{
string pc1WorkflowName;
string pc2WorkflowName;
if (!GetWorkflowSettings(out pc1WorkflowName, out pc2WorkflowName)) return;
int progressPercentage = 0;
var weWorkspaces = (List<WEWorkspace>) e.Argument;
foreach (WEWorkspace weWorkspace in weWorkspaces)
{
using (var spSite = new SPSite(weWorkspace.SiteId))
{
foreach (SPWeb web in spSite.AllWebs)
{
using (SPWeb spWeb = spSite.OpenWeb(web.ID))
{
PrintHeader(spWeb.ID, spWeb.Title, spWeb.Url, bwPatchApplier);
try
{
for (int index = 0; index < spWeb.Lists.Count; index++)
{
SPList spList = spWeb.Lists[index];
if (spList.Hidden) continue;
string listName = spList.Title;
if (listName.Equals("PC1") || listName.Equals("PC2"))
{
#region STEP 1
// STEP 1: Remove Workflow
#endregion
#region STEP 2
// STEP 2: Add Events: Adding & Updating
#endregion
}
if ((uint) spList.BaseTemplate == 10135 || (uint) spList.BaseTemplate == 10134)
{
#region STEP 3
// STEP 3: Configure Custom AssignedToEmail Property
#endregion
#region STEP 4
if (enableAssignToEmail)
{
// STEP 4: Install AssignedTo events to Work lists
}
#endregion
}
#region STEP 5
// STEP 5 Install Notification Events
#endregion
#region STEP 6
// STEP 6 Install Report List Events
#endregion
progressPercentage += TotalSteps;
UpdatePercentage(progressPercentage, bwPatchApplier);
}
}
catch (Exception exception)
{
progressPercentage += TotalSteps;
UpdatePercentage(progressPercentage, bwPatchApplier);
}
}
}
}
}
PrintMessage(string.Empty, bwPatchApplier);
PrintMessage("*** Process Completed", bwPatchApplier);
UpdateStatus("Process Completed", bwPatchApplier);
}
Thanks a lot for looking into this :)
The BackgroundWorker does its work within a single background (ThreadPool) thread. As such, if it's computationally heavy, it'll use one CPU heavily. The UI thread is still running on the second, but is probably (like most user interface work) spending almost all of its time idle waiting for input (which is a good thing).
If you want to split your work up to use more than one CPU, you'll need to use some other techniques. This could be multiple BackgroundWorker components, each doing some work, or using the ThreadPool directly. Parallel programming has been simplified in .NET 4 via the TPL, which is likely a very good option. For details, you can see my series on the TPL or MSDN's page on the Task Parallel Library.
Each BackgroundWorker uses only a single thread to do the stuff you tell it to do. To take advantage of multiple cores, you would need multiple threads. That would mean either multiple BackgroundWorkers or spawning multiple threads from within your DoWork method.
The backgroundworker, by itself, only provides one additional thread of execution. It's purpose is to get things off the UI thread, and it's very good at that job. If you want more threads, you need to provide them yourself.
It would be tempting here to build a method that accepts an SPWeb argument, and just call Thread.Start() over and over for each object; then finish with Thread.Join() or WaitAll() to wait for them to finish at the end of the BackgroundWorker. However, this would be a bad idea because you'll lose efficiency as the operating system spends time performing context switches among all the threads.
Instead, you want to force your system to run in only a few threads, but at least two (in this case). A good rule of thumb is (2n - 1), where "n" is the number of processor cores you have... but there are all kinds of cases where you want to break this rule. You can implement this by using a ThreadPool, by iterating over your SPWeb objects and adding them to a queue that you keep pulling from, or other means such as the TPL.
The BackgroundWorker is running a new thread on the second CPU core, leaving the UI responsive.
If you're using .NET 4, look into using the Task Parallel Library, which could give you better results and utilize both cores.
The BackgroundWorker itself is only creating a single thread apart from your main UI to do work in - it's not trying to parallelize the operations within that work thread. If you want to spread your work across multiple work threads you should look into using the TPL. Bear in mind that not all tasks translate well to parallel execution, so if freeing the UI is your only goal this may already be the best you can do.
There are potential pitfalls to this, but you might get some mileage out of utilizing Parallel.ForEach:
Instead of
foreach (SPWeb web in spSite.AllWebs)
{
//Your loop code here
}
You could:
Parallel.Foreach(spSite.AllWebs, web =>
{
//Your loop code here
});
This basically creates a Task (from the Task API in .NET 4.0) from each item and schedules that work with the TaskPool, which will give you some of the parallelism you will need to take advantage of those cores.
You will have to fix the inevitable concurrency problems that might arise from this, but it's a good starting point. You are going to at least fix the fact that you are maintaining a shared state across threads (the progress counter). Here's some guidance on that: http://msdn.microsoft.com/en-us/library/dd997392.aspx
I have to be able to save a file, unfortunatly it can potentially be very large so saving it can potentially take minutes. As I need to do this from a GUI thread I don't want to block the GUI from executing. I was thinking about attempting the save operation on a seperate thread to allow the primary GUI thread to continue executing.
Is there a nice (easy) way to spawn a new thread, save the file, and destroy the thread without any nasty side effects?!
It must be said that I have NEVER had to use threads before so I am a complete novice! Any and all help would be greatly appreciated!
BackgroundWorker (as suggested by Frederik) is a good choice, particularly if you want to report progress to the UI while you're saving. A search for BackgroundWorker tutorial gets a lot of hits, so you should be able to follow one of those to get you started.
One thing to be careful of: would there be any way of changing the data structure that you'll be trying to save from the UI thread? If so, you should disable those aspects of the UI while you're saving - it would (probably!) be bad to be half way through saving the data, then allow the user to change some of it. If you can get away with effectively handing off the data to the background thread and then not touching it from the UI thread, that will make your life a lot easier.
You could maybe use the BackGroundWorker component, as it will abstract a bit the Threading part for you.
Your problem might be that there are several nice and easy ways of doing it. If you just want to set off the file save and not worry about knowing when it has completed, then having a method
void SaveMyFile(object state)
{
// SaveTheFile
}
and calling it with
ThreadPool.QueueUserWorkItem( SaveMyFile );
will do what you want.
I would recommend doing Asynchronous I/O. It's a little bit easier to set up and doesn't require you to create new threads yourself.
Asynchronous programming is where you have, for example, a file stream you want to write to but does not want to wait for it to finish. You might want to be notified when it's finished but you don't want to wait.
What you do is using the BeginWrite/BeginRead and EndWrite/EndRead functions that are available on the Stream class.
In your method you start by calling BeginWrite with all the data you want to write and also pass in a callback function. This function will be called when BeginWrite has finished.
Inside the callback function you call EndWrite and clean up the stream and check for errors.
BeginWrite will not block which means that if it's called from within an event handler that thread can finish that handler and continue processing more event (such as other GUI events).
using System;
using System.IO;
using System.Text;
class Program
{
private static FileStream stream;
static void Main(string[] args)
{
stream = new FileStream("foo.txt",
FileMode.Create,
FileAccess.Write);
const string mystring = "Foobarlalala";
ASCIIEncoding encoding = new ASCIIEncoding();
byte[] data = encoding.GetBytes(mystring);
Console.WriteLine("Started writing");
stream.BeginWrite(data, 0, data.Length, callback, null);
Console.WriteLine("Writing dispatched, sleeping 5 secs");
System.Threading.Thread.Sleep(5000);
}
public static void callback(IAsyncResult ia)
{
stream.EndWrite(ia);
Console.WriteLine("Finished writing");
}
}
}
The sleeping is pretty important because the thread that's writing stuff will be killed if the main thread is killed off. This is not an issue in a GUI application, only here in this small example.
MSDN has a pretty good overview on how to write this stuff, and also some good articles on Asynch programming in general in case you go for the backgroundworker or ThreadPool.
or u could use old friends delegates.