I have a silverlight application which kicks off a number of jobs on the server side. Each job is a unit of work which are independent of each other.
I used Parallel.ForEach() and it works fine however I realized that if I had a large number of jobs (say over 300), when the thread count increases by 50 the silverlight application seems to stop responding (it does not freeze the browser but a grid which should have data populated in it is empty and the little doughnut keeps spinning).
Only when the thread counts drop again (i.e. all the jobs have finished processing) does the grid get populated.
I am testing with the Asp.net Development servers (cassini based) and was wondering has something to do with it.
I also switched my code to use the async programming model but I got the same problem as the threads increased.
Any ideas what may be causing this?
JD
I was thinking about doing ThreadPool.SetMaxThread() but I read somewhere that this may not work for web hosted apps.
If you go gobbling up all the threads with a Parallel for each then there aren't any threads left available to service the WCF requests that your Grid is likely depending on to fetch the data it needs to display.
You would probably want to use the ParallelOptions parameter in the ForEach method to specify the maximum number of parallel operations dispatched at one time.
Related
I am developing an application that uses Windows Workflow. One area of the workflow uses a Parallel.ForEach activity that runs a AsyncCodeActivity. You can see this in the screenshot below. The RunPolicyWorkflow Activity is an AsyncCodeActivity.
From what I understand from the Windows Workflow documentation is that this will create new threads as needed to operate on the collection that is being enumerated in the ParalleForEach Activity.
I have around 16000 items in the Parallel Loop. Inside the loop (during the RunPolicyWorkflow Activity) I am doing a little CPU work, but most of the time is spent saving the results to a SQL Server Instance. When using Resource Monitor to keep an eye on my process, I noticed that there were around 2,000 threads in this process that was running the workflow.
It looks like my application is getting slower the more threads it creates. My computer only has 8 logical processors so I'm pretty sure this isn't great optimization.
Does anyone know of a way to limit the amount of threads that Windows Workflow is creating? Or does anyone have any suggestions on a way that this could be done better? All the items in the Parallel loop are independent from each other and I want to process all of the items in the collection (16000) as fast as possible. Initially Its processing at about 300 items per minute but lows down to about 60 items per minute as the thread count gets higher and more items have been processed.
This is an example activity that throttles the parallel activities.
https://msdn.microsoft.com/en-us/library/vstudio/ee620808(v=vs.100).aspx
This program is used for taking data that has been imported and exported it line by line to normalized tables. It can take hours sometimes.
How it works is
Uses presses button to begin processing
A call is made from jquery to a method on the MVC controller.
That controller calls a DLL which then begins the long processing
Only one person uses this program at a time.
We are worried about it tying up a ASP.NET IIS thread. Would this be more efficient if instead of the web site running the code we could make a scheduled task that runs a EXE every 30 minutes to check and process the code..
EDIT 1: After talking to a coworker the work around we will do for now is simply remove the button from the web site that processes and instead refactor that processing into a scheduled task that runs every 5 minutes ... if there are any comments about this let me know.
The question is really about the differences between the web site running code vs. a completely separate EXE...IIS threads vs. processes... Does it help any ?
If the processing takes hours, it should definitely be in a separate process, not just a separate thread. You complicate thread locking and management, garbage collection and other things by dropping this into a separate process. For example, if your web server needs to be rebooted, your separate process can continue running without being affected. With a little work, you could even spin up this process on a separate server if you want (of course you would need to change the process start mechanism to do this)
When the task can run for hours having it block an ASP.Net thread is definitely the wrong thing to do. A web call should complete in a reasonable amount of time (seconds ideally, minutes at worst). Tasks which take considerably longer than that can be initiated from a web request but definitely shouldn't block the entire execution.
I think there are a couple of possible paths forward
If this is a task that does need to be executed on a semi-regular basis then factor it into an EXE and schedule the task to run at the correct interval
If this task should run on demand then factor it out into an EXE and have the web request kick off the EXE but not wait for its completion
Another possibility is to factor it out into a long running server process. Then use remoting or WCF to communicate between asp.net and the process
Does an HttpHandler listen for a disconnect from the browser?
My guess is "no" since it seems to be mostly/only used for dynamic file creation, so why would it?
But I can't find an answer in the docs or goog.
Many thanks in advance!
Background
I'd like to "abort" an HttpHandler because currently, I allow huge excel exports (~150k sql rows, so ~600k html lines). For reasons almost as ridiculous as the code, I have a query that fires for as many sql rows that the user tries to export. As you can imagine, this takes a very long time.
I think I'm getting backed up with worker processes because users probably get frustrated with the lag, and try again with a smaller result. I currently flush the worker procs automatically every 30 min, but I'd rather cleanup more quickly.
I don't have the time to clean up the sql right now, so I'd like to just listen for an "abort" from the client and kill the handler if "aborted".
What you're hoping to accomplish by listening for a client connection drop won't really help solve your problem at all. The core of your problem is a long running task being kicked off in an HttpHandler directly.
In this case, even if you could listen for a client disconnect it wouldn't ever be acted upon as your code will be too busy executing to listen for it.
The only way to properly determine progress and perform actions during long running processes such as this is to ensure that your code is multi-threaded. The problem with doing this in ASP.NET for long running processes is they'll suck up threads from the thread pool needed to serve your pages. This could result in your website hanging or responding very slowly, as you've been experiencing.
I would recommend writing a Windows Service to handle these long running jobs and having it spit the results into a staging directory. I would then use MSMQ or similar to throw the request to the service for processing.
Ultimately, you want to get this long running thread outside of ASP.NET where you can take advantage of the benefits that multi-threading can offer you. Such as, the ability to report back the progress and to abort when needed.
I have a question around threading and background workers that I hope you can help with.
I plan on making an ftp application to upload a file to 50 servers. Rather than the user having to wait for each upload to finish before the next one starts I was looking at threading/background workers. Once an upload finishes I want to report the status of the upload "completed/failed" back to the UI. From my understanding, I will need to use background workers for this so I know when the task has completed. I know with threading I can use producer/consumer queue or a semaphore to run a given amount of threads at once but I am not quite sure how I can achieve this with back ground workers.
So my question is, what would be a sensible number of background workers controlling uploading to run at once and what would be the best way to queue the rest?
There is no limit on the size of the upload file so this could be quite small or up to a few MB.
Thanks in advance.
Edit - I tested out one backgroundworker for each server running simultaneousness. The results where faster than just a single backgroundworker but I can't say that i was fully comfortable with running 50 plus background workers at once and since the server count may increase in the future, I decided to stick with just the one, which seems to be fast enough. I may in future look at increasing the count of workers to 2 or 3 but currently 1 seems to be adequate. Thanks for everyones help.
Thanks
I'd go in a completely different direction with it, tbh. Your app should take the file and store it once, responding to the client that it's got it. The file should then be propagated to the other servers. You can do this many ways, but if you want it controlled by the same application (i.e. not done using a windows service or the like) then a good way would be to use a message queue (either MSMQ or one of the OS ones).
This is much easier than using a semaphore or producer-consumer queue.
Put all your tasks in a queue (doesn't need to be a thread-safe queue, it will only be used from the UI thread).
Loop from 1 to N, taking out a task and starting a BackgroundWorker. (Be sure to handle the empty queue, when there were less than N tasks to begin with). In the RunWorkerCompleted event, update your UI, dequeue another task, and start another BackgroundWorker.
The bottleneck here is going to be your network bandwidth. If your local upstream connection is so fast that you can saturate the incoming connections on two or more remote hosts, then you'll benefit from running multiple uploads in parallel. If not, then it makes very little difference to the total upload time, since it'll be dictated by (file size * number of uploads) / (local bandwidth). In other words - if you do 20 uploads one at a time, it'll take an hour; if you do 20 uploads in parallel, it'll still take an hour. The advantage of the first approach is that if you lose connectivity you'll only need to resume/restart a single upload - whichever one was in progress when the connection was lost.
I'd therefore use a single background thread to sequentially upload the file to each server in turn. If you're using the .NET BackgroundWorker to do this, you can get it to ReportProgress at the end of each file (and you know in advance how many files are to be uploaded so you can calculate progress as a percentage), and attach some custom state to the progress update to inform the user whether the last upload succeeded or not.
The only way to know for sure is to test and measure, but it can be different from machine to machine, mostly depending on uplink speed.
Starting 50 backgroundworkers at the same time is a bit on the high end, but is not incredibly many. A simple approach would be to start 50 all at the same time and measure memory consumption and upload speed.
If the FTP servers are each much faster than the client uplink speed the most efficient would be to just upload one (or possibly two) at a time.
Let's say we are building some public service that grabs the setup of a user (what server, user and pwd he wants to perform the call), logs in into that server and do some processing...
the process takes about 15 seconds to complete
each user has a different setup (server/user/pwd), so the process needs to run against each one
if 1000 users tells the system to run the method at 1:00PM
How can I insure that the method is processed in the next 15 minutes?
What should be the correct approach to this little problem?
I'm thinking that I need to do something Asynchronously, and parallel processing could speed up things, maybe throttling the processes, maybe execute 100 calls per each 30 seconds?
I never did something like this and would love to get your feedback on ideas and future problems just to spend 100 hours of work and realize that I took a wrong road :(
Thank you.
added
The only thing to have in consideration is that this should be a 100% web solution.
If one call to your method does not affect the result of another method call (which seems to be the case here), parallel programming seems to be the way to go.
Consider not processing this in the asp.net application directly, but rather placing such requests on a queue and having another process (windows service may be a good candidate here) pulling items off the queue for processing. The windows service can have multiple threads and can pull as many items off the queue at once as there are processing threads available. With an appropriate queuing mechanism, the windows service can run on separate hardware if needed to reach your performance goals.
You can have the original web page query the result using e.g. Ajax to provide the user feedback if that's a requirement.
UPDATE:
Microsoft has recommended a pattern for long running tasks that can be used in a hosted environment.
Well, 1000 * 15 seconds is more than 4 hours, so you can only complete the entire task within the 15 minute time frame if you parallelize the batch.
I would set up a queue and have a sufficient number of threads or processes pull from that queue.
You can define an in-process queue with Queue<T> or out-of-process either with a database table or MSMQ.
If you don't want to write multithreaded code, you can just have a bunch of different processes running on different machines, all pulling from the same queue.
A console application can do this, but a Windows Service is definitely also an alternative.