I have nearly completed a Quartz.NET based Windows Service (using ADO.NET, not RAM jobs). The service copies/moves files to various paths depending upon a schedule. I have some concerns however. It is very important that this service has some sort of detection method/system that will detect when the program has failed for whatever reason - whether it's files failing to be copied, or the whole scheduler crashing . Just wondering what you guys think is the best way to do this? I have a couple of vague ideas but I'm looking to hear some more input.
Here are the methods that we use:
We monitor the windows service itself using the IT monitoring system. We use one of those commercial products that monitors servers, services, databases, etc, but there are open source projects that can do this for you if you don't already have one in place.
We log fatal execeptions to a database table and have a separate service monitoring that table for exceptions.
We also use an ADO.Net store, so we also monitor the Quartz.net tables for things like stuck triggers.
With things like this you can definitely go down the over engineering path. Just keep in mind the cost benefit of adding each of these options and then decide how much work you want to put into monitoring, VS the cost of an outage.
Related
I have SQL Server database with information for files - I'm talking about custom properties. These are categories and description for each file.
The Windows Forms application is for the user. But I will also make a Windows Service that will track any changes with the files. If a change happens(renamed,moved,deleted) the service has to update that same database accordingly. And I think it should do it right away, without any delay.
Now this is going to be my first time making WS plus the first time I will have to handle concurrency (theoretically I know about threads and so on).
So:
First of all, is it OK if one process is updating a database another process may be using at the same time? Do you need to handle that situation on the first place? (Probably, fx in our daily "user lives" we can't modify a file when it's being used by another process)
Is the idea these two to share one data source good ?
If it is, then how to handle the concurrency ? I can use WCF for the messages between the two, but then does the solution have something to do with WCF ? Because I'm going to use this for the first time as well :D.
Any help is appreciated. Thanks in advance for the time !
Since MS SQL is transactional there will be no big deal. You just have to watch out for data wich might be read and updated by one process - there it can be neccessary to use a Transaction scope (that's a .NET Class ;)).
From the Software architectural Point of view you should conside using a three-tier and not a two-tier application:
Two Tier:
Essentially your System with the persistance-layer (DB) communicating with the Clients directly
Three Tier:
Persistance-Layer <--> Logic-Layer (e.g. a WCF-Service handling the app logic) <--> Clients (Service and Forms - triggering app logic and showing results)
When it comes to concurrency it's going to be really straight forward. The MSSQL database engine handles just about all of it (e.g. locking and sharing). Further, if you leverage the SqlCommandBuilder to build your statements, the statements will automatically use optimistic concurrency.
As for the Windows service and how it gets notified, use a FileSystemWatcher, it going to be more efficient and you won't be published some service port on the local box.
I'd normally give you some good code examples but I'm answering this from my phone.
Question:
Is there a way to force the Task Parallel Library to run multiple tasks simultaneously? Even if it means making the whole process run slower with all the added context switching on each core?
Background:
I'm fairly new to multithreading, so I could use some assistance. My initial research hasn't turned up much, but I also doubt I know what exactly to search for. Perhaps someone more experienced with multithreading can help me better understand TPL and/or find a better solution.
Our company is planning on deploying a piece of software to all users' machines that will connect to a central server a few times a day, and synchronize some files and MS Access data back to the user's machine. We would like to load-test this concept first and see how the Access DB holds up to lots of simultaneous connections.
I've been tasked with writing a .NET application that behaves like the client app (connecting & syncing with a network location), but does this on multiple threads simultaneously.
I've been getting familiar with the Task Parallel Library (TPL), as this seems like the best (newest) way to handle multithreading, and get return values back from each thread easily. However as I understand it, TPL decides how to run each "task" for the fastest execution possible, splitting the work among the available cores. So lets say I want to run 30 sync jobs on a 2-core machine... the TPL would run 15 on each core, sequentially. This would mean my load test would only be hitting the Access DB with at most 2 connections at the same time. I want to hit the database with lots of simultaneous connections.
You can force the TPL to do this by specifying TaskOptions.LongRunning. According to Reflector (not according to the docs, though) this always creates a new thread. I consider relying on this safe production use.
Normal tasks will not do, because they don't guarantee execution. Setting MinThreads is a horrible solution (for production) because you are changing a process global setting to solve a local problem. And still, you are not guaranteed success.
Of course, you can also start threads. Tasks are more convenient though because of error handling. Nothing wrong with using threads for this use case.
Based on your comment, I think you should reconsider using Access in the first place. It doesn't scale well and has problems once the database grows to a certain size. Especially if this is simply served off some file share on your network.
You can try and simulate load from your single machine but I don't think that would be very representative of what you are trying to accomplish.
Have you considered using SQL Server Express? It's basically a de-tuned version of the full-blown SQL Server which might suit your needs better.
I am requesting for an implementation suggestion in
OS: Windows server 2008
Platform : ASP.NET, C#
DB: MS SQL 2005
Scenario is:
We have to implement a monitoring daemon which will monitor set of DB tables in MS SQL in a frequent interval (Say 10 sec) and identify all critical entries.
The entries after analysis will associate with some actions based on its criticality or category. So assume one of a critical entry will be denoted as ACCT_USR_LIMIT_EXECEED : SomeName.
3.So ACCT_USR_LIMIT_EXECEED : SomeName shall associate to an action which shall be an email dispatch or a DB table update query execution or a folder size measurement or deleting a folder or cleaning up some files in the local HDD etc.
The amount critical entries to be analysed shall be moderate as of now but it has scope to increase too.
How do we approach this, and i see the possibilities are,
Shall we write one windows service for monitoring and dispatching actions or,
write two different services, one to monitor and dispatch its analysis to a MSMQ and the other service to dispatch an action reading the same MSMQ entry pushed.
Will having a single windows service will help us? or whats the best approach for this.
Kindly suggest
I think it all depends on the expected volume of dispatches. If you're going to do 2000 dispatches every second then I would think that a separation is a good idea, so one service won't impact the other and you could possibly have a separate environment (server) for each of them. If the expected amount is something like 10 every minute then I can't see why you should make it complicated, some threading and proper business layers will do just fine.
There is no straight forward answer to this as it depends on lot of parameters. I agree with what "Hyp" said.
As you said, some C# solution will help a lot - So, technically if you want to achieve this Windows Service / MSMQ stuff then you may have a look here -
http://stackoverflow.com/questions/1521841/receiving-msmq-messages-with-windows-service
http://stackoverflow.com/questions/3956467/how-to-create-a-c-sharp-listener-service-for-msmq-as-a-windows-service
Hope this helps.
I've got a for loop I want to parallelize with something like PLINQ's Parallel.ForEach().
The key here is that the C++ library i'm calling to do the computation is decidedly not thread safe, therefore, any plans to parallelize this need to do so across multiple processes.
I was thinking about using WCF to create a "distributor" process to which the "client" and multiple "calculators" could both connect and add/remove items to/from a queue and then the "calculator" sends the results directly back to the client which could update the gui as it receives them. This architecture would allow me to bring as many "calculators" online as I have processors and as I see it even bring them up across multiple computers creating a potential farm of processing power to which all the clients could share.
I'm just wondering if anyone has had any experience doing this and if there are existing application blocks or frameworks that I can use to build this for me. PLINQ does it within the process. is there like a DPLINQ (distributed) or something?
Also if that doesn't exist, does anybody want to give an opinion on my proposed architecture? Any obvious pitfalls? Does anyone think it will work!?!?!?
Sounds like you could be looking for Dryad. It's a Microsoft research project right now, but they do have an "academic release" available. My understanding is that they are also in the process of better productizing it (probably some kind of integration with Azure) for RTM sometime near the end of 2011. Mary Jo Foley covers more about this here.
A long time standard for controlling/dispatching distributed work is MPI. I've only ever used it from C++, but implementations from many languages exist. A quick google suggests that MPI.Net could be a good implementation for .Net!
Say I need to design an in-memory service because of a very high load read/write system. I want to dump the results of the objects every 2 minutes. How would I access the in-memory objects/data from within a web application?
(I was thinking a Windows service would be running in the background handling the in-memory service etc.)
I want the fastest possible solution, and I would guess most people would say use a web service? What other options would I have? I just don't understand how I could hook into the Windows service's objects etc.
(Please don't ask why I would want to do this, maybe you're right and it's a bad idea but I am also curious if this type of architecture is possible.)
Update
I was looking at this site swoopo.com that I would think has a lot of hits near the end of auctions, but since the auction keeps resetting the hits to the database would be just crazy so I was thinking if they did it in memory then dumped to db every x minutes...
What you're describing is called a cache, with a facade front-end.
You write a facade to which you commit your changes and acquire your datasets. The facade queues up reads and writes and commits when the queue is full or after a certain amount of time has passed. Your web application has a single point of access to the data (the facade), and the facade is structured in such a way to avoid writing and reading from storage too often.
Most relational database management systems do this for you. They do this kind of optimization and queuing internally so writing another layer on top of it would only slow things down. So don't write a cache if you're using an RDBMS.
Regarding the specifics of accessing such a facade, you can treat it as just an object, and implement it however you want (its own thread, a thread pool, a Web service, a Windows service, whatever).
Any remoting technology would work such as sockets, pipes and the like.
Check out: www.remobjects.com
You could use a Windows Message Queues or a Service Bus, or even .NET remoting.
See http://www.nservicebus.com/, or http://code.google.com/p/masstransit/.
You could hook into the Windows Services objects by using Remoting or WCF, both offer very fast interprocess communication. Sockets are fast too but are more cumbersome to program compared to WCF. There is a ton of WCF documentation and support online.
Databases provide a level of caching for you. The advantage of an in memory golden copy such as the one you propose is that it never has to read from disk when a request comes in and if you host it on the same machine as your IIS (provided you have enough RAM for both) there is no extra network hop, making it much faster that querying a db. However, the downside to this approach is that it does not scale as well if you need to add machines to load balance.
Third party messaging providers such as TIBCO are also worth looking at.