I design an application back-end. For now, it is a .NET process (a Console Application) which hosts various communication frameworks such as Agatha and NServiceBus.
I need to periodically update my datastore with values (coming from the application while it's running).
I found three possible ways:
Accept command line arguments, so I can call my console app with -update.
On start up a background thread will periodically invoke the update method.
Create an updater.exe app which will do the updates, but I will have code duplication since in some way it will need to query the data from the source in order to save it to the datastore.
Which one is better?
Use the simplest thing that will work. Sounds like option 1 is the way to go based on the info you have given.
Option 2 has threads, threads always complicate programs, more difficult to debug and write, greater chance of bugs.
Option 3, would mean that you have two apps, if you make a change you will have to deploy new versions of both, increasing maintenance costs.
Related
I am working on project where we have decided to split our background tasks (network, CPU and IO intensive) into three windows services.
Now the question is, whether we should host all three services into a single process or create three independent services with their own processes.
Windows Service project template allows multiple services to be created, when installed they'll create separate entries in Service Control Manager (SCM) and can be controlled independently. The benefit here is better code management and code reuse.
However, if there is any performance drawback, which is the primary reason why we're having multiple services in the first place, I'd rather let go this benefit.
Please advise.
My suggestion is to go for Seperarte windows services created using topshelf or other technology hence they are independent of paltform
Scalability easly scalable as per need ,if one service is being used more then other, then that one service can be scale up by running multiple instance of same.
parallel processing as services are independent they can work in prallel hence performance improved.
I've been working on an internal developer tool on and off for a few weeks now, but I'm running into an ugly stumbling block I haven't managed to find a good solution for. I'm hoping someone can offer some ideas or guidance on the best ways to use the existing frameworks in .NET.
Background: the purpose of this tool is to load multiple different types of log files (Windows Event Log, IIS, SQL trace, etc.) to the same database table so they can be sorted and examined together. My personal goal is to make the entire thing streamlined so that we only make a single pass and do not cache the entire log either in memory or to disk. This is important when log files reach hundreds of MB or into the GB range. Fast performance is good, but slow and unobtrusive (allowing you to work on something else in the meantime) is better than running faster but monopolizing the system in the process, so I've focused on minimizing RAM and disk usage.
I've iterated through a few different designs so far trying to boil it down to something simple. I want the core of the log parser--the part that has to interact with any outside library or file to actually read the data--to be as simple as possible and conform to a standard interface, so that adding support for a new format is as easy as possible. Currently, the parse method returns an IEnumerable<Item> where Item is a custom struct, and I use yield return to minimize the amount of buffering.
However, we quickly run into some ugly constraints: the libraries provided (generally by Microsoft) to process these file formats. The biggest and ugliest problem: one of these libraries only works in 64-bit. Another one (Microsoft.SqlServer.Management.Trace TraceFile for SSMS logs) only works in 32-bit. As we all know, you can't mix and match 32- and 64-bit code. Since the entire point of this exercise is to have one utility that can handle any format, we need to have a separate child process (which in this case is handling the 32-bit-only portion).
The end result is that I need the 64-bit main process to start up a 32-bit child, provide it with the information needed to parse the log file, and stream the data back in some way that doesn't require buffering the entire contents to memory or disk. At first I tried using stdout, but that fell apart with any significant amount of data. I've tried using WCF, but it's really not designed to handle the "service" being a child of the "client", and it's difficult to get them synchronized backwards from how they want to work, plus I don't know if I can actually make them stream data correctly. I don't want to use a mechanism that opens up unsecured network ports or that could accidentally crosstalk if someone runs more than one instance (I want that scenario to work normally--each 64-bit main process would spawn and run its own child). Ideally, I want the core of the parser running in the 32-bit child to look the same as the core of a parser running in the 64-bit parent, but I don't know if it's even possible to continue using yield return, even with some wrapper in place to help manage the IPC. Is there any existing framework in .NET that makes this relatively easy?
WCF does have a P2P mode however if all your processes are local machine you are better off with IPC such as named pipes due to the latter running in Kernel Mode and does not have the messaging overhead of the former.
Failing that you could try COM which should not have a problem talking between 32 and 64 bit processes. - Tell me more
In case anyone stumbles across this, I'll post the solution that we eventually settled on. The key was to redefine the inter-process WCF service interface to be different from the intra-process IEnumerable interface. Instead of attempting to yield return across process boundaries, we stuck a proxy layer in between that uses an enumerator, so we can call a "give me an item" method over and over again. It's likely this has more performance overhead than a true streaming solution, since there's a method call for every item, but it does seem to get the job done, and it doesn't leak or consume memory.
We did follow Micky's suggestion of using named pipes, but still within WCF. We're also using named semaphores to coordinate the two processes, so we don't attempt to make service calls until the "child service" has finished starting up.
I have nearly completed a Quartz.NET based Windows Service (using ADO.NET, not RAM jobs). The service copies/moves files to various paths depending upon a schedule. I have some concerns however. It is very important that this service has some sort of detection method/system that will detect when the program has failed for whatever reason - whether it's files failing to be copied, or the whole scheduler crashing . Just wondering what you guys think is the best way to do this? I have a couple of vague ideas but I'm looking to hear some more input.
Here are the methods that we use:
We monitor the windows service itself using the IT monitoring system. We use one of those commercial products that monitors servers, services, databases, etc, but there are open source projects that can do this for you if you don't already have one in place.
We log fatal execeptions to a database table and have a separate service monitoring that table for exceptions.
We also use an ADO.Net store, so we also monitor the Quartz.net tables for things like stuck triggers.
With things like this you can definitely go down the over engineering path. Just keep in mind the cost benefit of adding each of these options and then decide how much work you want to put into monitoring, VS the cost of an outage.
I'm making a small application that is supposed to download info from the web every day at 2am. It will download the information and write the strings to an XML file of my choosing.
Using .NET and C#.
My initial approach was to install a service on the users computer and have that run, but I'm not so sure. I've not even used it so much in the past, only once.
Which is the best (read: time tested :P ) approach to this very common problem.
You can either build your application as a Windows Service, as you mentioned.
Or else it would probably be a better idea to create a normal console application, and launch it automatically at 2.00am with the Windows Task Scheduler.
You can consider both methods as popular and "time-tested".
I would suggest having a console app, which calls data fetching algo in a separate public class (not the main method).
Like Daniel mentioned, run it via Windows Task Scheduler which itself will take care of most scheduling requirements.
This allows the solution to be scaled in the future if need be. E.g. convert into Windows Service, full GUI Winform or even SQL server scheduled tasks etc.
When launching a thread or a process in .NET or Java, is there a way to choose which processor or core it is launched on? How does the shared memory model work in such cases?
If you're using multiple threads, the operating system will automatically take care of using multiple cores.
is there a way to choose which processor or core it is launched on?
You can use the task manager to tell windows what CPU(s) your program should be allowed to run on. Normally this is only useful for troubleshooting legacy programs which have broken implementations of multi-threading. To do this,
Run task manager
Find your process in the Processes window.
Right click and choose Set Affinity...
Tick the checkboxes next to the CPU's you want to allow your application to run on. Windows will then only schedule threads from that process onto those particular CPU's
If I recall correctly, windows will 'remember' these settings for subsequent times your process is run, but please don't quote me on that - run some tests yourself :-)
You can also do this programatically in .NET after your program has launched using using the System.Diagnostics.Process.ProcessorAffinity property, but I don't think it will 'remember' the settings, so there will always be a short period in which your app is run on whichever CPU windows sees fit. I don't know how to do this in java sorry.
Note:
This applies at the entire process level. If you set affinity for CPU0 only, and then launch 50 threads, all 50 of those threads will run on CPU0, and CPU1, 2, 3, etc will sit around doing nothing.
Just to reiterate the point, this is primarily useful for troubleshooting broken legacy software. If your software is not broken, you really shouldn't mess with any of these settings, and let windows decide the best CPU(s) to run your program on, so it can take the rest of the system's performance into account.
As for the 'shared memory' model, it works the same, but there are more things that can go subtly wrong when your app is running on multiple CPU's as opposed to just timeslices on a single one.
For an eye-opening example, read this ridiculousfish article about CPU's and Memory Barriers.
It's aimed at OSX development on PowerPC, but general enough that it should apply everywhere. IMHO it's one of the top ten 'all developers should read this' articles I've read.
The operating system takes care of multi-threading when the virtual machine is using native threads (as opposed to green-threads), and you can't specify low level details, like choosing a processor for a certain thread. It is better that way because you usually have many more threads than you have processors available, so the operating system needs to do time-slicing to give all threads a chance to run.
That being said, you can set threads priorities if you have a critical task, and a threading API usually provides this possibility. See the Java API for example: http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Thread.html#setPriority(int)
PS: there's something broken in the parsing engine ... I had to add the above link as plain text
I would have a look at the Parallel extensions to the .NET framework. It is still in CTP, however it supposed to make the best use of multi core processors. The easiest place to get started for .NET is on the parallel teams blog.
As for Java I have no idea.
I have used this in a couple of programs because my core 0 was kind of messed up.
// Programmatically set process affinity
var process = System.Diagnostics.Process.GetCurrentProcess();
// Set Core 0
process.ProcessorAffinity = new IntPtr(0x0001);
or
// Set Core 1
process.ProcessorAffinity = new IntPtr(0x0002);
More on this in "Process.ProcessorAffinity Property".