I'm currently working on a Windows service (my first) and I'm wondering how to handle disconnect events and the like. In essence, this Windows service polls our Exchange servers for new emails. Once an email is received we parse it and insert it into a database. Now, I have everything working so long as everything is working in my favour. Since that is impossible to maintain I need to look for ways to ensure my service stays on line regardless of what may happen that is out of my control (minus the server hosting the service that is).
The main issues I can foresee are our Exchange servers going down for whatever reason or losing internet connectivity. Two problems which can happen several times a year.
Currently, if an exception is thrown regarding connectivity issues I keep attempting to connect every n minutes with a 30 second time out. So say our Exchange servers go down (either planned maintenance or unforeseen events) for 2 hours then the service would try and reconnect every n minutes until a connection is made.
Is this a sustainable strategy to ensure my service always stays online? If not, what is a better way?
What I want to avoid is my service going down because Exchange had issues making me have to manually restart my Windows service.
Thank you.
Your strategy sounds like the only thing that's practical.
It may also be worth considering adding the ability to view event logs from the service remotely so you can diagnose issues that you don't currently know about. If you're really paranoid, a second "watcher" service could be used to periodically check the primary service and report if it fails.
Related
If this has been asked before my apologies, and this is .NET 2.0 ASMX Web services, again my apologies =D
A .NET Application that only exposes web services. Roughly 10 million messages per day load balanced between multiple IIS Servers. Each incoming messages is XML, and an outgoing message is XML. (XMLElement) (we have beefy servers that run on steroids).
I have a SLA that all messages are processed in under X Seconds.
One function, Linking Methods, in the process is now taking 10-20 seconds, it is required for every transaction, however is not critical that it happens before the web service returns the results. Because of this I made a suggestion to throw it on another thread, but now realize that my words and the eager developers behind them might have not fully thought this through.
The below example shows on the left the current flow. On the right what is being attempted
Effectively what I'm looking for is to have a web service spawn a long running (10-20 second) thread that will execute even after the web service is completed.
This is what, effectively, is going on:
Thread linkThread= new Thread(delegate()
{
Linkmembers(GetContext(), ID1, ID2, SomeOtherThing, XMLOrSomething);
});
linkThread.Start();
Using this we've reduced the time from 19 seconds to 2.1 seconds on our dev boxes, which is quite substantial.
I am worried that with the amount of traffic we get, and if a vendor/outside party decides to throttle us, IIS might decide to recycle/kill those threads before they're done processing. I agree our solution might not be the "best" however we don't have the time to build in a Queue system or another Windows Service to handle this.
Is there a better way to do this? Any caveats that should be considered?
Thanks.
Apart from the issues you've described, I cannot think of any. That being said, there are ways to fix the problem that do not involve building your own solution from scratch.
Use MSMQ with WCF: Create a WCF service with an MSMQ endpoint that is IIS hosted (no need to use a windows service as long as WAS is enabled) and make calls to the service from within your ASMX service. You reap all the benefits of reliable queueing without having to build your own.
Plus, if your MSMQ service fails or throws an exception, it will reprocess automatically. If you use DTC and are hitting a database, you can even have the MSMQ transaction flow to the DB.
We have a number of different old school client-server C# WinForm client-side apps that are essentially front-ends for the database. Then there is a C# server-side windows service that waits on the client apps to submit orders and then it processes them.
The way the server-side service finds out whether there is work to do is that it polls the database. Over the years the logic of polling for waiting orders has gotten a lot more complicated due to the myriad of business rules. So because of this, the polling stored proc itself uses quite a bit of SQL Server resources even if there is nothing to do. Add to this the requirement that the orders be processed the moment they are submitted and you got yourself a performance problem, as the database is being polled constantly.
The setup actually works fine right now, but the load is about to go through the roof and, it is obvious, that it won't hold up.
What are some effective ways to communicate between a bunch of different client-side apps and a server-side windows service, that will be more future-proof than the current method?
The database server is SQL Server 2005. I can probably get the powers that be to pony up for latest SQL Server if it really comes to that, but I'd rather not fight that battle.
There are numerous options ways you can notify the clients.
You can use a ready-made solution like NServiceBus, to publish information from the server to the clients or other servers. NServiceBus uses MSMQ to publish one message to multiple subscribers in a very easy and durable way.
You can use MSMQ or another queuing product to publish messages from the server that will be delivered to the clients.
You can host a WCF service on the Windows service and connect to it from each client using a Duplex channel. Each time there is a change the service will notify the appropriate clients or even all of them. This is more complex to code but also much more flexible. You could probably send enough information back to the clients that they wouldn't need to poll the database at all.
You can have the service broadcast a UDP packet to all clients to notify them there are changes they need to pull. You can probably add enough information in the packet to allow the clients to decide whether they need to pull data from the server or not. This is a very lightweight for the server and the network, but it assumes that all clients are in the same LAN.
Perhaps you can leverage SqlDependency to receive notifications only when the data actually changes.
You can use any messaging middleware like MSMQ, JMS or TIBCO to communicate between your client and the service.
By far the easiest, and most likely the cheapest, answer is to simply buy a bigger server.
Barring that, you are in for a development effort that has a high probability of early failure. By failure I don't mean that you end up scraping whatever it is you end up building. Rather, I mean you launch the changes and orders will be screwed up while you are debugging your myriad of business rules.
Quite frankly, I wouldn't consider approaching a communications change under pressure; presuming your statement about load going "through the roof" in the near term.
If your risk exposure is such that it has to be 100% functional day one (which is normal when you are expecting a large increase in orders), with no hiccups then just upsize the DB server. Heck, I wouldn't even install the latest sql server on it. Instead, just buy a larger machine, install the exact same OS and DB server (and patch levels) and move your database.
Then look at your architecture to determine what needs to go away and what can be salvaged.
If everybody connects to SQL Server then there is also the option of Service Broker. Unlike other messaging/queueing solution recommended so far it is entirely contained in your database (no separate product to deploy, administer and configure), it offers a single story vis-a-vis your backup/recovery and high availability needs ( no separate backup for message store, no separate DR/HA, whatever is your DB solution is also your messaging solution) and overs a uniform programming API (SQL).
Even when everything is within one single SQL Server instance (ie. there is no need to communicate over network between multiple SQL Service instances) Service Broker still has an ace that no one can match: activation. With activation you eliminate completely the need to poll because the system itself will launch your processing code (will 'activate') when there are events to process. The processing code can be internal (T-SQL procedure or SQLCLR .Net procedure) or external (see external activator).
Scenario
I have a Windows service that is in charge of listening for various things and act accordingly. Most of the time this service modifies the database in some fashion (adding, removing and cancelling items). This is on the servers side and requires no user involvement. The client side is quite simple. Each user (5-10) each use an application to work on the items inserted by the Windows service.
Problem
The issue is that the clients are working in a disconnected state. What I mean by that a client could reload their list of items needing to be worked on and 5 minutes later actually begin their work.
Also, due to various issues no administrative assistance must be required in order to install the client application. This rules out libraries such NServiceBus (or any library requiring the use of MSMQ) and WCF (seems like so because it requires the use of opening up URLs using netsh http add urcl.
I'm not looking to send messages to clients. All I'm looking for is an easy way to have the Windows service poke each subscribed client instructing them to reload their list. I do realize that I could simply create a background thread (or the like) and refresh every n minutes but there's really no sense in doing that (IMO) if nothing has happened in the last few hours.
Edit
Researching some more it appears I can use raw sockets for this. Will update once I get something working.
Take a look at 0MQ, it might meet your requirements.
I am kind of stumped with this one, and was hoping I could find some answers here.
Basically, I have an ASP.NET application that is running across 2 servers. Server A has all of the business logic/data access exposed as web services, and Server B has the website which talks to those services (via WCF, with net.tcp binding).
The problem occurs a few seconds after a recycle of my app pool is initiated by IIS on Server A. The recycle happens after the allotted time (using the default of 29 hours set in IIS).
In the server log (of Server A):
A worker process with process id of
'####' serving application pool
'AppPoolName' has requested a recycle
because the worker process reached its
allowed processing time limit.
I believe that this is normal behavior. The problem is that a few seconds later, I get this exception on Server B:
This channel can no longer be used to
send messages as the output session
was auto-closed due to a
server-initiated shutdown. Either
disable auto-close by setting the
DispatchRuntime.AutomaticInputSessionShutdown
to false, or consider modifying the
shutdown protocol with the remote
server.
This doesn't happen on every recycle; I assume that it happens when someone is hitting the site with a request WHILE the recycle happens.
Furthermore, my application is down until I intervene; this exception continues to occur every time a subsequent request is made to the page. I intervene by editting the web.config (by adding a space or something benign to the end of file) and saving it- I assume that that causes my application to recompile and brings the services back up. I also have experimented with running a batch file that does this for me every time the exception happens ;)
Now, I could barely find any information on this exception, and I've been looking for a while. Most of the information I did find pertains to WCF settings that I am not using.
I already read up on "DispatchRuntime.AutomaticInputSessionShutdown" and I don't think it pertains to this situation. This particular property refers to the service shutting down automatically in response to behavior on the client side, which is not what is happening here. Here, the service is shutdown because of IIS.
I did read this which went through some sort of work around to bring the service back up automatically, but I am really looking to understand what is going on here, not to hack around it!
I have started playing around with the settings in IIS7, specifically turning on/off Overlapped Recycling and increasing the process startup/shutdown times. I am wondering whether it is safe to turn off recycling completely (I believe if I put 0 for the recycling time interval?) But again, I want to know what's going on!
Anyway, if you need more information, let me know. Thanks in advance!
This is probably related to how you open and close WCF connections.
If you open a proxy when your app starts and then continue to use this, a break in the connection, which is caused by a restart on the server side. Results in a error on the client side, since the server that the proxy was talking to is no longer there.
When you restart the client side (changing the web.config) new proxies are created against a server that is running.
The way to fix this is to make sure that you close a WCF connection after you use it.
http://www.codeguru.com/csharp/.net/net_wcf/article.php/c15941/
You should also make sure that you're using the correct SessionMode for your Web Service. I remember having similar trouble with some of my Services until I sorted out the correct mode. This is especially true when you're mixing this with any other authentication mode that is not "None".
This link might have some pointer.
http://msdn.microsoft.com/en-us/library/ms731193.aspx
My suggestion is to simply stop using IIS to host your services. Unless there is something you really need from IIS, I would recommend just writing a standard Windows Service to host your WCF endpoints.
If you can't do that, then by all means turn off recycling. AppPool recycling is mainly there because web developers write crappy code. I know that sounds rather blunt, but if you have enough sense to write code that doesn't leak then there is no reason to have IIS constantly restart your program.
I have a windows service written in C# .NET framework 3.5 and would like to know the best way to check if previous shutdown of a service was regular.
Upon starting the service, there should be a check if the last shutdown was regular (via stop service button in services management) or if somebody just killed the process (or it crashed for some reason not directly linked to the service itself).
I thought about writing encrypted XML on a hard drive upon starting a service, and then editing it with some values when service is being stopped. In that way, after I start the service again next time, I could check the XML and see if the values were edited in correct way during shutdown, and if they were not I'd know the process was killed or it crashed.
This way seems too unreliable and not a good practice. What do you suggest?
Clarification:
What the service does is it sits on a server and listens to connections from client machines. Once the connection has been established, it communicates to a remote database via web services and determines whether they have right to connect (and therefore use application that is the caller). One of the aspects of protection is concurrency check, and if I have a limit set to 5 work stations, I keep the TcpClient connection alive from windows service to, let's say 5 workstations, and the sixth one cannot connect.
If I kill the service process and restart it, the connections are gone and I have 5 "licensed" apps running on workstations, and now there are 5 free connection slots to be taken by 5 more.
I also cant see anything bad using a file. You could even use this file to log some more information.
Eg. you could attach to the AppDomains Unhanded Exception event and try to log that exception.
Or you could evaluate how log your service has been running/not running (parsing a logfile for that task is a little bit harder).
Of course - this is not an excuse for not using logfiles.
I went with this in the end:
Service used to check up on the connected workstations to see if they're alive, but now I've built in periodical check from all the workstations as well (they connect through a common router dll where I've built in the check). Every 10 seconds the connection is verified, and if there is none, the client will try to reconnect in 15 seconds, which will be successful if there was just a temporary network problem, but will fail if the service was shut down forcefully (since all it's Tcp objects will be lost).
I would suggest to use the EventLog. Add a log event when a service start or stops and read through the event logs to detect anomalies.
Here's a basic sample from CodeProject.
Here's a walkthrough from MSDN how to create/delete/read event logs and entries.
Unless the service is running some sort securiy system that you need to have a "tamper" proof system i dont see why using a file is a bad solution.
Personaly i think a encrpted xml file is overkill, a simple text file should be enough.
I think you are on the right track, I'm not sure why you want to edit the values, just use the file (or a registry key) as a marker to indicate that the service was started and is running. During a graceful shutdown remove the marker. You then just need to look for the existence of the marker to know whether you were shutdown gracefully or crashed.
If you are finding that the file isn't created reliably, then make sure you are closing and flushing and disposing of the file object rather than relying on the garbage collector.
--- EDIT following clarification ---
So the requirement is for a licensing system and not simply to determine if the service was shutdown gracefully. I'm guessing that the desire is for the 'licenses' to be cleared on a graceful shutdown and restored following a crash, the scenarios are interchangeable.
I would probably use a database backing store, with suitable security, to hold the license keys at the server. As each client connects and requests a license they are provided with a key that has to be presented for each communication from the client. The server is obviously verifying that the presented key is valid for the current session. Should the server be gracefully shutdown it can clear the key table, if it crashes then the keys would still be present and can be honoured. That's probably the simplest approach I can think of that's secure.
If there's yet more to the story then let us know.