This is driving me crazy. We use a fairly large number of private MSMQ queues in our C#/ASP.NET web application where I work and have a common library to send and receive messages from our queues. Yesterday, this stopped working for me altogether, but none of the other developers I work with are running into this issue, which makes me think it has something to do with my local dev environment or my Windows account settings.
I am now always getting "Timeout for the requested operation has expired" exceptions when the following line of messaging code is called:
var returnMessage = fromMessageQueue.ReceiveByCorrelationId(strCorrelationID, tsWait);
We basically have an "Inbound" and "Outbound" queue for each of our (business) clients. The Inbound queues look clean, but when I look in the Outbound queues, I can see "stuck" messages that are the responses I need.
I've even written a small test console application against a dummy queue I setup for troubleshooting, that still returns the same timeout exceptions.
I've checked the permissions on the private queues I've been troubleshooting with, EVERYONE and ANONYMOUS users have full control to the queues. I've even granted my own domain login account to a few queues, but that didn't work either.
I'm afraid I'm very stuck until I can get this resolved.
I usually get this when I have installed the software and have it running as a service whilst trying to run a debug copy through visual studio (2 services running on 1 queue)
Related
I am hitting into a problem with my company application.
I am going to summarize the system key elements:
My company's system is running since few years on Windows XP and 7 (Home, Pro, Basic) machines.
It has been written in .NET 4.0 and based upon WCF.
It uses the default throttling values (MaxConcurrentSessions = 100 * CPU (4) : enough for our workload).
The main service is hosted by a stand alone deamon process (not IIS).
The main service is configured as Multithraded/PerSession instances.
The protocol is Reliable NET.TCP.
No more than 10 clients access concurrently the service.
The problem is that only on Windows 7, intermittently, I get (I discovered that by the WCF full trace log) a "Server too busy exception" due to an exhausted MaxConcurrentSessions limit (impossible!!!).
Do you have any idea about this strange behaviour?
Thank you and have a Happy New Year!
Antonio
Do all your Clients properly close/dispose connection to service after use ? It's worth to check, "ghost" connections could maybe explain this.
We experienced a similar issue with a self-hosted WCF interface which provided a synchronous request/response web service for an asynchronous (2 one way service calls) backend request. Early in our testing, we noticed that after a somewhat variable amount of time, our service became unresponsive to new requests. After some research, we discovered that whenever the backend service (out of our control) did not send a response, we continued to wait indefinitely and as such we kept our client connection open.
We fixed the issue by providing a “time-to-wait” configuration value so we were sure to respond to the client and close the connection. We used something like the following …
Task processTask = Task.Factory.StartNew(() => Process(message));
bool isProcessSuccess = processTask.Wait(shared.ConfigReader.SyncWebServiceWaitTime);
if (!isProcessSuccess)
{
//handle error …
}
The following link, which provides information regarding WCF Service performance counters, may help further determine if the calls are being closed as expected. http://blogs.microsoft.co.il/blogs/idof/archive/2011/08/11/wcf-scaling-check-your-counters.aspx
Hope this helps.
Regards,
If this has been asked before my apologies, and this is .NET 2.0 ASMX Web services, again my apologies =D
A .NET Application that only exposes web services. Roughly 10 million messages per day load balanced between multiple IIS Servers. Each incoming messages is XML, and an outgoing message is XML. (XMLElement) (we have beefy servers that run on steroids).
I have a SLA that all messages are processed in under X Seconds.
One function, Linking Methods, in the process is now taking 10-20 seconds, it is required for every transaction, however is not critical that it happens before the web service returns the results. Because of this I made a suggestion to throw it on another thread, but now realize that my words and the eager developers behind them might have not fully thought this through.
The below example shows on the left the current flow. On the right what is being attempted
Effectively what I'm looking for is to have a web service spawn a long running (10-20 second) thread that will execute even after the web service is completed.
This is what, effectively, is going on:
Thread linkThread= new Thread(delegate()
{
Linkmembers(GetContext(), ID1, ID2, SomeOtherThing, XMLOrSomething);
});
linkThread.Start();
Using this we've reduced the time from 19 seconds to 2.1 seconds on our dev boxes, which is quite substantial.
I am worried that with the amount of traffic we get, and if a vendor/outside party decides to throttle us, IIS might decide to recycle/kill those threads before they're done processing. I agree our solution might not be the "best" however we don't have the time to build in a Queue system or another Windows Service to handle this.
Is there a better way to do this? Any caveats that should be considered?
Thanks.
Apart from the issues you've described, I cannot think of any. That being said, there are ways to fix the problem that do not involve building your own solution from scratch.
Use MSMQ with WCF: Create a WCF service with an MSMQ endpoint that is IIS hosted (no need to use a windows service as long as WAS is enabled) and make calls to the service from within your ASMX service. You reap all the benefits of reliable queueing without having to build your own.
Plus, if your MSMQ service fails or throws an exception, it will reprocess automatically. If you use DTC and are hitting a database, you can even have the MSMQ transaction flow to the DB.
I've been working on a Azure worker role that hosts a netTcpRelayBinding WCF service. All seems to work well until One of my connected hosts disconnects unexpectedly. Over the next couple of minutes, the role consistently loses stability and then reports itself as unhealthy.
I'm not sure where I should be looking. I've got IntelliTrace enabled, and I've got some exceptions, which start with the TimeoutException you'd expect, but then continue on. I get these messages:
System.ServiceModel.CommunicationException - Inactivity timeout of (00:00:10) has been exceeded.
System.InvalidProgramException - The Common Language Runtime detected an invalid program
After this, I get a series of communication exceptions, timeoutexceptions, and then eventually the whole host crashes with an OutOfMemoryException.
Things to note: I've got 1 client connected. No other calls or activity. When he disconnects unexpectedly, the above consistently happens.
Tried catching the servicehost Faulted but that seemed to do nothing (can't see where it was hit in IntelliTrace logs.
Any ideas on where I should be looking? Surely I don't need to recreate the service every time something like this happens right?
That doesn't sound familiar. Can you reproduce this and how's your role feeling overall, meaning what's the CPU load looking like and what else is going on in parallel?
Cheers,
Clemens
I have an Azure web role that accesses an external WCF based SOAP web service (port 80) for various bits of data. The response from this service is highly erratic. I routinely get the following error.
There was no endpoint listening at
http://www.myexternalservice.com/service.svc that could accept the message. This is
often caused by an incorrect address or SOAP action.
To isolate the problem I created a simple console app to repetitively call this service in 1 second intervals and log all responses.
using (var svc = new MyExternalService())
{
stopwatch.Start();
var response = svc.CallService();
stopwatch.Stop();
Log(response, stopwatch.ElapsedMilliseconds);
}
If I RDP to one of my Azure web instances and run this app it takes 10 to 20 attempts before it gets a valid response from the external service. These first attempts are always accompanied by the above error. After this "warm up period" it runs fine. If I stop the app and then immediately restart, it has to go back through the same "warm up" period.
However, if I run this same app from any other machine I receive valid responses immediately. I have run this logger app on servers running in multiple data centers (non Azure), desktops on different networks, etc... These test runs are always very stable.
I am not sure why this service would react this way in the Azure environment. Unfortunately, for the short term I am forced to call this service but my users cannot tolerate this inconsistency.
A capture of network traffic on the Azure server indicates a large number of SynReTransmit's in 10 second intervals during the same time I experience the connection errors. Once the "warm up" is complete the SynReTransmit's no longer occur.
The Windows Azure data center region where the Windows Azure application is deployed might not be near the external Web Service. The local machine you're trying (which works fine) might be close to the web service. That’s why there might be huge latency in Azure which would likely cause it to fail.
Success accessing WSDL from a browser in Azure VM might be due to browser caching. Making a function call from browser would tell you if it is actually making a connection.
We found a solution for this problem although I am not completely happy with it. After exhausting all other courses of action we changed the load balancer to Layer-7 Load Balancing from Layer-4 Load Balancing. While this fixed the problem of lost requests I am not sure why this made a difference.
I am working on one Asp.Net application and need to send mails periodically based on some event. First I thought of creating a thread in global.asax and start thread in application_start. But that becomes a bit of problem when application pool crashes or something. So I implemented a windows service and started thread in that and log any errors in windows event log. This works fine. But I need to know whether I am implementing it correctly or is there a better way of doing it?
I think you are moving (or moved already) to the right direction.
We have similar architectures as well, in some cases we used MSMQ to queue outgoing notifications from the ASP.NET application then the Windows Service, usually called Messaging Manager, can grab asynchronously the incoming messages and send the emails or alerts out.
this proves to be effective and robust, if anything crashes after the message has been queued, nothing will be lost because the windows service will always process the messages in the queue, so you can have ASP.NET recycling or the machine with the windows service being rebooted, nothing is lost ever. And in fact in normal production mode, messages are sent out instantly, the decoupling or loose of sync is mostly hidden when everything is working smooth and servers are not overloaded or suffering anything.
In a later project we are now implementing something similar using TIBCO technologies, EMS for the queues and Business Works for queue subscribers.
Using a Windows Service for this kind of tasks is the preferred way instead of doing it in the ASP.NET application. You may also take a look at Quartz.NET which could simply your code for scheduling the task execution and dealing with threads. But if you don't want to write Windows Services probably the simplest would be to have a console application that will do the job of sending emails and then simply use the Windows Task Scheduler to run it at regular intervals.
Another option is a message-based approach. You could have a Windows Service/Console Application reading messages of a message queue (like msmq) and send email when a message is recieved. You can then have your ASP.NET application publish messages to this queue.
Minibuss is a lightweight client for msmq which is very easy to work with. Another options is NServiceBus.