We have a ServiceStack service (API) which provides HTTP endpoints hosted using AppSelfHostBase.
Those services later query database using ServiceStack.OrmLite.MySql. All methods are implemented using async / await pattern. Database connections are registered manually to Funq with Request reuse scope, and injected to the property of base DAL class.
This all works fine when this service is accessed only by HTTP requests.
We have another Windows service which calls this API. Since they could be hosted on the same server, we’ve implemented local IRestClientAsync for wrapping service calls, so the API service methods could be loaded to the Windows service, and accessed more efficiently (eg 1200 req/sec compared to 400 req/sec). This Windows service has a lot of threads running at the same time.
By doing this, we broke the Request lifecycle and are getting
“There is already an open DataReader associated with this Connection which must be closed first.”
error. We tried handling this manually using custom connection providers separating connections through threads using ThreadLocal and CallContext. This didn’t work all the time.
We tried handling Request lifecycle by calling OnBeginRequest(null); and OnEndRequest(); manually, but the performance was bad (close to HTTP calls) and also, got “open DataReader” errors.
We are using RequestContext.UseThreadStatic option, since the threads are instantiated from Quartz .NET job.
What could be the best solution for managing database connections? Can we make the current solution working reliably?
First thing I would do is not bother using the Async API's with MySql since it's not truly asynchronous as it ends up creating new threads behind the scenes to fake asynchrony which makes it even less efficient then using the Sync API's. You also can't use multiple readers with the same db connection which will end up throwing this exception.
So I'd firstly go back to using sync API's with MySql, if it's still an issue use transient scope (i.e. no re-use) instead of Request scope and let the db connection pooling do its job. Request scope holds on to the connection longer, taking up more resources than necessary.
Related
The following link http://msdn.microsoft.com/en-us/library/hh211418(v=vs.110).aspx explains the new asynchronous programming features with the 4.5 .Net framework, which for the most part is fairly straight forward reading.
However, I just want to make sure of something....
If I am using a SQLDataReader in a web service and I create a service reference on the client to use that web service by generating asynchronous methods in the proxy (via the service reference dialog options), I very well imagine I wouldn't need to use the methodologies mentioned in the link above.
Rather, I would use the async, Task, and await keywords on the client appropriately when calling that web service method
Since checking those options on the Web Service dialog, it will automatically create asynchronous method calls for the ADO.Net call.
So, if you have a method called GetCategories in the web service, it will automatically create an asynchronous call called GetCategoriesAsync in the web service which could be called from the client. Again, no need to place the asynchronous attributes on the web service method call; only for an ADO.Net call which is not using a web service or one which is using a web service, but does not have the asynchronous options checked.
Am I correct in my thinking?
It depends on what you want to make asynchronous.
Network communications are inherently asynchronous. When Visual Studio creates a client proxy for your web service, it creates both asynchronous and synchronous methods (where the synchronous methods just block waiting for a response). So, your web client can be asynchronous, and this is true regardless of how the server is implemented. In other words, a server can be synchronous or asynchronous, and a client can be synchronous or asynchronous, completely independent from each other.
On the client side, you should almost always use asynchronous methods on the proxy. The primary benefit on the client is responsiveness.
On the server side, you can get a scalability benefit by using an asynchronous implementation. However, if your backend is a single SQL server database, and every request hits that database, then there usually isn't a benefit from making your web service asynchronous. This is because (in that case) the scalability bottleneck is the SQL server, not the web server. OTOH, if your backend is a SQL server cluster or Azure SQL, then you can get a scalability benefit by using an asynchronous web service implementation.
The old-style common scenario was client <-> API <-> DB, and in that architecture there's no real need for asynchronous DB access. However, more and more architectures are looking like client <-> API <-> cloud storage / APIs, and that's when async really brings benefits to the API layer.
I'm currently in the process of building an application that receives thousands of small messages through a basic web service, the messages are published to a message queue (RabbitMQ). The application makes use of Dependancy Injection using StructureMap as its container.
I have a separate worker application that consumes the message queue and persists the messages to a database (SQL Server).
I have implemented the SQL Connection and RabbitMQ connections as singletons (Thread Local).
In an ideal world, this all works fine but if SQL Server or RabbitMQ connection is broken I need to reopen it, or potentially dispose and recreate/reconnect the resources.
I wrote a basic class to act as a factory that before it returns a resource, checks it is connected/open/working and if not, dispose it and recreate it - I'm not sure if this is "best practice" or if I'm trying to solve a problem that has already been solved.
Can anyone offer suggestions on how I could implement long running tasks that do a lot of small tasks (in my case a single INSERT statement) that don't require object instantiation for each task, but can gracefully recover from errors such as dropped connections?
RabbitMQ connections seem to be expensive and during high work loads I can quickly run out of handles so I'd like to reuse the same connection (per thread).
The Enterprise Library 5.0 Integration Pack for Windows Azure contains a block for transient fault handling. It allows you to specify retry behavior in case of errors.
It was designed with Windows Azure in mind but I'm sure it would be easy to write a custom policy based on what it offers you.
You can make a connection factory for RabbitMQ that has a connection pool. It would be responsible for handing out connections to tasks. You should check to see that the connections are ok. If not, start a new thread that closes/cleans the connection then returns it to the thread pool. Meanwhile return a functioning connection to the user.
It sounds complicated but it's the pattern for working with hard to initialize resources.
I have an application that sends email and fax notifications when an item is complete. My implementation is working but it takes several seconds to construct (i.e. connecting to our servers). This ends up freezing the UI for several seconds until the notification services have been fully constructed and used. I'm already pushing the problem as far as it will go by injecting factories and creating my services at the last possible minute.
What options do I have for injecting external services that takes several seconds to construct? I'm thinking of instructing my container that these services are singletons, which would only construct the services once per application start-up.
Simple solution
I would give those services a longer lifetime (singleinstance).
The problem though is that TCP connections are usually disconnected if nothing have happened for a while. Which means that you need to have some kind of keep alive packets to keep them open.
More robust solution
imho the connection setup is not the problem. Contacting a service should not take long. You haven't specified what the external services are. I'm guessing that they are some kind of web services hosted in IIS. If so, make sure that the application pools aren't recycled too often. Starting a new application in IIS can take time.
The other thought is if you really need to wait for the service to complete? Why not queue the action (to be handled by the thread pool or in a separate thread) and let the user continue? If required, simply use a messagebox when the service have been called and something failed.
These services should be bootstrapped on application startup and then configured via DI using a singleton which is then injected to any classes that use the class in their constructor.
I can recommend Unity or Spring.Net. I've found Unity very easy to use for simple injection, so give that a look.
Im writing an application which transfers files using WCF. The transfers are done in segments so that they can be resumed after any unforeseen interruption.
My question concerns the use of the client side proxy, is it better to keep it open and reuse it to transfer each file segment or should I be reopening it each time I want to send something?
The reason to close a proxy as quickly as possible is the fact that you might be having a session in place which ties up system resources (netTcpBinding uses a transport-level session, wsHttpBinding can use security or reliability-based sessions).
But you're right - as long as a client proxy isn't in a faulted state, you can totally reuse it.
If you want to go one step further, and if you can share a common assembly with the service and data contracts between server and client, you could split up the client proxy creation into two steps:
create a ChannelFactory<IYourServiceContract> once and cache that - this is a very expensive and resource-intensive operation; since you need to make this a generic using your service contract (interface), you need to be able to share contracts between server and client
given that factory, you can create your channels using factory.CreateChannel() as needed - this operation is much less "heavy" and can be done quickly and over and over again
This is one possible optimization you could look into - given the scenario that you control both ends of the communication, and you can share the contract assembly between server and client.
You can reuse your WCF client proxy and that will make your client application faster, as proxy just will initialize once.
Creation of a new proxy takes up about 50-100 ms of time, if your system needs good scaling, it's quite a significant time.
When reusing the proxy, you have to be careful of its state and threading issues. Do not try to send data using a proxy that is already busy with sending data. (or receiving) You'll have terrible sleepless nights.
One way of reusing is, having a [ThreadStatic] private field for the proxy and testing its state & presence each time you need to send data. If a new thread was created, the thread static field will be null and you'll need to create a proxy. Assuming you have a simple threading model, this will keep the different threads from stepping on each other's toes and you'll have to worry only about the faulted state of the proxy.
We are currently working on an application that will use a WCF service. The host (the client) is using the excellent WCF Service Proxy Helper from Erwyn van der Meer.
What I would like to know... is if I open this object multiple times... will it lead to multiple (expensive) connections or will WCF manage it and pool the connections.
The reason why I want to know this is because we will be calling methods of the service at different point in time within the same web request and we currently have wrapped the instanciation of the Service proxy class within the call.
Eg.:
MyService.MyMethod1() // wraps the connection usage as well as the call to the service
Any suggestions about how I would go to minimize the amount of connection while keeping the code conform with SRP would be excellent.
So? Any idea?
You should try to minimize the number of proxy objects you create. Proxies in WCF are quite expensive to set up, so creating one and calling functions on it multiple times is definitely more efficient than creating a new one for each method invocation.
The relationship between proxy objects and connections depends on the transport used. For http transports, an HTTP connection is initiated for each function invocation. For the net.tcp transport, the connection is established at Open() time and kept until a Close(). Certain binding settings (eg those supporting WS-SecureConversation) will incur extra "housekeeping" connections and message exchanges.
AKAIK, none of the out-of-the-box bindings perform connection pooling.
It doesn't do pooling like SqlConnection, if that is what you mean.
[caveat: I use "connection" here loosely to mean a logical connection, not necessarily a physical connection]
Between using a connection on-demand, and keeping one around, there are advantages and disadvantages to both approaches. There is some overhead in initializing a connection, so if you are doing 5 things you should try to do them on the same proxy - but I wouldn't keep a long term proxy around just for the sake of it.
In particular, in terms of life-cycle management, once a proxy has faulted, it stays faulted - so you need to be able to recover from the occasional failure (which should be expected). Equally, in some configurations (some combinations of session/instancing), a connection has a definite footprint on the server - so to improve scalability you should keep connections short-lived. Of course, for true scalability you'd usually want to disable those options anyway!
The cost of creating a connection also depends on things like the security mode. IIRC, it will be more expensive to open a two-way validated connection using message security than it will to set up a TransportWithMessageCredential connection - so "YMMV" is very much the case here.
Personally, I find that the biggest common issue with proxy performance isn't anything to do with the time to set up a conncetion - it is the chattiness of the API. i.e.
Scenario A:
open a connection
perform 1 operation with a large payload (i.e. a message that means "do these 19 things")
close the proxy
Scenario B:
open a connecton
perform 19 operations with small payloads
close the connection
Then scenario A will usually be significantly faster due to latency etc. And IMO don't even think about distributed transactions over WCF ;-p