Intro:
Using VS 2013, .Net 4
creating a library to connect/use a Microsoft Access database (part of 3rd party application - choice of database is not an option) to be used by our parent product.
Reason to pool: connections being made by multiple tablet PCs located throughout an industrial facility. Concerns regarding performance.
What do I need to add to the connection string, how do I initialize it?
When and how do I kill it?
Has anyone dealt with this before?
Why:
Answers I have found so far are vaugue
For a System.Data.OleDb connection you apparently don't need to do anything to enable connection pooling. According to the MSDN article OLE DB, ODBC, and Oracle Connection Pooling (ADO.NET):
Connection Pooling for OleDb
The .NET Framework Data Provider for OLE DB automatically pools connections using OLE DB session pooling.
For an application using System.Data.Odbc you need to enable connection pooling for the Access ODBC driver by double-clicking the "Microsoft Access Driver ..." name on the "Connection Pooling" tab of the ODBC Administrator control panel (odbcad32.exe) and choosing "Pool Connections to this driver"
As stated in answers and comments to similar earlier questions (like this one), it's not too clear whether connection pooling will offer a significant benefit to an application that uses an Access database, but it is supported (ref: here, item #3) and it does seem to work based on what perfmon.exe displays for the "ODBC Connection Pooling" counters.
Just to have it available somewhere to help people (and to expand upon the info provided by #Gord Thompson), and since the documentation seems to be misleading at best...
In my experience and environment, Access MDB and OLEDB + Jet 4.0 do not have connection pooling on by default, it is actually off unless you turn it on. I'm pretty sure this is true for the ACE drivers as well, but I haven't recently done testing like I have for Jet.
Unlike ODBC (as far as I know), you can turn connection pooling on in the connection string with OLEDB. In your connection string, add the following command to turn on pooling (again, this default pertains to Access and may not be true of other databases): OLE DB Services = -1;
Here is a link to what different values are available and what they do
https://learn.microsoft.com/en-us/cpp/data/oledb/overriding-provider-service-defaults?view=msvc-170
Backup link in case MS does what MS does and kills doc links
https://www.ibm.com/docs/en/db2/11.1?topic=tips-connection-pooling
Does connection pooling help with Access?
Short answer, yes, it definitely does. To see the difference, brutalize Access by writing a loop that opens a connection, executes a query, then closes the connection and leave the "OLE DB SERVICES" command out, put a stopwatch in so you can time things and wrap it all up in the appropriate USING statements. You will see that the open/close overhead is significant and you will also very likely see Access will start choking on that overhead on occasion throwing errors like:
The database has been placed in a state by user 'Admin' on machine
'DESKTOP-XXXXXXX' that prevents it from being opened or locked.
It will also cause other applications connected to the database to choke with the same error occasionally.
Now turn on pooling in the connection string and you will see a HUGE performance increase and the occasional errors will go away.
This is not without drawback though. Access and OLEDB have an issue where if one connection writes data, it may not be available to other connections for up to 5 seconds or, in some cases even longer due to not only a read buffer, but Jet's built in async write to disk process that runs on another thread.
In many cases with OLEDB, when you open/close the connection and query data, data written from another connection is available much more quickly as I believe the connection open process starts you with a fresh read buffer. Connection pooling does exactly what you expect in this case and since the connection is already actually open, you are dealing with the existing connection buffer instead of starting clean and may have to wait up to 5 sec (or more) before data written by another connection is available.
Within the same app & pooling turned on, this may not be an issue because in the end, they'll likely (but probably not guaranteed to be) using the same connection anyways --- but if you have a different application (or same app on a different machine) needing data, you've got a problem.
So - depending on your environment and needs, connection pooling might be an asset or could be a liability.
I posted some additional info here on the buffer and ways to get around it
https://stackoverflow.com/a/74627552/2336839
Related
I am working in a small team of about a dozen developers on a project being written in C# WPF as the infrastructure/dba. As I run traces against the SQL Server to see how performance is going, what I am seeing is a constant:
open connection
run some statement
close connection
exec sp_reset_connection
open connection
run some statement
close connection
exec sp_reset_connection
open connection
run some statement
close connection
exec sp_reset_connection
And so on and on and on. I have spoke with the devs about this and some have mentioned possible situations where a foreach loop may be encompassing a using statement so a foreach through a datatable would open and close connections throughout the contents of the datatable.
Question: Is getting better control of the constant opening and closing of connections a worthy goal or are connections really that cheap? My logic is that while opening and closing a connection may relatively be cheap, nothing is cheap when done in sufficiently large numbers.
Details:
.Net Framework 4.5.1
SQL Server 2014
Entity Framework 6
If you use entity framework, you should create the context just before you need it and dispose it as soon as possible:
using (var someContext = new SomeContext())
{
}
The reason is to avoid memory building up and to avoid thread-safety issues.
Of course, don't do this in a loop - this is at the level of a request.
Opening and closing connections to a database are relatively expensive, as can be read in detail here: Performance Considerations (Entity Framework), but I think the same concept mostly applies without EF. During loops, it's usually not recommended to open and close the connection every time, but instead open it, process all rows and close the connection.
The answer would be to let the using encompass the loop, instead of the other way around. If performance is relevant (it almost always is), it definately pays to put effort into efficiënt data access, especially early in the development process.
If performance is an issue, but you don't want to refactor code you should consider setting a ConnectionPooling = true in the connections string.
Connection pooling allows one to keep physical connection, which is generally expensive to setup, while disposing logical connection.
Need some help from Oracle app developers out there:
I have an C#.NET 4.0 application which updates and inserts into a table using DDTek.Oracle library. My app runs everyday for about 12 hours and this exception came exactly twice and 15 days apart and never before. On these days, it was running fine for hours(it did both inserts and updates during this period). And then this exception comes. I have read that this exception could be from a bad connection string, but as I said before, the app has been running fine for a while. Could this be a db or network issue or could it be something else?
System.InvalidOperationException: Connection must be valid and open
at DDTek.Oracle.OracleConnection.get_Session()
at DDTek.Oracle.OracleConnection.BeginDbTransaction(IsolationLevel isolationLevel)
at DDTek.Oracle.OracleConnection.BeginTransaction()
FYI(if this could be the cause), I have two connections on two threads. Each thread updates a different table.
PS: If anyone know a good documentation for DDTek. Please reply with a link.
From what you describe I can only speculate - there are several possibilities:
most providers offer built-in pooling, sometimes a connection in the pool becomes invalid and you get some strange behaviour
sometimes the network settings/firewalls/IDS... limit how long a TCP connection can stay open
sometimes a subtle bug (accessing same connection from 2 different threads) leads to strange problems
sometimes the DB server (or a DB firewall) limits how long a session can stay connected
sometimes memory problems cause such behaviour, almost every Oracle provider uses OCI under the hood which requires using unmanaged heap etc.
I had one provider leaking unmanaged memory (diagnosed via a memory profiler and fixec by the vendor rather fast)
sometimes when connected to a RAC one listener/node goes down and/or some failover happens leaving current connections invalid
As for a link to comprehensive documentation of DDTek.Oracle see here and here.
I'm modifying a Winforms app to use connection pooling so data access can occur in background threads. The business logic is implemented in PL/SQL and there are a couple of security related stored procedures that have to be called in order to make use of the business logic.
What I need is a way to tell if the connection has been used without a round-trip to the database. I don't think I can keep track of them in a HashSet because I doubt Equals or even ReferenceEquals could be relied upon. Any ideas?
EDIT:
Just to be clear, I plan to use ODP.NET's built-in connection pooling mechanism. If I rolled my own connection pool, keeping track of which connections were new vs. used would be extremely trivial.
The connection pooling provided by ODP.NET is completely opaque. That is, it isn't leaky in the way I'd like it to be - there is no way of knowing if a connection has been used before or is brand new. However it is a leaky abstraction in another way: Any session state (e.g. package scoped variables, which are session scoped) is preserved between usages of the connection. Since this is a question about determining the used vs. new state of a connection without going to the database, the answer is that it simply cannot be done using ODP.NET's built-in connection pool.
That leaves two options:
Create a connection pool implementation that either provides that information or performs user-defined initialisation upon creation of each new connection; or
Perform a round-trip to the database to determine the used vs. new state of the connection.
ADO.NET manages a connection pool for you. It's even configurable. Why would you ever try to track these connections yourself?
http://msdn.microsoft.com/en-us/library/bb399543.aspx
And, specifically for Oracle:
http://msdn.microsoft.com/en-us/library/ms254502.aspx
The .NET Framework Data Provider for Oracle provides connection
pooling automatically for your ADO.NET client application. You can
also supply several connection string modifiers to control connection
pooling behavior (see "Controlling Connection Pooling with Connection
String Keywords," later in this topic).
Pool Creation and Assignment
When a connection is opened, a connection pool is created based on an
exact matching algorithm that associates the pool with the connection
string in the connection. Each connection pool is associated with a
distinct connection string. When a new connection is opened, if the
connection string is not an exact match to an existing pool, a new
pool is created.
Once created, connection pools are not destroyed until the active
process ends. Maintaining inactive or empty pools uses very few system
resources.
BTW, I guess I'm not totally hip on all the OracleClient changes that have been going on. It seems like Microsoft may be dropping support? Last I knew ODP.NET was based on ADO.NET... but, even if I'm mistaken about that, ODB.NET claims to support connection pooling out of the box as well:
http://download.oracle.com/docs/html/E10927_01/featConnecting.htm#CJAFIDDC
If what you need is to just know whether you ever had some connections not come from pool but a fresh new one, I think that you can use the HardConnectsPerSecond and SoftconnectsPerSecond performance counter provided by ODP.NET.
This won't tell you exactly which OracleConnection.Open() leads to a hard connection, though. I was also thinking about combining other ODP.NET perf counter to determine if a new hard connection is created, but after some experiments this is not easy because ODP.NET will also purge connections every three minutes (depending on the Decr Pool Size setting).
I work with Windows-Mobile and Windows-CE using SqlCE and I dont know what better to do.
To open connection when the program open, run any query's... update...delete database and close the connection after the program close?
Or open connection run any query's..update...delete database and close the connection immediately?
Nice. The answers are all over the place. Here's what I know from experience and interacting with the SQL Compact team:
Closing the connection flushes the changes you've made, otherwise the engine waits for the flush period before doing it. It's a good idea to close the connection when you're done using it to ensure that your changes actually go to the store. A power loss after a write and before a flush will lose data.
There is no official connection pool, but opening the first connection is expensive (i.e. slow), all others are quick. The recommendation I got from the team is to actually create a connection when the app starts up and just leave it open. You don't actually need to use it, but keeping it open keeps a lot of connection info cached so that subsequent connections to the same store are quick.
So the answer, actually, is both.
Edit
For those interested, a good example of how this works can be seen in the OpenNETCF ORM library. The library, by default, creates a "maintenance" connection that remains open and is used for doing things like schema queries. All other data operations use their own connection. You also have to option to configure the library to reuse a single connection for the life of the Store, or to use a new connection every time it touches the store. Perfomance and behavior has always been best in all of my projects using the default (which is why I made it the default).
Always keep a connection open for the lifetime of your Windows Mobile app. Opening a SQL Server Compact database is a costly operation.
You should close your connection each time you have completed an sql transaction to free connection ports. Always a good practice to avoid security breach.
Connection establishment is a slow operation, so, creating and closing it can slow down the application. On the opposite hand, if you have a lot of clients, the connection pool will be filled very quickly and other clients won't be able to connect.
There are already some conflicting answers here.
To be honest, I'm not enirely sure how WinCE deals with connections. I don't think there is a ConnectionPool.
But the general pattern in .NET is to keep connections open as short as possible. This improves reliability and prevents resource leaks. Make sure you know about the using (var conn = ...) { ... } pattern.
So I would say: go with your second option, and only keep connections longer if you really experience a performance problem, and if opening the connection is the cause. I don't think it will be with SqlCE
On a single-user platform such as wince, there's no harm in keeping the connection open, and you may get better performance.
If worry about data lost because you are not calling Close() frequently, you can execute your code within a transaction that commits changes to disk immediately:
using (SqlCeTransaction transaction = this.connection.BeginTransaction())
{
using (SqlCeCommand command = new SqlCeCommand(query, connection))
{
command.Transaction = transaction;
command.ExecuteNonQuery();
}
transaction.Commit(CommitMode.Immediate);
}
Of course, there is still some performance lost when using CommitMode.Immediate too frequently.
Is it smart to keep the connection open throughout the entire session?
I made a C# application that connects to a MySql database, the program both reads and writes to it and the application has to be running about 10 hours a day non-stop.
Are there any risk attached to keeping the connection open instead of calling the close() function every time after you've plucked something from the database and opening it again when you need something new?
Leaving a connection open for a while is fine, as long as:
you don't have so many concurrently idle connections that you hit the MySQL connection limit;
you don't leave it open for hours without doing anything. The default MySQL connection wait_timeout is 8 hours; leave a connection inactive for that long and when you next come to use it you'll get a “MySQL server has gone away” error.
Since you're using ADO.NET, you can use ADO.NET's inbuilt connection pooling capabilities. Actually, let me refine that: you must always use ADO.NET's inbuilt connection pooling capabilities. By doing so you will get the .NET runtime to transparently manage your connections for you in the background. It will keep the connections open for a while even if you closed them and reuse them if you open a new connection. This is really fast stuff.
Make sure to mention in your connection string that you want pooled connections as it might not be the default behaviour.
You only need to create connections locally when you need them, since they're pooled in the backrgound so there's no overhead in creating a new connection:
using (var connection = SomeMethodThatCreatesAConnectionObject())
{
// do your stuff here
connection.Close(); // this is not necessary as
// Dispose() closes it anyway
// but still nice to do.
}
That's how you're supposed to do it in .NET.
Yes you can, provided:
You will reconnect if you lose the connection
You can reset the connection state if something strange happens
You will detect if the connection "goes quiet", for example if a firewall timeout occurs
Basically it requires a good deal of attention to failure cases and correct recovery; connecting and disconnecting often is a lot easier.
I think, if there is a connection pooling mechanism, you'd better close the connection.
One reason for it is that you do not need to re-check if your connection is still alive or not.
If the application is using the connection there is no reason to close it. If you don't need the connection you should close it. If you were to have multiple applications connect to the database, you have a fixed number of connections to that database. That's why it's better to close when you are done and reopen when you need it.
From a security point of view, I'd say its better to close it after a query, just to be sure that no other program can inject it's own things into the opened connection.
As performance is conered, it is clearly better to have the connection opened through the whole time.
Your choice^^
No, I don't see any reason why not to leave a connection open and re-use it: after all, this is the whole point behind the various connection-pool technologies that are about (although these are generally reserved for multi-threaded situations where works are all operating on the same data source).
But, to expand on the answer by bobince, - just beacause you are not closing the connection, don't assume that something else won't: the connection could timeout, there could be connection issues or a hundred and one other reasons why your connection dies. You need to assume that the connection may not be there, and add logic to code for this exception-case.
It is not good practise in my opinion to keep the connections open.
Another aspect that speaks for closing connections every time is scaleability. It might be fine now to leave it open but what if you app is used by twice 3-times the amount of users. It's a pain in the neck to go back and change all the code. (i know i've done it :-)
Your problem will be solved if you use connection pooling in your code. You don't need to open and close connection so you save precious resources which are used while opening a connection. You just return the connection to a pool which when requested for a connection returns back a idle connection.
Of course I am of the opinion, get an instance of the connection, use it, commit/rollback your work and return it to the pool. I would not suggest keeping the connection open for so long.
One thing I didn't see in the other answers, yet: In case you have prepared statements or temporary tables they might block server resources till the connection is closed. But on the other hand it can be useful to keep the connection around for some time instead of recreating them every few moments.
You'll pay a performance penalty if you're constantly opening and closing connections. It might be wise to use connection pooling and a short wait_timeout if you are concerned that too many running copies of your app will eat up too many database connections.