I've got problem with the connection pulling with continuously running program. The problem occurs when I'm doing a lot of queries (every 4 minutes 5x (querying 3 tables and saving result to one)) to DB in the Tasks. The connection pools run out of max pool connection size. The strange thing about this that I have on DB 100 of AWAITING COMMAND entries for that particular connection string / machine / user entries. My understanding is that AWAITING COMMAND means that this connection can be reused, but from some strange unknown reason to me when running commands from Tasks cannot reuse available connections and they just wait for no one, and after some time got error that I've reached the max pool connections size.
Assumptions so far:
When running commands from tasks DB interpret this as invalid to reuse available connections
Connections aren't closing, but why? Seems to closing them with using keyword. More over that is 100 AWAITING COMMANDS one the DB.
The handlers aren't garbage collected for some reason? But the 100 AC telling sth else.
UPDATE: LOCALDB OBSERVATIONS/SUMMARY:
When I'm trying to replicate this on local DB SQL Server Express this problem happen in very awkward situation. I had to add the Thread.Sleep(600000) to kind a simulate the situation. And eventually after that I was able to get the max pool error, but in this case all connections are open so its rather self explanatory.
In local -> server scenario, I don't think so that I could have 100 connections open in one time, they rather stay open for some reason. When launching this program on the localMachine -> serverDB situation I don't even need to add the Thread.Sleep(600000) in order to crash program.
All those are my assumptions based on observations. I can't think of what casing this in my continuous running service where querying the DB every 4 minutes.
PS. After my full local testing I'm confused if COMMAND AWAITING means that this connection can be reused?
UPDATE 2 Forgot to mention that my initial program can run couple of days before I eventually encounter this max pool error.
Below is the program that can generate this kind of problem:
using System;
using System.Data.SqlClient;
using System.Threading;
using System.Threading.Tasks;
namespace Pooling
{
class Program
{
private static int connectionIterations;
private static string connectionString = "Data Source=localhost;Initial Catalog=localDB;Integrated Security=True";
static void Main(string[] args)
{
try
{
Iterations();
while(true)
{
ConnectionSnowball();
}
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
private static void ConnectionSnowball()
{
Parallel.For(0, connectionIterations, i =>
{
try
{
Console.WriteLine($"Connection id: {i}");
using (SqlConnection connection = new SqlConnection(connectionString))
{
SqlCommand cmd = new SqlCommand("SELECT 1 FROM test_table", connection);
connection.Open();
cmd.ExecuteNonQuery();
Thread.Sleep(600000);
}
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
});
}
private static void Iterations()
{
connectionIterations = 200;
}
}
}
I debugged your code and found no connection leaks. You just have a connection pool overflow. I checked two possible solutions for you.
Disable pooling connections
private static string connectionString = "Data Source=localhost;Initial Catalog=localDB;Integrated Security=True;Pooling=False";
Increase connection pool
private static string connectionString = "Data Source=localhost;Initial Catalog=localDB;Integrated Security=True;Max Pool Size=200";
To test how the connections will increase and decrease before, during and after the ConnectionSnowball() call, you can use this SQL query
select count(1) from sys.dm_exec_sessions where database_id = DB_ID(N'localDB')
More details about connection string parameters
SqlConnection.ConnectionString Property
Other possible solutions is the use of SQL jobs. For this task, this may be a more appropriate solution, since a large number of connections are very resource intensive.
As there are no connection leaks in your code, Did you try Restarting IIS?
Related
I would like to detect connection state to MySql database. My database is deployed in different server than my app and there is good chances to lose connection to it via network. So I have to take this scenario into consideration.
Here is what I tried so far (a simplified test example):
static string connectionString = "***";
public static MySqlConnection Connection;
static System.Timers.Timer _timer;
static void _timer_Elapsed(object sender, ElapsedEventArgs e)
{
try
{
if (Connection.State != System.Data.ConnectionState.Open)
Connection.Open();
// Call method to invoke MySqlCommand.ExecuteNonQuery
mysqlCommand.ExecuteNonQuery();
}
catch (MySqlException ex)
{
Console.WriteLine("SQL EXCEPTION: " + ex);
// Handle all type of database exceptions
switch(ex.Number)
{...}
}
catch (Exception ex)
{
Console.WriteLine("OTHER EXCEPTION: " + ex);
}
}
static void Main(string[] args)
{
Connection = new MySqlConnection(connectionString);
_timer = new System.Timers.Timer(3000);
_timer.Elapsed += new ElapsedEventHandler(_timer_Elapsed);
_timer.Enabled = true;
Console.ReadKey();
}
If the connection to MySql is lost, I got a general exception:
IOException : Unable to write data to the transport connection: An
established connection was aborted by the software in your host
machine.
I was expecting MySqlException to be fired but that was not the case.
Also, if the connection to MySql is restored, I still get the IOException instead of executing the query. Seems like, MySqlConnection object has not been updated and it doesn't care about new connection state.
What's the best way to handle connection lost exception?
How can I refresh MySqlConnection when connection is restored?
Note: that I can't instantiate a new MySqlConnection object for each new query, because the program I'm trying to change has a Singleton of type MySqlConnection which is initialized only once. I know that's a bad design but I don't want to change this design now. I just want to catch connection lost exception and try to refresh MySqlConnection to continue to work correctly.
If your MySqlConnection instance loses its connection to your MySQL server, you cannot expect that instances's connection to be restored automatically or otherwise.
You need to try to reconnect with a new instance of MySqlConnection. The one that has lost the connection is now in a terminal state and cannot be reused.
To do this, I suppose you could do something like this
...
catch (MySqlException ex)
{
if (/*ex is a connection drop */) {
Connection?.Dispose();
Connection = new MySqlConnection(...);
Connection.ConnectionString = /* your connection string */;
Connection.Open();
}
else {
throw;
}
}
You are correct that your design has a flaw. Whether or not your flaw is fatal is hard to tell without testing.
These Connection instances are not thread safe or in any way reentrant. If you use one in a timer handler or thread, you may only use it in that context. Otherwise, if it's already in use when your timer or thread is invoked, things will get dicey. If you're lucky you'll get an exception. If you're less lucky your MySQL server will receive gibberish from your client and detect it. If you're even less lucky your data will get scrambled up.
ADO.NET and the MySqlConnection object implement connection pooling. This matters because it makes opening connections, using them, and then closing them, a lot cheaper than you might have expected.
Sometimes MySQL drops connections that your programs have held open for long periods of time. This post may help if that is your problem.
How can I change the default Mysql connection timeout when connecting through python?
When you lost your connection by networks problems, the connection object does not change Status property so evaluate it before executting commands doesn't work.
However, the database property (connection.database) goes to empty string so you can evaluate it so can close the connection an restores it:
oConn is an instance of MySQLConnection (it works on odbcconnection)
[VB.NET]
If Not IsNothing(oConn) Then
If (oConn.Database.Equals(String.Empty)) Then oConn.Close()
End If
[C#]
If (Not IsNothing(oConn)){
If (oConn.Database.Equals(String.Empty)) oConn.Close();
}
I am creating a console application and keep the SQL operations in a separate class file. When I execute the application it raises an exception:
ExecuteNonQuery requires an open and available Connection. The connection's current state is closed.
However, in SQL class file's constructor I wrote the code for SqlConnection.Open().
Code for Main application:
using SQL;
class MyClass
{
SQL.executeSQL runSQL=new SQL.executeSQL();
static void Main(string[] args)
{
CheckCounts();
}
public void CheckCounts()
{
string sql="select count(*) from table_name";
runSQL.executeQuery(sql);
}
}
Code for SQL class file:
public class executeSQL
{
SqlConnection con=new SqlConnection(ConfigurationManager.ConnectionString["dbConnection"].ToString());
public executeSQL()
{
if(con.State!=ConnectionState.Open)
{
con.Open();
}
}
public void executeQuery(string sql)
{
SqlCommand cmd=new SqlCommand(sql,con);
cmd.ExecuteNonQuery();
}
}
When I execute the application for the first time for that day it raises the exceptions as follows
ExecuteNonQuery requires an open and available Connection. The connection's current state is closed.
But for next time it runs properly without any exception for the whole day.
Again if I run the application for the next day it raises the same exception and consecutive successful execution.
What should I do to run the application successfully for the first time of the day?
I created a batch file for the application and I scheduled the task using the Task Scheduler with the batch file. If I run the application manually I do not get the error. Using the Task Scheduler, I am getting the error.
This can happen for a few reasons.
Chances are, you're connecting to a remote SQL server.
If this is the case, then there are a variety of reasons why the connection may be closed on you by the time you make your call.
Things like network dropouts etc can cause the connection between your app and SQL to break and require reconnection.
It can also be caused by the agreed timeout between your app and SQL being exceed, however I doubt this is the case given your code sample.
I have personally experienced issues where I have made what appears to be a valid connection to the server, and then on the very next line attempt to make a SQL query, only to receive the same error as yourself.
In those instances, I've added a check before making my SQL query around the SQLConnection.State property, which can return any of the following:
Broken
The connection to the data source is broken. This can occur only after
the connection has been opened. A connection in this state may be
closed and then re-opened. (This value is reserved for future versions
of the product.)
Closed
The connection is closed.
Connecting
The connection object is connecting to the data source.
Executing
The connection object is executing a command. (This value is reserved
for future versions of the product.)
Fetching
The connection object is retrieving data. (This value is reserved for
future versions of the product.)
Open
The connection is open.
If the SqlConnection.State is Open, then I run my SQL query as intended.
If it's Closed, I attempt to re-open it then re-call my SQL query.
If it's Connecting, I pause for a specific amount of time, then retest.
Depending on how you want your app to perform, I would look at either moving the creation of the SQLConnection into the method that makes the SQL call, as follows:
public class executeSQL
{
private void CheckSQLConnection(SqlConnection con)
{
...
}
public void executeQuery(string sql)
{
using (var con = new SqlConnection(ConfigurationManager.ConnectionStrings["dbConnection"].ToString()))
{
con.Open();
using (var cmd = new SqlCommand(sql, con))
{
CheckSQLConnection();
cmd.ExecuteNonQuery();
}
con.Close();
}
}
}
As in my example, it's also worth implementing using statements around your SQLConnection and SQLCommand objects to ensure they're properly disposed of and don't hang around, potentially holding onto SQL connections you might need.
Hope this helps!
I am trying to configure connection pooling for my .NET application using ODP.NET version 2.111.6.20. The database is Oracle 11.1.
I am using the following connection string in my .NET 2.0 application:
Data Source=prod; User Id=FAKE_USER; Password=FAKE_PASS; Pooling=true; Min Pool Size=2; Max Pool Size=5; Connection Timeout=30;"
According to the documentation the connection pool should initialize with 2 connections and and increment up to 5 connections as needed. It should never get higher than 5 connections.
What I am seeing is the the connections are growing 2 at a time and growing up to 10 connections. I am monitoring the connections in the Oracle database by querying the v$session table so I know the connections are from that specific application originating from my application.
If anyone can help me identify what might be happening in the connection pool inside this application that might be allowing for more than the Max number of connections I would appreciate it.
Sample C# Code
Here is a sample of the code making the calls to the database:
const string connectionString = "Data Source=prod; User Id=FAKE_USER; Password=FAKE_PASS; Pooling=true; Min Pool Size=5; Max Pool Size=5; Connection Timeout=30;";
using (OracleConnection connection = new OracleConnection(connectionString)) {
connection.Open();
using (OracleCommand command = new OracleCommand("ALTER SESSION SET TIME_ZONE='UTC'", connection)) {
command.ExecuteScalar();
}
using (OracleTransaction transaction = connection.BeginTransaction()) {
const string procSql = #"BEGIN P_SERVICES.UPDATE_VERSION(:id, :version, :installDate); END;";
using (OracleCommand command = new OracleCommand(procSql, connection)) {
command.Parameters.Add(new OracleParameter("id", OracleDbType.Varchar2) { Value = id });
command.Parameters.Add(new OracleParameter("version", OracleDbType.Varchar2) { Value = version });
command.Parameters.Add(new OracleParameter("installDate", OracleDbType.TimeStamp) { Value = dateUpdated });
try {
command.ExecuteNonQuery();
} catch (OracleException oe) {
if (Log.IsErrorEnabled) {
Log.ErrorFormat("Update Error: {0}", oe.Message);
}
throw;
}
transaction.Commit();
}
}
}
I have found the reason that the Maximum connections seen in the database is increasing past the number allowed in the connection pool settings in the connection string.
The Application Pool in IIS was configured to have "Maximum number of worker processes" set different than the default of 1. What I have found is that the number of connections seen in the database can grow up to the Max Pool Size * Number of Worker Processes.
So if I have Max Pool Size of 5 and 5 Worker Processes then the total number of connections allowed is 25. So it seems that each Worker Process has it's own instance of a connection pool that is not shared across other worker processes.
You can use this query to monitor your connection counts & statuses. Using this query, I was able to confirm that the connection string settings are working, explanation below.
select COUNT(*) AS Connections
,s.username
,s.status
,s.module
,s.osuser
from V$process p
join V$session s on s.paddr = p.addr
where NOT s.UserName IS NULL
group by s.username
,s.status
,s.module
,s.osuser
I ran this with 2 pages that did a bunch of database retrievals. Here are my differing results:
Max Pool Size=5
I saw fluctuations in the count under the empty module with same username as the webserver. I'm not sure why they showed up under that bucket as well as the webserver.
Max Pool Size=1
When I restricted the pool size, I only ever saw 1 connection for the empty module, and 1 connection for the webserver, but then connections popped up under DBMS_SCHEDULER, which indicates to me that the rest of the retreivals were pending?
I think this proves that the Max Pool Size is working, but I'm not certain.
According to Tom kyte:
A connection is a physical circuit between you and the database. A connection might be
one of many types -- most popular begin DEDICATED server and SHARED server.
Zero, one or more sessions may be established over a given connection to the database
A process will be used by a session to execute statements. Sometimes
there is a one to one relationship between CONNECTION->SESSION->PROCESS (eg: a normal
dedicated server connection). Sometimes there is a one to many from connection to
sessions. A process does
not have to be dedicated to a specific connection or session however, for example when
using shared server (MTS), your SESSION will grab a process from a pool of processes in
order to execute a statement. When the call is over, that process is released back to
the pool of processes.
So running
select username from v$session where username is not null
will show current seesions (not connections)
To see the connections you may use
select username, program from v$process;
A useful book about JDBC and Session VS Connection could be found here
If you absolutely have to fix this, and are willing to get down & dirty with performance counters, this blog post might be of help. At the very least it might help narrow down a discrepency between how many connections Oracle is reporting vs. how many pooled & non-pooled connections .NET claims to have.
http://blog.ilab8.com/2011/09/02/odp-net-pooling-and-connection-request-timed-out/
These counters seem like they would be particularly useful:
NumberOfActiveConnectionPools
NumberOfActiveConnections
NumberOfFreeConnections
NumberOfInactiveConnectionPools
NumberOfNonPooledConnections
NumberOfPooledConnections
NumberOfReclaimedConnections
NumberOfStasisConnections
I know this is one of the most popular questions on SO: The Famous "SQL error 26" the difference is that my C# application establishes a connection successfully the first time and then refuses to establish a connection the second time. I restart my computer and the application establishes a connection the first time and then requires a restart again.
The fact that I can establish a connection the first time makes me feel confident that: My server name is correct, My instance name is correct, the username and password combination I use are correct, the server machine is on, the SQL Browser service on the server is running, and I can get through the firewall.
I have a bunch of methods that all look very similar to the following:
private static string connection_string = #"Server=my_server\MS_SQL;User Id=user1;Password=password1"
public static List<string> GetListOfExistingItems(int item_id)
{
List<string> list_items = new List<string>();
try
{
using (SqlConnection sql_conn = new SqlConnection(connection_string))
{
sql_conn.Open();
SqlCommand sql_comm = new SqlCommand("SELECT Name FROM dbo.table1 WHERE ID=" + item_id,
sql_conn);
using (SqlDataReader sql_reader = sql_comm.ExecuteReader())
{
while (sql_reader.Read())
{
list_items.Add(sql_reader["Name"].ToString());
}
sql_reader.Close();
}
}
}
catch (Exception excp)
{
throw new Exception(excp.Message);
}
return list_items;
}
A few interesting facts:
A restart fixes the problem (once).
A log out and log in does NOT fix the problem.
If I do not close the application but instead run the query multiple times, the error does not show up.
I cannot connect to my database from SQL Server Management Studio after I run my application and close it once.
When I restart my computer I see the "Waiting for background programs to close" without any programs being listed. I wait for some time (maybe 10-20 seconds) and the message goes away and the computer eventually restarts.
Thanks in advance.
Somewhere you're not closing your connection.
My next troubleshooting step would be to string-search your code (all of it) for sql_conn.Open();, and find the one(s) not in a using block, or otherwise not getting explicitly closed.
I´m using Transaction Binding=Explicit Unbind in the connection string as recommended here since I´m also using TransactionScope with timeout. The problem is that the connections does not seem to close after being disposed and eventually there are no more connections available in the connection pool. I got the same result when I modified the TransactionTimeoutIssueDemo (see the link) and ran TransactionScopeTest() (with the explicit unbind connection string) enough times in a loop to use up all available connections in the connection pool. Default value for connections in the pool is 100 but this can be changed by using the setting Max Pool Size =10 for instance. It seems that the connections will not be released when using explicit unbind even though both the SqlConnection and the TransactionScope are used with the using clause. Anyone know how to handle this?
The connections only seem to stay in the pool and not being reused in case you get an exception, just like the example. If you increase the timeout the connection will be reused.
A workaround to this problem is to clear the connection pool in case you get an exception like this:
using (SqlConnection con = new SqlConnection(connectionString))
{
con.Open();
try
{
Console.WriteLine("Server is {0}", con.ServerVersion);
Console.WriteLine("Clr is {0}", Environment.Version);
for (int i = 0; i < 5; i++)
{
using (SqlCommand cmd = con.CreateCommand())
{
cmd.CommandText = "insert into TXTEST values ( " + i + " )";
cmd.ExecuteNonQuery();
}
Console.WriteLine("Row inserted");
}
Thread.Sleep(TimeSpan.FromSeconds(1));
}
catch
{
SqlConnection.ClearPool(con);
throw;
}
}
In most cases the transaction will complete within the timeout and everything will be fine and dandy. When the transaction actually do timeout you clear the pool in order to clean up the dirty connections that won't get reused. This will of course affect other connections in the pool that isn't affected by this problem.
This is a ugly workaround but it seems to work.
For what it's worth, this issue was fixed in .Net 4.0.