I am running into issues working with a very large table from C# .Net 4.0.
For reference the table has ~120 Million rows.
I can't do even a simple query like
SELECT TOP(50) *
FROM TableName
WHERE CreatedOn < '2015-06-01';
From code it will timeout (Default setting - 15 seconds I believe), but in SSMS it is instant.
There is an index on the column in the WHERE clause. I have also tried explicitly casting the string to a DateTime, and using a DateTime parameter instead of a literal value.
I tried a different query that filters by the PK (bigint, identity, clustered index) If I do something like "Where TableRowID = 1" it works fine from code, but if I try to use "<" or "<=" instead it will timeout (returns instantly in SSMS), regardless of how many rows are turned.
The execution plans are very simple and are exactly the same.
I have tried changing ARITHABORT but that has had no effect.
I tried having the Application connect with my own account (SSPI) instead of its own credentials with no effect.
I have been researching this issue for a few days, but everything I have found blames different execution plans.
Does anyone have any idea what could be causing this issue?
The .Net code looks like this:
private DataSet ExecuteQuery(string query, string db, List<SqlParameter> parms = null)
{
string connectionString = ConfigurationManager.ConnectionStrings[db].ToString();
SqlConnection con = new SqlConnection(connectionString.Trim());
SqlDataAdapter sqlDataAdapter = new SqlDataAdapter();
try
{
con.Open();
DataSet ds = new DataSet();
sqlDataAdapter.SelectCommand = new SqlCommand(query, con);
sqlDataAdapter.SelectCommand.CommandType = CommandType.Text;
if (parms != null)
{
foreach (SqlParameter p in parms)
{
sqlDataAdapter.SelectCommand.Parameters.Add(p);
}
}
sqlDataAdapter.Fill(ds);
if (ds.Tables.Count > 0 && ds.Tables[0].Rows.Count > 0)
{
return ds;
}
return null;
}
finally
{
if (sqlDataAdapter != null)
sqlDataAdapter.Dispose();
if (con != null)
con.Dispose();
}
}
The error message I get in .Net is the standard timeout message:
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Here are my experience when dealing the issues.
Very C# sql code passed to sql server is
SELECT TOP(50) *
FROM TableName
WHERE CreatedOn < '2015-06-01'
Make sure the criteria. If it takes "instant time" to retrieve records on SSMS, the sql statement and db optimization is ok.
As other people have pointed out, you should post your C# codes and see what happens. There could be other issues. Could that be network? Could that be web.config? Do you call directly from C# code? Do you call via web service?
when you said time out? Does it time out at the time you execute the query. There is very little information you provide. Do you use third party routines (such as written by vendor or your company) to execute queries? If it possible, put the break point at the code that execute sql statement. What I mean is dig all the way to native codes, and put the break codes.
120 million records. Looks like the database has be optimized if it runs very fast on SSMS. I would take look outside SQL server.
good luck
My first step there would be to look at what your code is sending to the sql server. I'd begin by running the sql profiler. if you're not familiar with it. Here is a link on how to use it.
https://www.mssqltips.com/sqlservertip/2040/use-sql-server-profiler-to-trace-database-calls-from-third-party-applications/
After this you may want to look into network traffic times between the 2 servers.
Then look at IIS and see how it's setup. Could be a setting is wrong.
Check the error logs and see if you have any errors as well.
When you execute code in SSMS, the system is setting some default values on your connection. Specifically, look at Tools --> Options --> Query Execution --> Sql Server --> Advanced (and also ANSI). These statements are executed on your behalf when you open a new query window.
When you create your connection object in C#, are you setting these same options? If you don't explicitly set them here, you are taking the default values as defined in the actual SQL Server instance. In SSMS, you can get this by viewing the properties of the server instance and choosing the Connections page. This shows the default connection options.
You can also get this information without using the UI (you can do this in a separate application that uses the same connection string, for example). This article on MSSQLTips should guide you in the right direction.
Related
I have a strange issues reading data from an Eexcel file.
I made a query in SQL Server with OPENDATASOURCE:
SELECT COMPETENZA, [CODICE CLEINTE], [RAGIONE SOCIALE], MODELLO, VARIANTE,
TIPOLOGIA, ESSENZA, FINITURA, TAPPEZZERIA, COMMESSA, ROUND([% MOL SU RICAVI NETTI] * 100,2) as MARGINE
FROM OPENDATASOURCE('Microsoft.ACE.OLEDB.16.0',
'Data Source=K:\UTENTI\SAMUELE\temp id commesse\ID Commesse produzione.xls;Extended Properties=EXCEL 12.0')...['Consuntivi produzione$']
WHERE MODELLO IS NOT NULL AND [% MOL SU RICAVI NETTI] IS NOT NULL
If I execute this query in a stored procedure, it works fine.
But when I execute from C# by an HttpGet I get this issues:
System.Data.SqlClient.SqlException (0x80131904): Cannot initialize the data source object of OLE DB provider "Microsoft.ACE.OLEDB.16.0" for linked server "(null)".
This is my C# code:
[HttpGet]
public async Task<JsonResult> GetMarginiCommessaListJSON()
{
var yourdata = GetMarginiCommessaList();
return Json(new { data = yourdata });
}
public List<MarginiCommessa> GetMarginiCommessaList()
{
SqlDataReader reader;
SqlConnection sqlConn = new SqlConnection(_configuration["dbConnectionString"]);
string sqlCmdStr = "MARGINI_COMMESSE";
SqlCommand sqlCmd = new SqlCommand(sqlCmdStr, sqlConn);
sqlCmd.CommandType = CommandType.StoredProcedure;
List<MarginiCommessa> allestList = new List<MarginiCommessa>();
try
{
using (sqlConn)
{
sqlConn.Open();
reader = sqlCmd.ExecuteReader();
while (reader.Read())
{
allestList.Add(new MarginiCommessa
{
anno = reader.GetValue(0).ToString(),
codCli = reader.GetValue(1).ToString(),
nomCli = reader.GetValue(2).ToString(),
modello = reader.GetValue(3).ToString(),
variante = reader.GetValue(4).ToString(),
tipologia = reader.GetValue(5).ToString(),
essenza = reader.GetValue(6).ToString(),
finitura = reader.GetValue(7).ToString(),
tapezzeria = reader.GetValue(8).ToString(),
commessa_qta = reader.GetValue(9).ToString(),
margine = reader.GetValue(10).ToString()
});
}
reader.Close();
sqlConn.Close();
}
}
catch (Exception e)
{
System.IO.File.WriteAllText(GlobalVariables.errorFolderLocation + "GetMarginiCommessaListJSON.txt", e.ToString());
}
return allestList;
}
And this is my Stored procedure:
ALTER PROCEDURE [dbo].[MARGINI_COMMESSE]
AS
BEGIN
SET NOCOUNT ON;
SELECT COMPETENZA, [CODICE CLEINTE], [RAGIONE SOCIALE], MODELLO, VARIANTE,
TIPOLOGIA, ESSENZA, FINITURA, TAPPEZZERIA, COMMESSA, ROUND([% MOL SU RICAVI NETTI] * 100,2) as MARGINE
FROM OPENDATASOURCE('Microsoft.ACE.OLEDB.16.0',
'Data Source=K:\UTENTI\SAMUELE\temp id commesse/ID Commesse produzione.xls;Extended Properties=EXCEL 12.0')...['Consuntivi produzione$']
WHERE MODELLO IS NOT NULL AND [% MOL SU RICAVI NETTI] IS NOT NULL
END
I really don't understand why in SQL Server it works fine and in C# not.
How can I fix this?
OP #1:
I have a strange issues reading datas from an excel file. I made a query in mssql with OPENDATASOURCE: If i execute this query in a stored procedure, it works fine.
OP #2:
But when i execute from c# by an HttpGet i get this issues:
Firstly, that's an apples and oranges comparison. On one hand you are using MySQL and the other SQL Server. The latter case you also have a web app using ADO.NET in order to reach the stored proc.
Now I can't vouch for MySQL or whatever OPENDATASOURCE is doing in both occasions but one thing is certain - you appear to be using ACE OLEDB "Jet" in a server setting which may be the cause of your issues. Crashes aside, using ACE might have licensing issues. See below.
Microsoft (my emphasis):
Although such programmatic development can be implemented on a client system with relative ease, a number of complications can occur if Automation takes place from server-side code such as Microsoft Active Server Pages (ASP), ASP.NET, DCOM, or a Windows NT service.
NOTE: In this context, the Access Database Engine Redistributable and Access Runtime are considered Microsoft Office components.
...and:
Developers who try to use Office in a server-side solution need to be aware of five major areas in which Office behaves differently than anticipated because of the environment.
...and most importantly (my emphasis):
Besides the technical problems, you must also consider licensing issues. Current licensing guidelines prevent Office applications from being used on a server to service client requests, unless those clients themselves have licensed copies of Office. Using server-side Automation to provide Office functionality to unlicensed workstations is not covered by the End User License Agreement (EULA).
Conclusion
So even if you fix the technical issue, be mindful of that last pointer.
See also
Considerations for server-side Automation of Office, Microsoft
I am trying to setup my .NET 4.7.1 program that is connecting to a MySQL database 8.0 to use the minimum privileges to run.
The .NET program is using MySql.Data to make connection. The minimum right for a user to execute a stored procedure is typically only EXECUTE privilege. This works fine from MySQL workbench or command line.
Upon running the .NET program this does return the following exception:
System.Data.SqlTypes.SqlNullValueException: 'Data is Null. This method or property cannot be called on Null values.'
To make it easy, I have create a very small demo program to demonstrate the issue.
Setup of the database:
CREATE DATABASE Spike;
CREATE PROCEDURE TestAccess()
BEGIN
END;
CREATE USER Spike#localhost IDENTIFIED WITH mysql_native_password BY 'sample';
GRANT EXECUTE ON PROCEDURE `TestAccess` TO Spike#localhost;
Setup program code:
static void Main(string[] args)
{
using (MySqlConnection conn = new MySqlConnection("Server=localhost;Database=Spike;uid=Spike;pwd=sample"))
{
conn.Open();
Console.WriteLine("Connection open");
MySqlCommand cmd = new MySqlCommand();
cmd.Connection = conn;
cmd.CommandText = "TestAccess";
cmd.CommandType = System.Data.CommandType.StoredProcedure;
cmd.ExecuteNonQuery();
Console.WriteLine("Query executed");
}
Console.ReadKey();
}
The crash happens at the line cmd.ExecuteNonQuery();
The stack from the crash is interesting, since it seems to indicate that the information_schema is queried. When logging all statements I can see that the last statement before the exception is:
SELECT * FROM information_schema.routines WHERE 1=1 AND routine_schema LIKE 'Spike' AND routine_name LIKE 'TestAccess'
I cannot grant different rights on information_schema, but I could give more rights on the stored procedure to make more information visible in the routines table, this feels wrong however. Simple tests with granting CREATE and ALTER access also did not work.
Is there something else I can do, without granting too much privileges?
This appears to be a bug in Connector/NET, similar to bug 75301 but a little different. When it's trying to determine parameter metadata for the procedure, it first creates a MySqlSchemaCollection named Procedures with all metadata about the procedure. (This is the SELECT * FROM information_schema.routines WHERE 1=1 AND routine_schema LIKE 'Spike' AND routine_name LIKE 'TestAccess' query you see in your log.)
The Spike user account doesn't have permission to read the ROUTINE_DEFINITION column, so it is NULL. Connector/NET expects this field to be non-NULL and throws a SqlNullValueException exception trying to read it.
There are two workarounds:
1) The first, which you've discovered, is to set CheckParameters=False in your connection string. This will disable retrieval of stored procedure metadata (avoiding the crash), but may lead to harder-to-debug problems calling other stored procedures if you don't get the order and type of parameters exactly right. (Connector/NET can no longer map them for you using the metadata.)
2) Switch to a different ADO.NET MySQL library that doesn't have this bug: MySqlConnector on NuGet. It's highly compatible with Connector/NET, performs faster, and fixes a lot of known issues.
I found an answer with which I am quite pleased. It is changing the connection string by adding CheckParameters=false:
using (MySqlConnection conn = new MySqlConnection("Server=localhost;Database=Spike;uid=Spike;pwd=sample;CheckParameters=false"))
This disables parameter checking, and thereby information_schema queries.
I have a query to a DB2 database on an iSeries that I wish to run from a C# program. The query is:
SELECT C.CLINO, SUM(T.VALUE) AS VALUE
FROM GTDATA.CLIENT AS C
INNER JOIN TABLE(GETHOLDDTL(C.CLINO,20150630,'','Y','Y','','','','','','','','','','M')) AS T ON T.CLIENT = C.CLINO
GROUP BY C.CLINO
GETHOLDDTL is an SQL function supplied to us by a third party to allow us to retrieve data from their system in our own programs.
When I run the query in the iSeries interactive SQL environment, saving the output to a file to ensure that I'm getting a time for the execution over the whole dataset, it takes approximately 1.5 hours to run. This is fine, as there's a lot of data which is being calculated on the fly. The problem is that when I run my C# program, the process takes in excess of 24 hours (ie that's the point at which I gave up, when it hadn't got past this point the morning after I started execution).
The C# code I'm using to execute the query is:
var conn = new OdbcConnection();
conn.ConnectionString = #"FILEDSN=S:\qsys2.dsn;UID=XXXXXXX;PWD=XXXXXXXX";
conn.ConnectionTimeout = 0;
conn.Open();
var com = new OdbcCommand(selectCommand, conn);
var reader = com.ExecuteReader();
var dt = new DataTable();
dt.Load(reader);
conn.Close();
The delay is happening on the dt.Load() line - this is presumably where the query is actually executed?
What might account for the difference? Is there a better way to implement this? If necessary, I can produce the data using the interactive SQl environment, and change my C# code to use the file that produces, but I'm trying to get this to run without any user interaction so I'd like to avoid that if possible.
Thanks for any help.
I am developing a web site, it uses SQL Server 2008 R2 Express for its database. And in testing, there is a lot of data and images stored into this database.
According to wiki, the SQL Server Express edition has a 10 GB size limit. When I insert data and reach the limit, what exception will be thrown? Or, how do I detect the approaching limit problem by codes ?
I use EF 5 with code-first approach to insert large data set.
In tests I have seen that:
sp_spaceused
won't work as expected, it showed 12GB after deleting lots of records. And the other answers regarding query sys.databases were not clear enough to me.
Searching around I found a very good explanation regarding SQL Server 2012 Express Edition 10GB Size Limit on Ramons weblog [EDIT2018 updated link]
SELECT
[name] AS [Filename],
[size]/128.0 AS [Filesize],
CAST(FILEPROPERTY([name],'SpaceUsed') AS int)/128.0 AS [UsedSpaceInMB],
[size]/128.0 - CAST(FILEPROPERTY([name],'SpaceUsed') AS int)/128.0 AS [AvailableSpaceInMB],
[physical_name] AS [Path]
FROM sys.database_files
"... space includes the transaction log and it also includes all unused space within these files. .... SQL Server Express will start complaining when it cannot reserve any more space for the datafile."
So checking
CAST(FILEPROPERTY([name],'SpaceUsed') AS int)/128.0 AS [UsedSpaceInMB]
seems to be the best option.
In combination with EF in c# my request to the DB looks like
string sqlSelect = "SELECT CAST(FILEPROPERTY([name],'SpaceUsed') AS int)/128.0 AS [UsedSpaceInMB] FROM sys.database_files";
var dbResult = dbInstance.Database.SqlQuery<Decimal>(sqlSelect).FirstOrDefault();
double spaceUsedInGb = Convert.ToDouble(dbResult)/1024;
Execute this SQL command, and it will reveal the disk-space usage of current database.
sp_spaceused
It also can be used to query the space usage of specific table. This link provides useful information about this problem.
To check the database size query:
sys.databases
Just query this, perhaps with C# or if you use SSMS (sql server management studio) shell, you can schedule a job that emails you or whatever you want.
Example:
SQL Server 2008: How to query all databases sizes?
Edit: NOT sure if error is thrown, it should log to event log or a sql log...
Side note:
Developer version is only $50 and holds same as Datacenter which hold 524 PB
http://technet.microsoft.com/en-us/library/cc645993%28v=sql.105%29.aspx
To Check the Size of the Database Two Ways:
/* new school way - data plus log and run in the local db that you want to see
here you can see the log and the mdf file.
*/
SELECT size*8.0/1024.0 as size_in_gb, *
FROM sys.database_files
GO
/* old school way, run for all db size*/
sp_helpdb
FYI - the MDF and NDF files are the only ones that attribute to the file size exceeding 10GB.
I am using the following method to calculate database current size crucial for comparing with sql size limitations:
public static int GetDbSizeInMB([NotNull] string connectionString) {
using (SqlConnection sqlConnection = new SqlConnection(connectionString)) {
sqlConnection.Open();
using (var sqlCommand = new SqlCommand()) {
sqlCommand.CommandType = CommandType.Text;
sqlCommand.CommandText = #"
SELECT SUM(CAST(FILEPROPERTY([name],'SpaceUsed') AS int)/128.0) AS [UsedSpaceInMB]
FROM sys.database_files
WHERE type_desc like 'ROWS' or type_desc like 'FULLTEXT'
";
sqlCommand.Connection = sqlConnection;
return Convert.ToInt32(sqlCommand.ExecuteScalar());
}
}
)
I have a very odd issue. When I execute a specific database stored procedure from C# using SqlCommand.ExecuteNonQuery, my stored procedure is never executed.
Furthermore, SQL Profiler does not register the command at all. I do not receive a command timeout, and no exeception is thrown.
The weirdest thing is that this code has worked fine over 1,200,000 times, but for this one particular file I am inserting into the database, it just hangs forever.
When I kill the application, I receive this error in the event log of the database server: "A fatal error occurued while reading the input stream from the network. The session will be terminated (input error: 64, output error: 0). Which makes me think that the database server is receiving the command, though SQL Profiler says otherwise.
I know that the appropiate permissions are set, and that the connection string is right as this piece of code and stored procedure works fine with other files.
Below is the code that calls the stored procedure, it may be important to note that the file I am trying to insert is 33.5MB, but I have added more than 10,000 files larger than 500MB, so I do not think the size is the issue:
using (SqlConnection sqlconn = new SqlConnection(ConfigurationManager.ConnectionStrings["TheDatabase"].ConnectionString))
using (SqlCommand command = sqlconn.CreateCommand())
{
command.CommandText = "Add_File";
command.CommandType = CommandType.StoredProcedure;
command.CommandTimeout = 30 //should timeout in 30 seconds, but doesn't...
command.Parameters.AddWithValue("#ID", ID).SqlDbType = SqlDbType.BigInt;
command.Parameters.AddWithValue("#BinaryData", byteArr).SqlDbType = SqlDbType.VarBinary;
command.Parameters.AddWithValue("#FileName", fileName).SqlDbType = SqlDbType.VarChar;
sqlconn.Open();
command.ExecuteNonQuery();
}
There is no firewall between the server making the call and the database server, and the windows firewalls have been disabled to troubleshoot this issue.
I've seen this once before when uploading XML via a stored proc from one workstation only
We changed the network cable (which routed differently in our big building) and it worked.
Bizarre as this sounds, can you somehow re-mount the server or change cables or bypass a switch etc.