Develop Reference

Why can I not get the names of my tables using System.Data.SqlClient?

I did not create and do not own this database. My job is usually just to put data in it, so I don't have to think too much about the DBA side. I am messing with a side project and can not figure out why I can't list all the table names. I have found several methods to gather this information, but none work for me in a c# context. I get 0 rows back every time. They do work when executed as queries in azure data studio. I get 99 rows back (the correct number of tables) no matter which method I use to enumerate the tables. My connection string specifies the database to use within the server.
The query named workingQuery returns the expected information (1 row with 15 fields). So I know my connection string is correct and my code is functional.
I have a test function that I am trying many queries on.
using (SqlConnection conn = new SqlConnection(connectionString))
{
conn.Open();
List<string> fields = new List<string>();
string workingQuery = "SELECT TOP (1) * FROM myRealTable";
string failingTablesQuery1 = "SELECT name FROM sysobjects WHERE xtype = 'U'";
string failingTablesQuery2 = "SELECT name AS table_name FROM sys.tables GO";
string failingTablesQuery3 = "SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_TYPE = 'BASE TABLE'";
using (var command = conn.CreateCommand())
{
command.CommandText = workingQuery;
SqlDataReader reader = command.ExecuteReader();
while (reader.Read())
for (int i = 0; i < reader.FieldCount; i++)
{
fields.Add(reader[i].ToString());
}
conn.Close();
return fields;
}
}
In addition to the raw sql queries I have also tried SqlConnection.GetSchema("Tables"). It also returns no rows.
I tried prepending USE myDatabaseName; to the front of the queries to no effect.
I can only assume this is an issue with the way the database is set up. Can anyone help me track down what is causing this issue?
SOLUTION:
My connection string had credentials that did not have the necessary permissions for this operation.

In SQL Server access to metadata is controlled by the VIEW DEFINITION permission. You might have permission to SELECT from a table, but not to view its definition or discover it in the catalog. By default any permission on an object implies VIEW DEFINITION, but it can be DENY'd to a user. See Metadata Visibility Configuration.
So check that you're using the same identity in both places and that that identity has VIEW DEFINITION permissions on the tables.

Get output 'inserted' on update with Entity Framework

SQL Server provides output for inserted and updated record with the 'inserted' keyword.
I have a table representing a processing queue. I use the following query to lock a record and get the ID of the locked record:
UPDATE TOP (1) GlobalTrans
SET LockDateTime = GETUTCDATE()
OUTPUT inserted.ID
WHERE LockDateTime IS NULL
This will output a column named ID with all the updated record IDs (a single ID in my case). How can I translate this into EF in C# to execute the update and get the ID back?

Entity Framework has no way of doing that.
You could do it the ORM way, by selecting all the records, setting their LockDateTime and writing them back. That probably is not safe for what you want to do because by default it's not one single transaction.
You can span your own transactions and use RepeatableRead as isolation level. That should work. Depending on what your database does in the background, it might be overkill though.
You could write the SQL by hand. That defeats the purpose of entity framework, but it should be just as safe as it was before as far as the locking mechanism is concerned.
You could also put it into a stored procedure and call that. It's a little bit better than the above version because at least somebody will compile it and check that the table and column names are correct.

Simple Example #1 to get a data table:
I did this directly against the connection:
Changed the command.ExecuteNonQuery() to command.ExecuteReader()
var connection = DbContext().Database.Connection as SqlConnection;
using (var command = connection.CreateCommand())
{
command.CommandText = sql;
command.CommandTimeout = 120;
command.Parameters.Add(param);
using (var reader = command.ExecuteReader())
{
var resultTable = new DataTable();
resultTable.Load(reader);
return resultTable;
}
}
FYI, If you don't have an OUTPUT clause in your SQL, it will return an empty data table.
Example #2 to return entities:
This is a bit more complicated but does work.
using a SQL statement with a OUTPUT inserted.*
var className = typeof(T).Name;
var container = ObjContext().MetadataWorkspace.GetEntityContainer(UnitOfWork.ObjContext().DefaultContainerName, DataSpace.CSpace);
var setName = (from meta in container.BaseEntitySets where meta.ElementType.Name == className select meta.Name).First();
var results = ObjContext().ExecuteStoreQuery<T>(sql, setName, trackingEnabled ? MergeOption.AppendOnly : MergeOption.NoTracking).ToList();
T being the entity being worked on

Best performance in reading million records of data

I have a database with a large number of data (millions of rows), and also is updating during the day with large number of data, I have a back up of this database for reporting, so getting report of data does not affect on the performance of main database.
For syncing back up database with main database, I wrote a windows service which queries the main database and inserts new data into backup database... every time the query gets 5000 rows from the main database...
EDIT:
the query is like below:
const string cmdStr = "SELECT * FROM [RLCConvertor].[dbo].[RLCDiffHeader] WHERE ID >= #Start and ID <= #End";
Here is the code:
using (var conn = new SqlConnection(_connectionString))
{
conn.Open();
var cmd = new SqlCommand(cmdStr, conn);
cmd.Parameters.AddWithValue("#Start", start);
cmd.Parameters.AddWithValue("#End", end);
SqlDataReader reader = cmd.ExecuteReader(CommandBehavior.SequentialAccess);
while (reader.Read())
{
var rldDiffId = Convert.ToInt32(reader["ID"].ToString());
var rlcDifHeader = new RLCDiffHeader
{
Tech_head_Type = long.Parse(reader["Tech_head_Type"].ToString()),
ItemCode = long.Parse(reader["ItemCode"].ToString()),
SessionNumber = long.Parse(reader["SessionNumber"].ToString()),
MarketFeedCode = reader["MarketFeedCode"].ToString(),
MarketPlaceCode = reader["MarketPlaceCode"].ToString(),
FinancialMarketCode = reader["FinancialMarketCode"].ToString(),
CIDGrc = reader["CIDGrc"].ToString(),
InstrumentID = reader["InstrumentID"].ToString(),
CValMNE = reader["CValMNE"].ToString(),
DEven = reader["DEven"].ToString(),
HEven = reader["HEven"].ToString(),
MessageCodeType = reader["MessageCodeType"].ToString(),
SEQbyINSTandType = reader["SEQbyINSTandType"].ToString()
};
newRLCDiffHeaders.Add(rldDiffId, rlcDifHeader);
}
conn.Close();
}
but when I started the service... the performance of main database got worse... is the code not efficient? Is there any better way? Because I searched and found that dataReader is the best for this case... or should I use DataTable and SqlDataAdapter?

You cannot treat this an a correct answer or solution for your problem.
Since the comment goes big, I am providing a solution to you.
Can you try using the concept of Ad hoc queries
Using this you can query another database using the following way
SELECT a.*
FROM OPENROWSET('SQLNCLI', 'Server=Seattle1;Trusted_Connection=yes;',
'SELECT GroupName, Name, DepartmentID
FROM AdventureWorks2012.HumanResources.Department
ORDER BY GroupName, Name') AS a;
Read more
http://technet.microsoft.com/en-us/library/ms187569.aspx
http://technet.microsoft.com/en-us/library/ms190312.aspx
Since you are using a service, the service account surely have access to read the main db and insert to report db. I will suggest you to have a SP in your report DB , that can access the main DB using OpenRowSet and insert to it.
Query will be similar like this.
Insert into tbl
SELECT a.*
FROM OPENROWSET('SQLNCLI', 'Server=Seattle1;Trusted_Connection=yes;',
'SELECT GroupName, Name, DepartmentID
FROM AdventureWorks2012.HumanResources.Department
ORDER BY GroupName, Name') AS a;
Form the service, you need to invoke the SP.
We had a similar issue and this was done by openrowset and I don't know how much performance impact this can provide. But I suggest you to do a POC and just analyze it.
Once again please consider this as a suggestion.

Why code is maxing out the CPU while querying the database?

My C# code below checks a SQL database to see if a record matches a ClientID and a User Name. If more than 15 or more matching records are found that match, the CPU on my Windows 2008 server peaks at about 78% while the 15 records are found while the below C# code executes. The SQL Server 2008 database and software is located on another server so the problem is not with SQL Server spiking the CPU. The problem is with my C# software that is executing the code below. I can see my software executable that contains the C# code below spike to 78% while the database query is executed and the records are found.
Can someone please tell me if there is something wrong with my code that is causing the CPU to spike when 15 or more matching records are found? Can you also please tell/show me how to optimize my code?
Update: If it finds 10 records, the CPU only spikes at 2-3 percent. It is only when it finds 15 or more records does the CPU spike at 78% for two to three seconds.
//ClientID[0] will contain a ClientID of 10 characters
//output[0] will contain a User Name
char[] trimChars = { ' ' };
using (var connection = new SqlConnection(string.Format(GlobalClass.SQLConnectionString, "History")))
{
connection.Open();
using (var command = new SqlCommand())
{
command.CommandText = string.Format(#"SELECT Count(*) FROM Filelist WHERE [ToAccountName] = '" + output[0] + #"'");
command.Connection = connection;
var rows = (int) command.ExecuteScalar();
if (rows >= 0)
{
command.CommandText = string.Format(#"SELECT * FROM Filelist WHERE [ToAccountName] = '" + output[0] + #"'");
using (SqlDataReader reader = command.ExecuteReader())
{
if (reader.HasRows)
{
while (reader.Read())
{
//Make sure ClientID does NOT exist in the ClientID field
if (reader["ClientID"].ToString().TrimEnd(trimChars).IndexOf(ClientID[0]) !=
-1)
{
//If we are here, then do something
}
}
}
reader.Close();
reader.Dispose();
}
}
// Close the connection
if (connection != null)
{
connection.Close();
}
}
}

You can decrease the number of database access from 2 to 1 if will remove first query, it is not necessary.
using (SqlConnection connection = new SqlConnection(connectionString))
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = "SELECT ClientID FROM dbo.Filelist WHERE ToAccountName = #param"; // note single column in select clause
command.Parameters.AddWithValue("#param", output[0]); // note parameterized query
connection.Open();
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read()) // reader.HasRow is doubtfully necessary
{
// logic goes here
// but it's better to perform it on data layer too
// or return all clients first, then perform client-side logic
yield return reader.GetString(0);
}
} // note that using block calls Dispose()/Close() automatically
}

Change this:
SELECT * FROM Filelist
To this:
SELECT ClientID FROM Filelist
And check for performance.
I suspect there is a blob field on your select.
Also select * is not recommended, write your exact interested fields in your query.

Nothing looks obviously CPU intensive, but one problem does stand out.
You are running a query to count how many records there are
"SELECT Count(*) FROM Filelist WHERE [ToAccountName] = '" + output[0] + #"'"
Then, if more than 0 is returned, you are running another query to get the data.
"SELECT * FROM Filelist WHERE [ToAccountName] = '" + output[0] + #"'"
This is redundant. Get rid of the first query, and just use the second one, checking to see if the reader has data. You can also get rid of the HasRows call and just do
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
}
}

Please consider what already said about parametrized queries.
Beside that, I think that the only big issue could arise in the following block:
while (reader.Read())
{
//Make sure ClientID does NOT exist in the ClientID field
if (reader["ClientID"].ToString().TrimEnd(trimChars).IndexOf(ClientID[0]) != -1)
{
//If we are here, then do something
}
}
So try to just cache your reader.Read() data in some local variable, releasing the SQL resources asap, then you can work on the data you just retrieved. Eg:
List<string> myRows = new List<string>();
while (reader.Read())
{
myRows.Add(reader["ClientID"].ToString();
}
/// quit the using clause
/// now elaborate what you got in myRows

There is nothing in the code to indicate a performance problem.
What does SQL Profiler show?
(Both in terms of query plan, and server resources used.)
Edit: To make this clearer: you have one measurement that might indicate an issue. You now need to measure more deeply to understand if it really is a problem, only you can do this (no one else has access to the hardware).

I strongly recommend that you get a copy of dotTrace from JetBrains.
At the very least, profiling the client code will help you identify/eliminate the source of the CPU spike.

I recommend using parameters as suggested, however, I have seen performance problems where the type of the string column does not match the C# string. In these cases, I suggest specifying the type explicitly.
Like this:
command.CommandText = "SELECT ClientID FROM dbo.Filelist WHERE ToAccountName = #accountName";
command.Parameters.Add("#accountName", SqlDbType.NVarChar, 16, output[0]);
Or this:
SqlParameter param = command.Parameters.Add(
"#accountName", SqlDbType.NVarChar);
param.Size = 16; //optional
param.Value = output[0];

Getting the Last Insert ID with SQLite.NET in C#

I have a simple problem with a not so simple solution... I am currently inserting some data into a database like this:
kompenzacijeDataSet.KompenzacijeRow kompenzacija = kompenzacijeDataSet.Kompenzacije.NewKompenzacijeRow();
kompenzacija.Datum = DateTime.Now;
kompenzacija.PodjetjeID = stranka.id;
kompenzacija.Znesek = Decimal.Parse(tbZnesek.Text);
kompenzacijeDataSet.Kompenzacije.Rows.Add(kompenzacija);
kompenzacijeDataSetTableAdapters.KompenzacijeTableAdapter kompTA = new kompenzacijeDataSetTableAdapters.KompenzacijeTableAdapter();
kompTA.Update(this.kompenzacijeDataSet.Kompenzacije);
this.currentKompenzacijaID = LastInsertID(kompTA.Connection);
The last line is important. Why do I supply a connection? Well there is a SQLite function called last_insert_rowid() that you can call and get the last insert ID. Problem is it is bound to a connection and .NET seems to be reopening and closing connections for every dataset operation. I thought getting the connection from a table adapter would change things. But it doesn't.
Would anyone know how to solve this? Maybe where to get a constant connection from? Or maybe something more elegant?
Thank you.
EDIT:
This is also a problem with transactions, I would need the same connection if I would want to use transactions, so that is also a problem...

Using C# (.net 4.0) with SQLite, the SQLiteConnection class has a property LastInsertRowId that equals the Primary Integer Key of the most recently inserted (or updated) element.
The rowID is returned if the table doesn't have a primary integer key (in this case the rowID is column is automatically created).
See https://www.sqlite.org/c3ref/last_insert_rowid.html for more.
As for wrapping multiple commands in a single transaction, any commands entered after the transaction begins and before it is committed are part of one transaction.
long rowID;
using (SQLiteConnection con = new SQLiteConnection([datasource])
{
SQLiteTransaction transaction = null;
transaction = con.BeginTransaction();
... [execute insert statement]
rowID = con.LastInsertRowId;
transaction.Commit()
}

select last_insert_rowid();
And you will need to execute it as a scalar query.
string sql = #"select last_insert_rowid()";
long lastId = (long)command.ExecuteScalar(sql); // Need to type-cast since `ExecuteScalar` returns an object.

last_insert_rowid() is part of the solution. It returns a row number, not the actual ID.
cmd = CNN.CreateCommand();
cmd.CommandText = "SELECT last_insert_rowid()";
object i = cmd.ExecuteScalar();
cmd.CommandText = "SELECT " + ID_Name + " FROM " + TableName + " WHERE rowid=" + i.ToString();
i = cmd.ExecuteScalar();

I'm using Microsoft.Data.Sqlite package and I do not see a LastInsertRowId property. But you don't have to create a second trip to database to get the last id. Instead, combine both sql statements into a single string.
string sql = #"
insert into MyTable values (null, #name);
select last_insert_rowid();";
using (var cmd = conn.CreateCommand()) {
cmd.CommandText = sql;
cmd.Parameters.Add("#name", SqliteType.Text).Value = "John";
int lastId = Convert.ToInt32(cmd.ExecuteScalar());
}

There seems to be answers to both Microsoft's reference and SQLite's reference and that is the reason some people are getting LastInsertRowId property to work and others aren't.
Personally I don't use an PK as it's just an alias for the rowid column. Using the rowid is around twice as fast as one that you create. If I have a TEXT column for a PK I still use rowid and just make the text column unique. (for SQLite 3 only. You need your own for v1 & v2 as vacuum will alter rowid numbers)
That said, the way to get the information from a record in the last insert is the code below. Since the function does a left join to itself I LIMIT it to 1 just for speed, even if you don't there will only be 1 record from the main SELECT statement.
SELECT my_primary_key_column FROM my_table
WHERE rowid in (SELECT last_insert_rowid() LIMIT 1);

The SQLiteConnection object has a property for that, so there is not need for additional query.
After INSERT you just my use LastInsertRowId property of your SQLiteConnection object that was used for INSERT command.
Type of LastInsertRowId property is Int64.
Off course, as you already now, for auto increment to work the primary key on table must be set to be AUTOINCREMENT field, which is another topic.

database = new SQLiteConnection(databasePath);
public int GetLastInsertId()
{
return (int)SQLite3.LastInsertRowid(database.Handle);
}

# How about just running 2x SQL statements together using Execute Scalar?
# Person is a object that has an Id and Name property
var connString = LoadConnectionString(); // get connection string
using (var conn = new SQLiteConnection(connString)) // connect to sqlite
{
// insert new record and get Id of inserted record
var sql = #"INSERT INTO People (Name) VALUES (#Name);
SELECT Id FROM People
ORDER BY Id DESC";
var lastId = conn.ExecuteScalar(sql, person);
}

In EF Core 5 you can get ID in the object itself without using any "last inserted".
For example:
var r = new SomeData() { Name = "New Row", ...};
dbContext.Add(r);
dbContext.SaveChanges();
Console.WriteLine(r.ID);
you would get new ID without thinking of using correct connection or thread-safety etc.

If you're using the Microsoft.Data.Sqlite package, it doesn't include a LastInsertRowId property in the SqliteConnection class, but you can still call the last_insert_rowid function by using the underlying SQLitePCL library. Here's an extension method:
using Microsoft.Data.Sqlite;
using SQLitePCL;
public static long GetLastInsertRowId(this SqliteConnection connection)
{
var handle = connection.Handle ?? throw new NullReferenceException("The connection is not open.");
return raw.sqlite3_last_insert_rowid(handle);
}