C# ADO.net query runs slow - c#

I have a query that .net builds and executes via ADO to Sql Server 2012. If ADO executes the query it times out waiting for the database to return but if I copy and past it into SQL Management studio and execute it against the same database it returns in less then a second.
I found that wrapping the query that ADO generates in ' storing it in a SQL variable and using SP_EXECUTESQL makes it run nice and fast. Why would there be a difference? I have not changed the query in any way.
Here is one of the queries it builds and executes (I left out all the code that dynamically builds the string that CommandText gets set to).
using(SqlConnection conn = ConnectionStringHelper.GetOpenConnection)
using(SqlCommand cmd = conn.CreateCommand)
{
cmd.CommandType = CommandType.Text;
cmd.Parameters.AddWithValue("agencyID", broadcastAgencyID);
cmd.Parameters.AddWithValue("tableName", mapping.TableName);
cmd.Parameters.AddWithValue("schemaName", mapping.SchemaName);
cmd.Parameters.AddWithValue("broadcastEntityMappingID", mapping.BroadcastEntityMappingID);
cmd.CommandText = #"SET NOCOUNT ON;
DECLARE #currentAgencyID NVARCHAR(MAX)
DECLARE #currentFacilityID NVARCHAR(MAX)
DECLARE #currentAgencyEntityBroadcastID INT
SELECT broadcastEntity.AgencyID, broadcastEntity.FacilityID
INTO #missingBroadcasts
FROM [Resource].[AgencyFacility] broadcastEntity
LEFT JOIN(
SELECT keys.AgencyID, keys.FacilityID
, record.BroadcastAgencyID
, record.AgencyEntityBroadcastID
FROM [Propagation].[AgencyEntityBroadcast] record
INNER JOIN (
SELECT AgencyEntityBroadcastID
,[AgencyID], [FacilityID]
FROM (
SELECT AgencyEntityBroadcastID
,ColumnName AS [PropagationColumnName]
,ColumnValue AS [PropagationColumnValue]
FROM Propagation.AgencyEntityBroadcastKeys
) Keys
PIVOT(MAX(PropagationColumnValue) FOR PropagationColumnName IN (
[AgencyID], [FacilityID]
)) pivoted
) keys ON keys.AgencyEntityBroadcastID = record.AgencyEntityBroadcastID
WHERE record.BroadcastAgencyID = #agencyID
AND record.BroadcastEntityMappingID = #broadcastEntityMappingID
) keys ON keys.BroadcastAgencyID = broadcastEntity.AgencyID
AND keys.AgencyID = broadcastEntity.AgencyID
AND keys.FacilityID = broadcastEntity.FacilityID
WHERE broadcastEntity.AgencyID = #agencyID
AND keys.AgencyEntityBroadcastID IS NULL
DECLARE entity_cursor CURSOR FOR
SELECT * FROM #missingBroadcasts
OPEN entity_cursor
FETCH NEXT FROM entity_cursor
INTO #currentAgencyID, #currentFacilityID
WHILE ##FETCH_STATUS = 0
BEGIN
INSERT INTO Propagation.AgencyEntityBroadcast(
BroadcastAgencyID,
BroadcastEntityMappingID,
BroadcastTypeID,
CreatedOn,
ModifiedOn
)
VALUES(
#agencyID,
#broadcastEntityMappingID,
1,
GETUTCDATE(),
GETUTCDATE()
)
SET #currentAgencyEntityBroadcastID = (SELECT SCOPE_IDENTITY())
INSERT INTO Propagation.AgencyEntityBroadcastKeys(
AgencyEntityBroadcastID,
ColumnName,
ColumnValue
) VALUES (
#currentAgencyEntityBroadcastID,
'AgencyID',
#currentAgencyID
)
INSERT INTO Propagation.AgencyEntityBroadcastKeys(
AgencyEntityBroadcastID,
ColumnName,
ColumnValue
) VALUES (
#currentAgencyEntityBroadcastID,
'FacilityID',
#currentFacilityID
)
FETCH NEXT FROM entity_cursor INTO #currentAgencyID, #currentFacilityID
END
CLOSE entity_cursor
DEALLOCATE entity_cursor
DROP TABLE #missingBroadcasts
SET NOCOUNT OFF;";
cmd.ExecuteNonQuery();
}

This kind of behaviour is usually caused by parameter sniffing problems. You can try with option recompile or optimize for unknown .
Option recompile
....
WHERE broadcastEntity.AgencyID = #agencyID
AND keys.AgencyEntityBroadcastID IS NULL
OPTION (RECOMPILE)
Optimize for unknown
.....
WHERE broadcastEntity.AgencyID = #agencyID
AND keys.AgencyEntityBroadcastID IS NULL
OPTION (OPTIMIZE FOR (#agencyID UNKNOWN, #broadcastEntityMappingID UNKNOWN))

Related

Instead Of Insert trigger doesn't work for my insert via C# code

I have the following trigger:
CREATE TRIGGER insert_or_update_AccountNews
ON AccountNews
INSTEAD OF INSERT
AS
BEGIN
SET NOCOUNT ON;
DECLARE
#AccountNumber bigint,
#NewsId int,
#TariffPlan nvarchar(1024)
SELECT #AccountNumber = INSERTED.AccountNumber,
#NewsId = INSERTED.NewsId,
#TariffPlan = INSERTED.TariffPlan
FROM INSERTED
IF EXISTS (SELECT NewsId FROM [AccountNews]
WHERE AccountNumber = #AccountNumber AND NewsId = #NewsId)
UPDATE [AccountNews]
SET TariffPlan = #TariffPlan
WHERE AccountNumber = #AccountNumber
AND NewsId = #NewsId
ELSE
INSERT INTO [AccountNews] (NewsId, AccountNumber, TariffPlan)
SELECT #NewsId, #AccountNumber, #TariffPlan
END;
And I have a table, as you can see, that is called AccountNews. It has the following columns:
AccountNumber, NewsId, TariffPlan
The idea is when I insert something into the table the trigger will determine if the data exists (I have unique constraint by AccountNumber and NewsId) or not exists. If the data not exists - the insert, otherwise - update.
And it works perfectly via the SQL console, like:
insert into AccountNews (NewsId, AccountId, TariffPlan)
values (12345, 777777, 'Hello world');
insert into AccountNews (NewsId, AccountId, TariffPlan)
values (12345, 777777, 'Hello world 2');
Next, I have this C# code to insert data:
DataTable table = await ReadAsStringAsync(file, newsId);
var connectionString = config.GetConnectionString("MyDbConnection");
using (SqlConnection connection = new SqlConnection(connectionString))
using (SqlBulkCopy bcp = new SqlBulkCopy(connection))
{
connection.Open();
bcp.BatchSize = 1000;
bcp.DestinationTableName = "[dbo].[AccountNews]";
bcp.ColumnMappings.Add("NewsId", "NewsId");
bcp.ColumnMappings.Add("AccountNumber", "AccountNumber");
bcp.ColumnMappings.Add("TariffPlan", "TariffPlan");
bcp.ColumnMappings.Add("Date", "Date");
await bcp.WriteToServerAsync(table);
}
In this case I don't see a result of my trigger. When I load some data that is already in my database I have a unique constraint exception.
To create an UPSERT-only table you can add a trigger like this:
use tempdb
go
drop table if exists AccountNews
create table AccountNews
(
AccountNumber bigint,
NewsId int,
TariffPlan nvarchar(1024),
constraint pk_AccountNews
primary key (AccountNumber, NewsId)
)
go
CREATE TRIGGER insert_or_update_AccountNews
ON AccountNews
INSTEAD OF INSERT
AS
BEGIN
SET NOCOUNT ON;
merge AccountNews as target
using (select * from inserted) as source
on (target.AccountNumber = source.AccountNumber and target.NewsId = source.NewsId)
when matched then
update set TariffPlan = source.TariffPlan
when not matched then
insert (AccountNumber, NewsId, TariffPlan)
values (source.AccountNumber, source.NewsId, source.TariffPlan);
END;
or wihout MERGE (which doen't permit duplicates within a single batch):
CREATE TRIGGER insert_or_update_AccountNews
ON AccountNews
INSTEAD OF INSERT
AS
BEGIN
SET NOCOUNT ON;
with q as
(
select a.*, i.TariffPlan NewTariffPlan
from AccountNews a
join inserted i
on a.AccountNumber = i.AccountNumber
and a.NewsId = i.NewsId
)
update q set TariffPlan = NewTariffPlan;
insert into AccountNews(AccountNumber,NewsId,TariffPlan)
select AccountNumber,NewsId,TariffPlan
from inserted i
where not exists
(
select *
from AccountNews a
where a.AccountNumber = i.AccountNumber
and a.NewsId = i.NewsId
);
END;
go
And you opt-in for triggers and constraint checking with SqlBulkCopyOptions, which you should normally do when bulk loading from an application because bypassing constraints or triggers requires ALTER TABLE privileges on the table.

Creating on-the-fly SQL stored procedure in C# then select it and then delete it

Hey all I am trying to figure out how to go about this. I am wanting to send a parameter that is the name of my table to a query in my C# program. I've read that this is not possible and they suggested that you make a stored procedure to do this.
So this is my code so far:
CREATE PROCEDURE _tmpSP
#TableName NVARCHAR(128)
AS
BEGIN
SET NOCOUNT ON;
DECLARE #Sql NVARCHAR(MAX);
SET #Sql = N'SELECT TOP 1 HelpMsg FROM ' + QUOTENAME(#TableName)
EXECUTE sp_executesql #Sql
DROP PROCEDURE [_tmpSP]
END
When I execute that in Server Management Studio it creates the SP but never executes that store procedure nor deletes it.
When I run that SP in Server Management Studio (right-clicking on it under Programmability>dbo._tmpSP and choosing Execute Stored Procedure) and give it the table name, it populates and then deletes the SP. This is the end result I want without having to make 2 query's.
The SQL query for when the SP runs is this (tHelp being the table name):
USE [TTool]
GO
DECLARE #return_value int
EXEC #return_value = [dbo].[_tmpSP]
#TableName = N'tHelp'
SELECT 'Return Value' = #return_value
GO
I get the returned help message and also returned value 0.
How can I modify this SP in order to do that?
Just do this, forget stored procedures:
EXECUTE sp_executesql 'SELECT TOP 1 HelpMsg FROM '+QUOTENAME(#TableName)
Dirty C#...
string qry = string.Format("SELECT TOP 1 HelpMSG FROM {0}", myTableName.Replace("'", "''"));
cmd = conn.CreateCommand();
cmd.CommandText = qry;
string helpMsg = conn.ExecuteScalar();
Where conn is an instance of System.Data.SqlClient.SqlConnection
I agree with #SsJVasto. If you still need your query not be hard coded in the C# program you can use an xml and keep the query in it. And fetch the xml and execute the query. I guess you would like to handle some dynamic stuff.
There is no point in doing this because there is quite complicated and also incurs the overhead of creating and dropping of the stored procedure. If you have a dynamic query that deals with some dynamic elements that cannot be pushed as parameters, you can construct the query string:
var query = $"SELECT TOP 1 col FROM {tableName}";
However, you must take care to avoid SQL injection if tableName is constructed based on user input. This question and its answers deal with this problem:
DbConnection connection = GetMyConnection();
DbProviderFactory factory = DbProviderFactories.GetFactory(connection);
DbCommandBuilder commandBuilder = factory.CreateCommandBuilder();
var tableName commandBuilder.QuoteIdentifier(rawTableName);
If "normal" (non table name) parameters are needed, pass them as usual within the query. E.g. #param1, #param2
You need create another SP to apply your logic. First let's see your SP:
CREATE PROCEDURE [_tmpSP]
#TableName NVARCHAR(128)
AS
BEGIN
DECLARE #Sql NVARCHAR(MAX);
SET #Sql = N'SELECT TOP 1 HelpMsg FROM ' + #TableName
EXEC(#Sql)
END
Then create another SP only if you need to drop the first one after return the result. The logic will be :
Create procedure auto_delete
#NewTableName
as
begin
EXEC _tmpSP #TableName = #NewTableName
Drop procedure [_tmpSP]
End
In C# (I assume you are using the 2nd SP above):
Your code could be like this:
..
using System.Data.SqlClient;
..
string a = YourTableName;
using (SqlConnection sqlCon = new SqlConnection(YourDatabaseConnection))
{
sqlCon.Open()
using (SqlCommand sqlCmd = sqlCon.CreateCommand())
{
sqlCmd.CommandText = "auto_delete";
sqlCmd.CommandType = CommandType.StoredProcedure;
sqlCmd.Parameters.Add(new SqlParameter("NewTableName", a));
sqlCmd.ExecuteNonQuery();
}
sqlCon.Close();
}

Safely query SQL table with variable table name

I'm trying to make some common code for retrieving identities from tables and that involves making an unsafe query string to inject the table name.
I read everywhere that I cannot safely inject the table name. So I want to query if the table exists, then based on the result, perform a real or dummy query.
var unsafeTableQuery = "SELECT [Id] FROM [dbo].[" + tableName + "] WHERE [BulkInsertSessionID] = #bulkInsertSessionId";
var guardQuery =
"DECLARE #Exists BIT = ( SELECT CAST( COUNT(1) AS BIT ) FROM sys.tables WHERE name = #TableName AND type = 'U' );" +
"IF (#Exists = 0) SELECT TOP 0 NULL 'Id'" +
"ELSE " + unsafeTableQuery;
var cmd = new SqlCommand(guardQuery, conn, tran);
cmd.Parameters.Add(new SqlParameter("#TableName", tableName));
cmd.Parameters.Add(new SqlParameter("#bulkInsertSessionId", bulkInsertSessionId));
using (SqlDataReader reader = cmd.ExecuteReader())
{
int index = 0;
while (reader.Read())
{
int id = (int)reader[0];
entities[index++].Id = id;
}
}
Even though I have an unsafe concatenation, I'm first querying the table name against the sys.tables by a parameter. And if it doesn't exist, the IF..ELSE block will never step into the unsafe query.
For easier readability I'm expecting to run the following query:
DECLARE #Exists BIT = ( SELECT CAST( COUNT(1) AS BIT ) FROM sys.tables WHERE name = #TableName AND type = 'U' );
IF(#Exists = 0)
SELECT TOP 0 NULL 'Id'
ELSE
SELECT [Id] from <InjectedTableName> where BulkInsertSessionID = #bulkSessionId
Am I correct in my assumption that this is safe?
Suppose your users have an access to change the variable tableName. I suppose some user types it on some form. Suppose he types this:
Users]; DROP TABLE Users;--
Then your whole command will be:
DECLARE #Exists BIT = ( SELECT CAST( COUNT(1) AS BIT ) FROM sys.tables WHERE name = #TableName AND type = 'U' );
IF(#Exists = 0)
SELECT TOP 0 NULL 'Id'
ELSE
SELECT [Id] from [Users]; DROP TABLE Users;-- where BulkInsertSessionID = #bulkSessionId
This will do its IF ELSE part and then will go to next statement which is:
DROP TABLE Users;
Note that drop statement will execute in any case even if ELSE part is not executed, because you don't have BEGIN END. Note that the rest is commented out... This is most basic injection method...

stored procedure calling from c# and iteration issue in Merge

I have over a million records in the list. I pass all records at once from table to stored procedure .In stored procedure i have to have iteration to go thorugh all the rows in the table and for each row it takes table row modified date based on jobid and checks if it exist in database and based on it either it updates or insert the record. I feel that my procedure is not correct, would be glad if someone help on this.
foreach (No_kemi no_list in newforSQL)
{
DataTable _dt = new DataTable("table");
_dt.Columns.Add("JobID", typeof(string));
_dt.Columns.Add("CreatedDate", typeof(datetime));
_dt.Columns.Add("ModifiedDate", typeof(datetime));
_dt.Columns.Add("DbDate", typeof(datetime));
_dt.Columns.Add("SubGUID", typeof(string));
_dt.Columns.Add("eType", typeof(string));
// adding over a million records in the table
_dt.Rows.Add(no_list.ID,no_list.CreatedDate,no_list.ModifiedDate,no_list.DbDate,no_list.SubGUID,no_list.eType);
}
using (SqlCommand sqlCommand = new SqlCommand())
{
sqlCommand.CommandType = CommandType.StoredProcedure;
sqlCommand.CommandText = "Process_NO_table";
sqlCommand.Connection = connection;
SqlParameter typeParam = sqlCmd.Parameters.AddWithValue("#track", _dt);
typeParam .SqlDbType = SqlDbType.Structured;
sqlCmd.ExecuteNonQuery();
}
my tabletype and procedure:
CREATE TYPE TrackType AS TABLE
(
t_Id uniqueidentifier, t_JobID nvarchar(50), t_CreatedDate datetime2(7), t_ModifiedDate datetime2(7), t_DbDate datetime2(7)
t_SubGUID nvarchar(MAX), t_eType nvarchar(MAX)
);
GO
ALTER/CREATE PROCEDURE [dbo].[Process_NO_table] // i will change to alter after i create it
#track TrackType READONLY
AS
// i need to iterate all the rows of the table(over a million)
Declare #rows INT
Declare #i int = 0
Declare #count int = (SELECT COUNT(*) FROM #track)
DECLARE #is INT
WHILE (#i < #count)
BEGIN
-- first i check modified date from the database table
SELECT #is = COUNT(*) FROM NO_table WHERE [JobID] IN (SELECT [t_JobID] FROM #track)
MERGE [dbo].[NO_table] AS [Target]
USING #track AS [Source]
-- if the database modifed date is less than the modifeid date from the proceduretable(#track) then it updates the records
ON [Target].[ModifiedDate] < [Source].[t_ModifiedDate] AND JobID = t_JobID
WHEN MATCHED THEN
UPDATE SET [JobID] = [Source].[t_JobID],
[CreatedDate] = [Source].[t_CreatedDate]
[DbDate]= [Source].[t_DbDate]
[ModifiedDate] = [Source].[t_ModifiedDate]
[SubGUID] = [Source].[t_SubGUID]
[eType] = [Source].[t_eType]
-- if the database modifed dateis not existing then it insert the record
MERGE [dbo].[NO_table] AS [Target]
USING #track AS [Source]
ON (#is != 0)
WHEN NOT MATCHED THEN
INSERT INTO [NO_table] ( [JobID], [CreatedDate], [ModifiedDate], [DbDate], [SubGUID], [eType] )
VALUES ( [Source].[t_JobID], [Source].[t_CreatedDate], [Source].[t_ModifiedDate], [Source].[t_DbDate], [Source].[t_SubGUID], [Source].[t_eType] );
SET #i = #i + 1
END
GO
I think you have a large number of syntax errors in your SQL (assuming MS SQL), but your merge condition is probably giving you the invalid syntax near WHERE, because you need to use AND, not WHERE.
ON [Target].[ModifiedDate] < [Source].[t_ModifiedDate] WHERE JobID = t_JobID
should be
ON [Target].[ModifiedDate] < [Source].[t_ModifiedDate] AND JobID = t_JobID
The Select Top 1 and the WHEN MATCHED THEN after the null check for #dbmoddate need to go away as well, as those are also causing syntax issues.
The insert after the null check for #dbmoddate needs a table specified so it actually knows what to insert into.
You also need to end your merge statement with a semicolon.
UPDATED ANSWER:
Now that you have this more cleaned up, I can better see what you're trying to do. At a high level, you want to simply update existing records where the modified date is less than the modified date of on your custom type. If there does not exist a record in your table that does exist in your custom type, then insert it.
With that said, you don't actually need to loop because you aren't doing anything with your loop. What you currently have and what I'm posting below this is all set-based results, not iterative.
You can make this much simpler by getting rid of the merge statements and doing a simple Update and Insert like I have below. The merge would make more sense if your condition between the two statements was the same (i.e. if you didn't have the check for modified date, then merge would be OK) because then you can use the keywords WHEN MATCHED and WHEN NOT MATCHED and have it in one single merge statement. I personally stay away from MERGE statements because they tend to be a little buggy and there are a number of things you have to watch out for.
I think this solution will be better in the long run as it is easier to read and more maintainable...
CREATE TYPE TrackType AS TABLE
(
t_Id uniqueidentifier, t_JobID nvarchar(50), t_CreatedDate datetime2(7), t_ModifiedDate datetime2(7), t_DbDate datetime2(7)
,t_SubGUID nvarchar(MAX), t_eType nvarchar(MAX)
);
GO
CREATE PROCEDURE [dbo].[Process_NO_table] -- i will change to alter after i create it
#track TrackType READONLY
AS
-- i need to iterate all the rows of the table(over a million)
Update [NO_table]
SET [JobID] = T.[t_JobID],
[CreatedDate] = T.[t_CreatedDate],
[DbDate]= T.[t_DbDate],
[ModifiedDate] = T.[t_ModifiedDate],
[SubGUID] = T.[t_SubGUID] ,
[eType] = T.[t_eType]
From #track T
Where [NO_table].[JobID] = T.[t_JobID]
And [NO_table].[ModifiedDate] < T.[t_ModifiedDate]
Insert [NO_Table]
(
[JobID],
[CreatedDate],
[ModifiedDate],
[DbDate],
[SubGUID],
[eType]
)
Select T.[t_JobID],
T.[t_CreatedDate],
T.[t_ModifiedDate],
T.[t_DbDate],
T.[t_SubGUID],
T.[t_eType]
From #track T
Where Not Exists (Select 1 From [NO_table] where T.[t_JobID] = [NO_table].[JobID])
GO

Getting autonumber primary key from MS SQL Server

I am currently working in C#, and I need to insert a new record into one table, get the new primary key value, and then use that as a foreign key reference in inserting several more records. The Database is MS SQL Server 2003. All help is appreciated!
The way to get the identity of the inserted row is with the SCOPE_IDENTITY() function. If you're using stored procedures then this would look something like the following to return the row identity as an output parameter.
CREATE PROCEDURE dbo.MyProcedure
(
#RowId INT = NULL OUTPUT
)
AS
INSERT INTO MyTable
(
Column1
,Column2
,...
)
VALUES
(
#Param1
,#Param2
,...
);
SET #RowId = SCOPE_IDENTITY();
You can then use this value for any subsequent inserts (alternatively, if you can pass the data all into the stored procedure, then you can use it in the remainder of the procedure body).
If you're passing the SQL in dynamically then you use much the same technique, but with a single string with statement delimiters (also ; in SQL), e.g.:
var sql = "INSERT INTO MyTable (Column1, Column2, ...) VALUES (#P1, #P2, ...);" +
"SELECT SCOPE_IDENTITY();";
Then if you execute this using ExecuteScalar you'll be able to get the identity back as the scalar result and cast it to the right type. Alternatively you could build up the whole batch in one go, e.g.
var sql = "DECLARE #RowId INT;" +
"INSERT INTO MyTable (Column1, Column2, ...) VALUES (#P1, #P2, ...);" +
"SET #RowId = SCOPE_IDENTITY();" +
"INSERT INTO MyOtherTable (Column1, ...) VALUES (#P3, #P4, ...);";
This may not be exactly the right syntax, and you may need to use SET NOCOUNT ON; at the start (my mind is rusty as I rarely use dynamic SQL) but it should get you on the right track.
The best way of doing this is the use SCOPE_IDENTITY() function in TSQL. This should be executed as part of the insert i.e.
SqlCommand cmd = new SqlCommand(#"
INSERT INTO T (Name) VALUES(#Name)
SELECT SCOPE_IDENTITY() As TheId", conn);
cmd.AddParameter("#Name", SqlDbType.VarChar, 50).Value = "Test";
int tId = (int)cmd.ExecuteScalar();
Alternatively you can assign SCOPE_IDENTITY() to a variable to be used in successive statements. e.g.
DECLARE #T1 int
INSERT INTO T (Name) VALUES('Test')
SELECT #T1 = SCOPE_IDENTITY()
INSERT INTO T2 (Name, TId) VALUES('Test', #T1)
If you are just using SQL then check Duncan's answer. If however you are using LINQ then you can create the entity, save it to the DB and the ID parameter will be populated automatically.
Given a user entity and a user table it might look like this:
using(var db = new DataContext()) {
var user = new User { Name = "Jhon" };
db.Users.InsertOnSubmit(user);
db.SubmitChanges();
/* At this point the user.ID field will have the primary key from the database */
}

Categories

Resources