Develop Reference

SqlBulkCopy truncating string in VARCHAR(MAX) column

I have a simple table on SQL Server 2016:
CREATE TABLE AgentDownload
(
Download VARCHAR(max)
)
I'm using the below code to populate this table using the SqlBulkCopy class in C#.
var agentDataTable = new DataTable();
agentDataTable.Clear();
agentDataTable.Columns.Add("Download");
var agentDownloadRow = agentDataTable.NewRow();
var veryLongString = "aaa..." // 200,000 a's repeated
agentDownloadRow["Download"] = veryLongString;
agentDataTable.Rows.Add(agentDownloadRow);
using (var connection = new SqlConnection("Data Source=Server;Initial Catalog=Database;Integrated Security=True;"))
{
connection.Open();
var transaction = connection.BeginTransaction();
using (var sbCopy = new SqlBulkCopy(connection, SqlBulkCopyOptions.CheckConstraints, transaction))
{
sbCopy.BulkCopyTimeout = 0;
sbCopy.BatchSize = 10000;
sbCopy.DestinationTableName = "AgentDownload";
sbCopy.WriteToServer(agentDataTable);
}
transaction.Commit();
}
On the database when I retrieve this entry, the cell value is truncated at 65,535 characters. This is a limit of some sort but I'm not sure where this is coming from or if there is any way around this? The column can contain far more characters and a string type in C# can also contain far more characters. Is there any to do this operation?
I thought this might be a limitation of SqlBulkCopy class but using alternative code such as below also produces the same result:
using (var connection = new SqlConnection("Data Source=Server;Initial Catalog=Database;Integrated Security=True;"))
{
connection.Open();
var transaction = connection.BeginTransaction();
var cmd = connection.CreateCommand();
cmd.Transaction = transaction;
cmd.CommandText = "insert into AgentDownload (Download) values (CAST('' AS VARCHAR(MAX)) + #testVariable)";
var parameter = new SqlParameter
{
ParameterName = "testVariable",
DbType = DbType.String,
Size = -1,
Value = veryLongString
};
cmd.Parameters.Add(parameter);
cmd.ExecuteNonQuery();
transaction.Commit();
}

It seems you are using SSMS to view the bulk inserted data. SSMS truncates large values by default to 65535 characters.
To see the entire value in the results grid for the current query window, change the Max Characters Retrieved value from the SSMS menu under Query-->Query Options-->Results-->Grid.
One can also specify the value for all new query windows under Tools-->Options-->Query Results-->SQL Server-->Results to Grid. However, consider the implications on client memory requirements with many rows containing large values. I suggest one change the global option judiciously.

How to increase the efficiency for saving multiple rows in .NET windows application?

In my .NET Windows application, I need to insert around 10 millions data per time.However, it takes more than 5 minuites to save into SQL server according to my inefficient code.Are there any best way minimize the time taken to save those data?
private void saveAllDataInGrid()
{
int rowCount = dataGridView1.RowCount;
String str = "server=DESKTOP-TDV8JQ7;database=ExcelFileApp;Integrated Security=SSPI";
SqlConnection con = new SqlConnection(str);
con.Open();
for (int count = 0; count < rowCount; count++)
{
try
{
String query = "insert into TemMainTable values (#p1,#p2,#p3,#p4,#p5,#p6,#p7,#p8,#p9,#p10,#p11,#p12,#p13,#p14,#p15,#p16,#p17,#p18,#p19,#p20,#p21,#p22,#p23)";
SqlCommand cmd = new SqlCommand(query, con);
cmd.Parameters.AddWithValue("#p1", dataGridView1.Rows[count].Cells[0].Value.ToString());
cmd.Parameters.AddWithValue("#p2", dataGridView1.Rows[count].Cells[1].Value.ToString());
cmd.Parameters.AddWithValue("#p3", dataGridView1.Rows[count].Cells[2].Value.ToString());
cmd.Parameters.AddWithValue("#p4", dataGridView1.Rows[count].Cells[3].Value.ToString());
cmd.Parameters.AddWithValue("#p5", dataGridView1.Rows[count].Cells[4].Value.ToString());
cmd.Parameters.AddWithValue("#p6", dataGridView1.Rows[count].Cells[5].Value.ToString());
cmd.Parameters.AddWithValue("#p7", dataGridView1.Rows[count].Cells[6].Value.ToString());
cmd.Parameters.AddWithValue("#p8", dataGridView1.Rows[count].Cells[7].Value.ToString());
cmd.Parameters.AddWithValue("#p9", dataGridView1.Rows[count].Cells[8].Value.ToString());
cmd.Parameters.AddWithValue("#p10", dataGridView1.Rows[count].Cells[9].Value.ToString());
cmd.Parameters.AddWithValue("#p11", dataGridView1.Rows[count].Cells[10].Value.ToString());
cmd.Parameters.AddWithValue("#p12", dataGridView1.Rows[count].Cells[10].Value.ToString());
cmd.Parameters.AddWithValue("#p13", dataGridView1.Rows[count].Cells[12].Value.ToString());
cmd.Parameters.AddWithValue("#p14", dataGridView1.Rows[count].Cells[13].Value.ToString());
cmd.Parameters.AddWithValue("#p15", dataGridView1.Rows[count].Cells[14].Value.ToString());
cmd.Parameters.AddWithValue("#p16", dataGridView1.Rows[count].Cells[15].Value.ToString());
cmd.Parameters.AddWithValue("#p17", dataGridView1.Rows[count].Cells[16].Value.ToString());
cmd.Parameters.AddWithValue("#p18", dataGridView1.Rows[count].Cells[17].Value.ToString());
cmd.Parameters.AddWithValue("#p19", dataGridView1.Rows[count].Cells[18].Value.ToString());
cmd.Parameters.AddWithValue("#p20", dataGridView1.Rows[count].Cells[19].Value.ToString());
cmd.Parameters.AddWithValue("#p21", dataGridView1.Rows[count].Cells[20].Value.ToString());
cmd.Parameters.AddWithValue("#p22", dataGridView1.Rows[count].Cells[21].Value.ToString());
cmd.Parameters.AddWithValue("#p23", dataGridView1.Rows[count].Cells[22].Value.ToString());
int i = cmd.ExecuteNonQuery();
lblDataSaved.Text = "No of rows Saved : " + rowCount;
}
catch (Exception es)
{
MessageBox.Show(es.Message);
}
}
con.Close();
MessageBox.Show("Successfully Saved");
}
Thanks!!

You can use User-Defined Table Types for bulk insert. You need to create a User-Defined Table Types in the database SQL Server Management Studio >> Programability >> User-Defined Table Types
The SQL Script Syntax will be like this to create the User-Defined Table Types
CREATE TYPE [dbo].[TableList] AS TABLE(
[SampleColumn1] [bigint] NOT NULL,
[SampleColumn2] [nvarchar](100) NOT NULL
)
The Sample SQL Insert Query will be as below -
INSERT INTO SampleTable
( SampleColumn1 ,
SampleColumn2
)
SELECT cusTbl.SampleColumn1 ,
cusTbl.SampleColumn2
FROM #TableList AS cusTbl;
In C# Code - the general Syntax to insert is as below -
SqlCommand cmd = new SqlCommand(query, con);
var tvparam = cmd.Parameters.AddWithValue("#TableList", tableList);
tvparam.SqlDbType = SqlDbType.Structured;
tvparam.TypeName = "dbo.TableList"
cmd.Connection.Open();
cmd.ExecuteNonQuery();
Note: You had used DataGridView in your Windows Application. So you can easily pass the DataTable which you had used to bind the DataGridView.
Take care that the DataTable have same ColumnName and DataType which is declared in SQL Server Database User-Defined Table Types
I need to insert around 10 millions data per time.However, it takes more than 5 minutes
Definitely time taken to insert will reduce - compared to sending rows one by one to SQL Server.
More Reading Here and in this Blog

To optimize the query performance

i have stored procedure which returns like minimum of 40K rows and it takes like 20 seconds in SSMS 2008 R2 where the database resides in Sql Azure but when i run the same Sp in my c# application using EF 5 or just Normal ADo.NET it took like 70-80 seconds.
table has a non-clustered index on ScenarioID
Sp is just a select statement with where condition. select * from Cost where ScenarioID= #ID
using (SqlConnection con = new SqlConnection(constr))
{
using (SqlCommand cmd = new SqlCommand("sp_GetActCostsByID", con))
{
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add("#ID", SqlDbType.VarChar).Value = ID;
con.Open();
DataTable dt = new DataTable();
DateTime timee = DateTime.Now;
Console.WriteLine(timee);
dt.Load(cmd.ExecuteReader());
timee = DateTime.Now;
Console.WriteLine(timee);
}
}
Is there any way to increase the performance:
My execution Plan:

Your nonclustered index on ScenarioID might not be helping, if you're not INCLUDEing all the columns you're trying to return, as it will need to do lookups to get those other columns - if there are lots of rows for that Scenario, you could end up with an ordinary table scan. And this comes down to statistics, so can vary from server to server.
If you can avoid the need for lookups, you'll get more consistent performance.

Select Into Explanation / Temporary Tables

Based on the tutorial on SQL Temporary Tables, it should be OK to create a temp table by using SELECT * INTO #tempTable FROM tableA but it's throwing me SQLException when I trying to SELECT * FROM #tempTable saying that Invalid object name '#tempTable'. May I know what's the proper way of using a temp table in C#?
string sql = "SELECT * INTO ##tempTable FROM (SELECT * FROM tableA)";
using (var command = new SqlCommand(sql, connection))
{
string sqlNew = "SELECT * FROM ##tempTable";
using (var command2 = new SqlCommand(sqlNew, connection))
{
using (var reader = command2.ExecuteReader())
{
while (reader.Read())
{
Console.WriteLine(reader["column1"].ToString());
}
Console.ReadLine();
}
}
}
My Objective is tryint to using the data retrieved from sqlVar and insert them into a tempTable and perform some operation on it. Very much appreciated if there is some sample code on how to fit the code into the above code. Thank You.

But why you need temp table at SQL server side..
1) if you wish to perform operation on C# side just take data in DATASET instead of DATAREADER .. and
DataSet dataset = new DataSet();
using (SqlConnection conn = new SqlConnection(connString))
{
SqlDataAdapter adapter = new SqlDataAdapter();
adapter.SelectCommand = new SqlCommand("select * from tableA", conn);
conn.Open();
adapter.Fill(dataset);
conn.Close();
foreach (DataRow row in dataset.Tables[0]) // Loop over the rows.
{
// perform your operation
}
}
2) if you need to perform operation on SQL side then create a stored procedure at SQL server .. in the stored procedure create #table and use it ..
3) and you do not want to create DATASET then you can take data LIST and perform your operation on C# side

You are not executing the first command at all, so the SELECT INTO isn't executed, so the temporary table is not created, so you get an error about the table not existing.
The code should read:
string sql = "SELECT * INTO ##tempTable FROM (SELECT * FROM tableA)";
using (var command = new SqlCommand(sql, connection))
{
command.ExecuteNonQuery(); // <-- THIS
string sqlNew = "SELECT * FROM ##tempTable";
using (var command2 = new SqlCommand(sqlNew, connection))
{
using (var reader = command2.ExecuteReader())
{
while (reader.Read())
{
Console.WriteLine(reader["column1"].ToString());
}
Console.ReadLine();
}
}
}

1-SELECT * INTO # tempTable FROM tableA (local temp)or
2-SELECT * INTO ## tempTable FROM tableA (global temp)then
Local temp tables are only available to the current connection for the user; and they are automatically deleted when the user disconnects from instances. Local temporary table name is stared with hash ("#") sign.
Global Temp Table
Global Temporary tables name starts with a double hash ("##"). Once this table has been created by a connection, like a permanent table it is then available to any user by any connection. It can only be deleted once all connections have been closed.
Both, Temporary tables are stored inside the Temporary Folder of tempdb. Whenever we create a temporary table, it goes to Temporary folder of tempdb database.
temp table in SQL DB

Change your temp table from #tempTable to ##tempTable.
Using ## means a global temp table that stays around. You'll need to Drop it after you have completed your task.
If Exists(Select * from tempdb..sysobjects Where id = object_id('tempdb.dbo.#tempTable'))
DROP TABLE #tempTable

I think your answer is in the comment:
Temporary tables available during the session that creates them.
If you want to actualy get the data you have to perform a SELECT statement from this temporary table within the same scope.
One more thing:
I don't see you are executing the var command, you are missing this line:
string sql = "SELECT * INTO ##tempTable FROM (SELECT * FROM tableA)";
using (var command = new SqlCommand(sql, connection))
{
command.ExecuteNonQuery();// This line is missing..
string sqlNew = "SELECT * FROM ##tempTable";
using (var command2 = new SqlCommand(sqlNew, connection))
{
using (var reader = command2.ExecuteReader())
{
while (reader.Read())
{
Console.WriteLine(reader["column1"].ToString());
}
Console.ReadLine();
}
}
}
But missing the line isn't the reason why your implementation is wrong..

SqlBulkInsert with a DataTable to a Linked Server

I'm working with 2 SQL 2008 Servers on different machines. The server names are source.ex.com, and destination.ex.com.
destination.ex.com is linked to source.ex.com and the appropriate permissions are in place for source.ex.com to write to a database called bacon-wrench on destination.ex.com
I've logged into source.ex.com via SMS and tested this query (successfully):
INSERT INTO [destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]
(PunchID, BaconID) VALUES (4,6);
In a C# .NET 4.0 WebPage I connect to source.ex.com and perform a similar query (successfully):
using(SqlConnection c = new SqlConnection(ConfigurationManager.ConnectionStrings["SOURCE"].ConnectionString))
{
c.Open();
String sql = #"
INSERT INTO [destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]
(PunchID, BaconID) VALUES (34,56);";
using(SqlCommand cmd = new SqlCommand(sql, c))
{
cmd.ExecuteNonQuery();
}
}
For small sets of insert statements (say 20 or less) doing something like this performs fine:
using(SqlConnection c = new SqlConnection(ConfigurationManager.ConnectionStrings["SOURCE"].ConnectionString))
{
c.Open();
String sql = #"
INSERT INTO [destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]
(PunchID, BaconID) VALUES (34,56);
INSERT INTO [destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]
(PunchID, BaconID) VALUES (22,11);
INSERT INTO [destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]
(PunchID, BaconID) VALUES (33,55);
INSERT INTO [destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]
(PunchID, BaconID) VALUES (1,2);";
using(SqlCommand cmd = new SqlCommand(sql, c))
{
cmd.ExecuteNonQuery();
}
}
I'm trying to do something like this with around 20000 records. The above method takes 11 minutes to complete -- which I assume is the server sreaming at me to make it some kind of bulk operation. From other StackOverflow threads the SqlBulkCopy class was recommended and it takes as a parameter DataTable, perfect!
So I build a DataTable and attempt to write it to the server (fail):
DataTable dt = new DataTable();
dt.Columns.Add("PunchID", typeof(int));
dt.Columns.Add("BaconID", typeof(int));
for(int i = 0; i < 20000; i++)
{
//I realize this would make 20000 duplicate
//rows but its not important
dt.Rows.Add(new object[] {
11, 33
});
}
using(SqlConnection c = new SqlConnection(ConfigurationManager.ConnectionStrings["SOURCE"].ConnectionString))
{
c.Open();
using(SqlBulkCopy bulk = new SqlBulkCopy(c))
{
bulk.DestinationTableName = "[destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]";
bulk.ColumnMappings.Add("PunchID", "PunchID");
bulk.ColumnMappings.Add("BaconID", "BaconID");
bulk.WriteToServer(dt);
}
}
EDIT2: The below message is what I'm attempting to fix:
The web page crashes at bulk.WriteToServer(dt); with an error message Database bacon-wrench does not exist please ensure it is typed correctly. What am I doing wrong? How do I change this to get it to work?
EDIT1:
I was able to speed up the query significantly using the below syntax. But it is still very slow for such a small record set.
using(SqlConnection c = new SqlConnection(ConfigurationManager.ConnectionStrings["SOURCE"].ConnectionString))
{
c.Open();
String sql = #"
INSERT INTO [destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]
(PunchID, BaconID) VALUES
(34,56),
(22,11),
(33,55),
(1,2);";
using(SqlCommand cmd = new SqlCommand(sql, c))
{
cmd.ExecuteNonQuery();
}
}

If you are using SQL Server 2008+, you can introduce a Table user datatype. Prepare the type, receiving table and stored procedure something like below. Data type and stored procedure is on the local system. I generally have an if statement in the code detecting whether the table is remote or local, remote I do this, local I use SqlBulkCopy.
if(TYPE_ID(N'[Owner].[TempTableType]') is null)
begin
CREATE TYPE [Owner].[TempTableType] AS TABLE ( [PendingID] uniqueidentifier, [Reject] bit)
end
IF NOT EXISTS (SELECT * FROM [LinkedServer].[DatabaseOnLS].sys.tables where name = 'TableToReceive')
EXEC('
CREATE TABLE [DatabaseOnLS].[Owner].[TableToReceive] ( [PendingID] uniqueidentifier, [Reject] bit)
') AT [LinkedServer]
else
EXEC('
TRUNCATE TABLE [DatabaseOnLS].[Owner].[TableToReceive]
') AT [LinkedServer]
CREATE PROCEDURE [Owner].[TempInsertTable]
#newTableType TempTableType readonly
AS
BEGIN
insert into [LinkedServer].[DatabaseOnLS].[Owner].[TableToReceive] select * from #newTableType
END
In the C# code you can then do something like this to insert the DataTable into the table on the linked server (I'm using an existing UnitOfWork, which already have a connection and transaction):
using (var command = new SqlCommand("TempInsertTable",
oUoW.Database.Connection as SqlConnection) { CommandType = CommandType.StoredProcedure }
)
{
command.Transaction = oUoW.Database.CurrentTransaction as SqlTransaction;
command.Parameters.Add(new SqlParameter("#newTableType", oTempTable));
drResults = command.ExecuteReader();
drResults.Close();
}

After trying a number of things including linked server settings, collations, synonyms, etc., I eventually got to this error message:
Inserting into remote tables or views is not allowed by using the BCP utility or by using BULK INSERT.
Perhaps you can bulk insert to a staging table on your local server (your code works fine for this) and then insert from that staging table to your linked server from there, followed by a local delete of the staging table. You'll have to test for performance.