I'm working with 2 SQL 2008 Servers on different machines. The server names are source.ex.com, and destination.ex.com.
destination.ex.com is linked to source.ex.com and the appropriate permissions are in place for source.ex.com to write to a database called bacon-wrench on destination.ex.com
I've logged into source.ex.com via SMS and tested this query (successfully):
INSERT INTO [destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]
(PunchID, BaconID) VALUES (4,6);
In a C# .NET 4.0 WebPage I connect to source.ex.com and perform a similar query (successfully):
using(SqlConnection c = new SqlConnection(ConfigurationManager.ConnectionStrings["SOURCE"].ConnectionString))
{
c.Open();
String sql = #"
INSERT INTO [destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]
(PunchID, BaconID) VALUES (34,56);";
using(SqlCommand cmd = new SqlCommand(sql, c))
{
cmd.ExecuteNonQuery();
}
}
For small sets of insert statements (say 20 or less) doing something like this performs fine:
using(SqlConnection c = new SqlConnection(ConfigurationManager.ConnectionStrings["SOURCE"].ConnectionString))
{
c.Open();
String sql = #"
INSERT INTO [destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]
(PunchID, BaconID) VALUES (34,56);
INSERT INTO [destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]
(PunchID, BaconID) VALUES (22,11);
INSERT INTO [destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]
(PunchID, BaconID) VALUES (33,55);
INSERT INTO [destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]
(PunchID, BaconID) VALUES (1,2);";
using(SqlCommand cmd = new SqlCommand(sql, c))
{
cmd.ExecuteNonQuery();
}
}
I'm trying to do something like this with around 20000 records. The above method takes 11 minutes to complete -- which I assume is the server sreaming at me to make it some kind of bulk operation. From other StackOverflow threads the SqlBulkCopy class was recommended and it takes as a parameter DataTable, perfect!
So I build a DataTable and attempt to write it to the server (fail):
DataTable dt = new DataTable();
dt.Columns.Add("PunchID", typeof(int));
dt.Columns.Add("BaconID", typeof(int));
for(int i = 0; i < 20000; i++)
{
//I realize this would make 20000 duplicate
//rows but its not important
dt.Rows.Add(new object[] {
11, 33
});
}
using(SqlConnection c = new SqlConnection(ConfigurationManager.ConnectionStrings["SOURCE"].ConnectionString))
{
c.Open();
using(SqlBulkCopy bulk = new SqlBulkCopy(c))
{
bulk.DestinationTableName = "[destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]";
bulk.ColumnMappings.Add("PunchID", "PunchID");
bulk.ColumnMappings.Add("BaconID", "BaconID");
bulk.WriteToServer(dt);
}
}
EDIT2: The below message is what I'm attempting to fix:
The web page crashes at bulk.WriteToServer(dt); with an error message Database bacon-wrench does not exist please ensure it is typed correctly. What am I doing wrong? How do I change this to get it to work?
EDIT1:
I was able to speed up the query significantly using the below syntax. But it is still very slow for such a small record set.
using(SqlConnection c = new SqlConnection(ConfigurationManager.ConnectionStrings["SOURCE"].ConnectionString))
{
c.Open();
String sql = #"
INSERT INTO [destination.ex.com].[bacon-wrench].[dbo].[tblFruitPunch]
(PunchID, BaconID) VALUES
(34,56),
(22,11),
(33,55),
(1,2);";
using(SqlCommand cmd = new SqlCommand(sql, c))
{
cmd.ExecuteNonQuery();
}
}
If you are using SQL Server 2008+, you can introduce a Table user datatype. Prepare the type, receiving table and stored procedure something like below. Data type and stored procedure is on the local system. I generally have an if statement in the code detecting whether the table is remote or local, remote I do this, local I use SqlBulkCopy.
if(TYPE_ID(N'[Owner].[TempTableType]') is null)
begin
CREATE TYPE [Owner].[TempTableType] AS TABLE ( [PendingID] uniqueidentifier, [Reject] bit)
end
IF NOT EXISTS (SELECT * FROM [LinkedServer].[DatabaseOnLS].sys.tables where name = 'TableToReceive')
EXEC('
CREATE TABLE [DatabaseOnLS].[Owner].[TableToReceive] ( [PendingID] uniqueidentifier, [Reject] bit)
') AT [LinkedServer]
else
EXEC('
TRUNCATE TABLE [DatabaseOnLS].[Owner].[TableToReceive]
') AT [LinkedServer]
CREATE PROCEDURE [Owner].[TempInsertTable]
#newTableType TempTableType readonly
AS
BEGIN
insert into [LinkedServer].[DatabaseOnLS].[Owner].[TableToReceive] select * from #newTableType
END
In the C# code you can then do something like this to insert the DataTable into the table on the linked server (I'm using an existing UnitOfWork, which already have a connection and transaction):
using (var command = new SqlCommand("TempInsertTable",
oUoW.Database.Connection as SqlConnection) { CommandType = CommandType.StoredProcedure }
)
{
command.Transaction = oUoW.Database.CurrentTransaction as SqlTransaction;
command.Parameters.Add(new SqlParameter("#newTableType", oTempTable));
drResults = command.ExecuteReader();
drResults.Close();
}
After trying a number of things including linked server settings, collations, synonyms, etc., I eventually got to this error message:
Inserting into remote tables or views is not allowed by using the BCP utility or by using BULK INSERT.
Perhaps you can bulk insert to a staging table on your local server (your code works fine for this) and then insert from that staging table to your linked server from there, followed by a local delete of the staging table. You'll have to test for performance.
Related
I'm looking to do something simulair toward here: How do I insert multiple rows WITHOUT repeating the "INSERT INTO dbo.Blah" part of the statement?
except that in addition towards doing this in one query (faster then several dozen) I also want to do this parameterized as the input comes from the web.
Currently I have
foreach(string data in Scraper){
SqlConnection conn = new SqlConnection(WebConfigurationManager.AppSettings["ConnectionInfo"].ToString());
string query = "INSERT INTO DATABASE('web',#data)";
SqlCommand sqlCommand= new SqlCommand(query, conn);
sqlCommand.Parameters.AddWithValue("#data", data);
Command.executeNonQuery();
conn.close();
}
Which is a bit slugish (note the real example has a lot more colums but that would make things more confusing).
Since you are using c# and sql server 2008, you can use a table valued parameter to insert multiple rows to your database.
Here is a short description on how to do this:
First, you need to create a user defined table type:
CREATE TYPE MyTableType AS TABLE
(
Col1 int,
Col2 varchar(20)
)
GO
Then, you need to create a stored procedure that will accept this table type as a parameter
CREATE PROCEDURE MyProcedure
(
#MyTable dbo.MyTableType READONLY -- NOTE: table valued parameters must be Readonly!
)
AS
INSERT INTO MyTable (Col1, Col2)
SELECT Col1, Col2
FROM #MyTable
GO
Finally, execute this stored procedure from your c# code:
DataTable dt = new DataTable();
dt.Columns.Add("Col1", typeof(int));
dt.Columns.Add("Col2", typeof(string));
// Fill your data table here
using (var con = new SqlConnection("ConnectionString"))
{
using(var cmd = new SqlCommand("MyProcedure", con))
{
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add("#MyTable", SqlDbType.Structured).Value = dt;
con.Open();
cmd.ExecuteNonQuery();
}
}
You can make use of the SQL syntax:
INSERT INTO YOUR_TABLE (dataColumn) VALUES (data1),(data2),(data3)
So loop over your rows you wanna insert and append ",(datax)" to your query and also add the corresponding parameter.
Perhaps it helps.
I would like to insert all the id's in a sql table. The following way works but this take very long. What is the best or better way to do this to increase the speed.
using (SqlConnection connection = new SqlConnection(ConfigurationManager.ConnectionStrings["DefaultConnection"].ConnectionString))
{
string query = "";
foreach (var id in ids) // count = 60000
{
{
query += "INSERT INTO [table] (id) VALUES (" + id + ");";
}
}
SqlCommand command = new SqlCommand(query, connection);
connection.Open();
using (SqlDataReader reader = command.ExecuteReader())
{
reader.Close();
}
connection.Close();
}
You can use the SqlBulkCopy to insert large amounts of data - something like this:
// define a DataTable with the columns of your target table
DataTable tblToInsert = new DataTable();
tblToInsert.Columns.Add(new DataColumn("SomeValue", typeof (int)));
// insert your data into that DataTable
for (int index = 0; index < 60000; index++)
{
DataRow row = tblToInsert.NewRow();
row["SomeValue"] = index;
tblToInsert.Rows.Add(row);
}
// set up your SQL connection
using (SqlConnection connection = new SqlConnection(ConfigurationManager.ConnectionStrings["DefaultConnection"].ConnectionString))
{
// define your SqlBulkCopy
SqlBulkCopy bulkCopy = new SqlBulkCopy(connection);
// give it the name of the destination table WHICH MUST EXIST!
bulkCopy.DestinationTableName = "BulkTestTable";
// measure time needed
Stopwatch sw = new Stopwatch();
sw.Start();
// open connection, bulk insert, close connection
connection.Open();
bulkCopy.WriteToServer(tblToInsert);
connection.Close();
// stop time measurement
sw.Stop();
long milliseconds = sw.ElapsedMilliseconds;
}
On my system (PC, 32GB RAM, SQL Server 2014) I get those 60'000 rows inserted in 135 - 185 milliseconds.
Consider Table-Valued Parameters. They are an easy way to send a batch of data into a stored procedure that will then handle them on the SQL side, and they aren't restricted in most of the other approaches you will see are (insert limits, etc).
In the database create a custom Type that has the schema of your table.
CREATE TYPE dbo.TableType AS TABLE
( ID int )
Create a DataTable that matches your table schema (including column name and order).
DataTable newTableRecords = new DataTable();
// Insert your records, etc.
Create a stored procedure that receives a table parameter, and inserts the records from that parameter into your real table.
CREATE PROCEDURE usp_InsertTableRecords
(#tvpNewTableRecords dbo.TableType READONLY)
AS
BEGIN
INSERT INTO dbo.Table(ID)
SELECT tvp.ID FROM #tvpNewTableRecords AS tvp;
END
Call the procedure from your application code, passing in your data table as a parameter.
using (connection)
{
// Configure the SqlCommand and SqlParameter.
SqlCommand insertCommand = new SqlCommand(
"usp_InsertTableRecords", connection);
insertCommand.CommandType = CommandType.StoredProcedure;
SqlParameter tvpParam = insertCommand.Parameters.AddWithValue(
"#tvpNewTableRecords", newTableRecords);
tvpParam.SqlDbType = SqlDbType.Structured;
// Execute the command.
insertCommand.ExecuteNonQuery();
}
I've had really great performance at very large volumes with this approach, and it is nice because it allows everything to be set-based without any arbitrary insert limits like the INSERT INTO (Table) VALUES (1),(2),(3)... approach.
Based on the tutorial on SQL Temporary Tables, it should be OK to create a temp table by using SELECT * INTO #tempTable FROM tableA but it's throwing me SQLException when I trying to SELECT * FROM #tempTable saying that Invalid object name '#tempTable'. May I know what's the proper way of using a temp table in C#?
string sql = "SELECT * INTO ##tempTable FROM (SELECT * FROM tableA)";
using (var command = new SqlCommand(sql, connection))
{
string sqlNew = "SELECT * FROM ##tempTable";
using (var command2 = new SqlCommand(sqlNew, connection))
{
using (var reader = command2.ExecuteReader())
{
while (reader.Read())
{
Console.WriteLine(reader["column1"].ToString());
}
Console.ReadLine();
}
}
}
My Objective is tryint to using the data retrieved from sqlVar and insert them into a tempTable and perform some operation on it. Very much appreciated if there is some sample code on how to fit the code into the above code. Thank You.
But why you need temp table at SQL server side..
1) if you wish to perform operation on C# side just take data in DATASET instead of DATAREADER .. and
DataSet dataset = new DataSet();
using (SqlConnection conn = new SqlConnection(connString))
{
SqlDataAdapter adapter = new SqlDataAdapter();
adapter.SelectCommand = new SqlCommand("select * from tableA", conn);
conn.Open();
adapter.Fill(dataset);
conn.Close();
foreach (DataRow row in dataset.Tables[0]) // Loop over the rows.
{
// perform your operation
}
}
2) if you need to perform operation on SQL side then create a stored procedure at SQL server .. in the stored procedure create #table and use it ..
3) and you do not want to create DATASET then you can take data LIST and perform your operation on C# side
You are not executing the first command at all, so the SELECT INTO isn't executed, so the temporary table is not created, so you get an error about the table not existing.
The code should read:
string sql = "SELECT * INTO ##tempTable FROM (SELECT * FROM tableA)";
using (var command = new SqlCommand(sql, connection))
{
command.ExecuteNonQuery(); // <-- THIS
string sqlNew = "SELECT * FROM ##tempTable";
using (var command2 = new SqlCommand(sqlNew, connection))
{
using (var reader = command2.ExecuteReader())
{
while (reader.Read())
{
Console.WriteLine(reader["column1"].ToString());
}
Console.ReadLine();
}
}
}
1-SELECT * INTO # tempTable FROM tableA (local temp)or
2-SELECT * INTO ## tempTable FROM tableA (global temp)then
Local temp tables are only available to the current connection for the user; and they are automatically deleted when the user disconnects from instances. Local temporary table name is stared with hash ("#") sign.
Global Temp Table
Global Temporary tables name starts with a double hash ("##"). Once this table has been created by a connection, like a permanent table it is then available to any user by any connection. It can only be deleted once all connections have been closed.
Both, Temporary tables are stored inside the Temporary Folder of tempdb. Whenever we create a temporary table, it goes to Temporary folder of tempdb database.
temp table in SQL DB
Change your temp table from #tempTable to ##tempTable.
Using ## means a global temp table that stays around. You'll need to Drop it after you have completed your task.
If Exists(Select * from tempdb..sysobjects Where id = object_id('tempdb.dbo.#tempTable'))
DROP TABLE #tempTable
I think your answer is in the comment:
Temporary tables available during the session that creates them.
If you want to actualy get the data you have to perform a SELECT statement from this temporary table within the same scope.
One more thing:
I don't see you are executing the var command, you are missing this line:
string sql = "SELECT * INTO ##tempTable FROM (SELECT * FROM tableA)";
using (var command = new SqlCommand(sql, connection))
{
command.ExecuteNonQuery();// This line is missing..
string sqlNew = "SELECT * FROM ##tempTable";
using (var command2 = new SqlCommand(sqlNew, connection))
{
using (var reader = command2.ExecuteReader())
{
while (reader.Read())
{
Console.WriteLine(reader["column1"].ToString());
}
Console.ReadLine();
}
}
}
But missing the line isn't the reason why your implementation is wrong..
For SQL Server, I could do this
UPDATE [Table] SET b=2, c=3 WHERE a=1;
IF ##ROWCOUNT=0
INSERT INTO [Table] (a,b,c) VALUES (1,2,3)
How do you do a similar thing on MS Access (using OleDbConnection)?
Doing that I got
Characters found after end of SQL statement.
I don't think the Jet/ACE OleDB engine has an equivalent of T-SQL syntax for this kind of problem.
You should go for the long route of checking if record exist, then decide for INSERT or UPDATE.
However, being Access mainly a single/local user database system you should not have many problems doing something like this pseudocode:
using(OleDbConnection cn = new OleDbConnection(constring))
{
cn.Open();
using(OleDbCommand cmd = new OleDbCommand("select count(*) from table where pkID = ?", cn);
{
cmd.Parameters.AddWithValue("pk", myID);
int result = Convert.ToInt32(cmd.ExecuteScalar());
if(result == 0)
// do your insert command here
else
// do your update command here
}
}
Of course, as I have said, this doesn't take into account concurrency problems.
I am working on sql server monitoring product and i have database query that will fetch data regarding All Table details of all the Databases in SQL server.
For this i have two options.
Fire query on data base from code as select name from [master].sys.sysdatabases
Get the DB name of all the data base first then i will fire my main query on each DB
using "USE <fetched DB name>;"+"mainQuery";
Please check followin code for the same.
public DataTable GetResultsOfAllDB(string query)
{
SqlConnection con = new SqlConnection(_ConnectionString);
string locleQuery = "select name from [master].sys.sysdatabases";
DataTable dtResult = new DataTable("Result");
SqlCommand cmdData = new SqlCommand(locleQuery, con);
cmdData.CommandTimeout = 0;
SqlDataAdapter adapter = new SqlDataAdapter(cmdData);
DataTable dtDataBases = new DataTable("DataBase");
adapter.Fill(dtDataBases);
foreach (DataRow drDB in dtDataBases.Rows)
{
if (dtResult.Rows.Count >= 15000)
break;
locleQuery = " Use [" + Convert.ToString(drDB[0]) + "]; " + query;
cmdData = new SqlCommand(locleQuery, con);
adapter = new SqlDataAdapter(cmdData);
DataTable dtTemp = new DataTable();
adapter.Fill(dtTemp);
dtResult.Merge(dtTemp);
}
return dtResult;
}
I will use sys store procedure i.e.EXEC sp_MSforeachdb and fetched data will be stored store data in table datatype select from temptable; Drop Table temptable.
Check following query for the same
Declare #TableDetail table
(
field1 varchar(500),
field2 int,
field3 varchar(500),
field4 varchar(500),
field5 decimal(18,2),
field6 decimal(18,2)
)
INSERT #TableDetail EXEC sp_MSforeachdb 'USE [?]; QYERY/COMMAND FOR ALL DATABASE'
Select
field1,field2 ,field3 ,field4 ,field5,field6 FROM #TableDetail
Note : In second option query takes time because if number of database and number of table are huge then this will wait until all database get finish.
Now my question is which is the good option from above two options and why? or any other solution for the same.
Thanks in advance.
One key difference is the second option blocks until everything is done. All of the work is done sql server side. That has the issue of not being able to apply feedback to the user as it runs and it can potentially time out and not be resiliant to network blips. This option can be used as a pure sql script (some sql admins like that) where the first needs a program.
In the first example, the client is doing iterative more granular tasks where you can supply feedback to the user. You can also retry in the face of network blips without redoing all of the work. In the first example, you can also use SqlConnectionBuild instead of USE concatentation.
If performance is a concern, you could also potentially parallelize the first one with some locking around adapter.Fill
Both suck - they are both serial.
Use the first, get rid of the ridiculous objects (DataSet) and use TASKS to parallelize X databases at the same time. X determined by trying ut how much load the server can handle.
Finished.
If your queries are simple enough you can try to generate single script instead of execute queries in each DB one by one:
select 'DB1' as DB, Field1, Field2, ...
from [DB1]..[TableOrViewName]
union all
select 'DB2' as DB, Field1, Field2, ...
from [DB2]..[TableOrViewName]
union all
...
Everything is looking fine. I just want to add Using statements for IDisposable objects
public DataTable GetResultsOfAllDB(string query)
{
using (SqlConnection con = new SqlConnection(_ConnectionString))
{
string locleQuery = "select name from [master].sys.sysdatabases";
DataTable dtResult = new DataTable("Result");
using (SqlCommand cmdData = new SqlCommand(locleQuery, con))
{
cmdData.CommandTimeout = 0;
using (SqlDataAdapter adapter = new SqlDataAdapter(cmdData))
{
using (DataTable dtDataBases = new DataTable("DataBase"))
{
adapter.Fill(dtDataBases);
foreach (DataRow drDB in dtDataBases.Rows)
{
if (dtResult.Rows.Count >= 15000)
break;
locleQuery = " Use [" + Convert.ToString(drDB[0]) + "]; " + query;
cmdData = new SqlCommand(locleQuery, con);
adapter = new SqlDataAdapter(cmdData);
using (DataTable dtTemp = new DataTable())
{
adapter.Fill(dtTemp);
dtResult.Merge(dtTemp);
}
}
return dtResult;
}
}
}
}
}