Develop Reference

Inserting and updating records to database at the same time [duplicate]

I'm looking to do something simulair toward here: How do I insert multiple rows WITHOUT repeating the "INSERT INTO dbo.Blah" part of the statement?
except that in addition towards doing this in one query (faster then several dozen) I also want to do this parameterized as the input comes from the web.
Currently I have
foreach(string data in Scraper){
SqlConnection conn = new SqlConnection(WebConfigurationManager.AppSettings["ConnectionInfo"].ToString());
string query = "INSERT INTO DATABASE('web',#data)";
SqlCommand sqlCommand= new SqlCommand(query, conn);
sqlCommand.Parameters.AddWithValue("#data", data);
Command.executeNonQuery();
conn.close();
}
Which is a bit slugish (note the real example has a lot more colums but that would make things more confusing).

Since you are using c# and sql server 2008, you can use a table valued parameter to insert multiple rows to your database.
Here is a short description on how to do this:
First, you need to create a user defined table type:
CREATE TYPE MyTableType AS TABLE
(
Col1 int,
Col2 varchar(20)
)
GO
Then, you need to create a stored procedure that will accept this table type as a parameter
CREATE PROCEDURE MyProcedure
(
#MyTable dbo.MyTableType READONLY -- NOTE: table valued parameters must be Readonly!
)
AS
INSERT INTO MyTable (Col1, Col2)
SELECT Col1, Col2
FROM #MyTable
GO
Finally, execute this stored procedure from your c# code:
DataTable dt = new DataTable();
dt.Columns.Add("Col1", typeof(int));
dt.Columns.Add("Col2", typeof(string));
// Fill your data table here
using (var con = new SqlConnection("ConnectionString"))
{
using(var cmd = new SqlCommand("MyProcedure", con))
{
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add("#MyTable", SqlDbType.Structured).Value = dt;
con.Open();
cmd.ExecuteNonQuery();
}
}

You can make use of the SQL syntax:
INSERT INTO YOUR_TABLE (dataColumn) VALUES (data1),(data2),(data3)
So loop over your rows you wanna insert and append ",(datax)" to your query and also add the corresponding parameter.
Perhaps it helps.

Select all columns except the first column for any given SQL Server table

I have this code in C#, but I need it to select all columns EXCEPT the first column of the table (the identity column), so that when I insert the data into an identical table in a different database, the destination database assigns its own identity column values:
SqlCommand commandSourceData = new SqlCommand($"SELECT * FROM dbo.{tableName};", sourceConnection);
SqlDataReader reader = commandSourceData.ExecuteReader();
Is there a way to do this?

If you want a generic solution for every column in your database you can use this kind of code
public string GetColumnsWithoutIdentity(string tableName, SqlConnection con)
{
SqlDataAdapter da = new SqlDataAdapter($"SELECT * FROM dbo.{tableName} where 1=0", con);
DataTable dt = new DataTable();
da.FillSchema(dt, SchemaType.Source);
var cols = dt.Columns.Cast<DataColumn>().Where(x => !x.AutoIncrement).Select(x => x.ColumnName);
return string.Join(",", cols);
}
Now you can use the returned string to build an Sql statement without the autoincrement column.
Notice that this code is vulnerable to Sql Injection. You should be absolutely sure that the tableName parameter used to build the first query is not typed directly by your user. Let it choose from a whitelist (readonly) of predefined tables (and also this is not 100% safe)
Another drawback is the fact that you need to hit the database two times. Once to get the schema with the info about the AutoIncrement column and one to fill the datatable after that.

INSERT into an identity column not allowed on table variables

I am importing data from large excel sheet and storing it in a stateTable. Now I have to push this data into a database table. The table does have an identity column(1,1).
I have created a similar table type in DB and a procedure to take a parameter as table type to insert in the particular table. I have also set ON the identity insert.
My code is:
using (SqlCommand command = new SqlCommand("InsertStateTable") {
CommandType = CommandType.StoredProcedure})
{
SqlParameter param = command.Parameters.AddWithValue("#statetable", dt);
param.TypeName = "StateTable";
param.SqlDbType = SqlDbType.Structured;
command.Connection = con;
con.Open();
command.ExecuteNonQuery();
con.Close();
}
But the error that arises is
"INSERT into an identity column not allowed on table variables."
I have gone thru many sites but no specific reason is given.....
Thanks in advance.

The error is fairly clear: you are not allowed to do what you are trying to do. Basically, you are going to have to find a design that is not dependent on inserting the identity value into the table-variable / table-valued-parameter. My advice would be to create a separate table-variable (unrelated to the table-valued-parameter) which has the same data, but which does not have the IDENTITY column, so...
declare #foo table (id int not null, name nvarchar(200) not null /* etc */)
insert #foo (id, name /* etc */)
select id, name /* etc */ from #statetable
at which point #foo has the original data, but does not have an identity column - you can then do whatever you want with #foo.
Without seeing what you are doing with your identity insert, it is hard to comment much further.

Insert entire DataTable into database at once instead of row by row?

I have a DataTable and need the entire thing pushed to a Database table.
I can get it all in there with a foreach and inserting each row at a time. This goes very slow though since there are a few thousand rows.
Is there any way to do the entire datatable at once that might be faster?
The DataTable has less columns than the SQL table. the rest of them should be left NULL.

I discovered SqlBulkCopy is an easy way to do this, and does not require a stored procedure to be written in SQL Server.
Here is an example of how I implemented it:
// take note of SqlBulkCopyOptions.KeepIdentity , you may or may not want to use this for your situation.
using (var bulkCopy = new SqlBulkCopy(_connection.ConnectionString, SqlBulkCopyOptions.KeepIdentity))
{
// my DataTable column names match my SQL Column names, so I simply made this loop. However if your column names don't match, just pass in which datatable name matches the SQL column name in Column Mappings
foreach (DataColumn col in table.Columns)
{
bulkCopy.ColumnMappings.Add(col.ColumnName, col.ColumnName);
}
bulkCopy.BulkCopyTimeout = 600;
bulkCopy.DestinationTableName = destinationTableName;
bulkCopy.WriteToServer(table);
}

Since you have a DataTable already, and since I am assuming you are using SQL Server 2008 or better, this is probably the most straightforward way. First, in your database, create the following two objects:
CREATE TYPE dbo.MyDataTable -- you can be more speciifc here
AS TABLE
(
col1 INT,
col2 DATETIME
-- etc etc. The columns you have in your data table.
);
GO
CREATE PROCEDURE dbo.InsertMyDataTable
#dt AS dbo.MyDataTable READONLY
AS
BEGIN
SET NOCOUNT ON;
INSERT dbo.RealTable(column list) SELECT column list FROM #dt;
END
GO
Now in your C# code:
DataTable tvp = new DataTable();
// define / populate DataTable
using (connectionObject)
{
SqlCommand cmd = new SqlCommand("dbo.InsertMyDataTable", connectionObject);
cmd.CommandType = CommandType.StoredProcedure;
SqlParameter tvparam = cmd.Parameters.AddWithValue("#dt", tvp);
tvparam.SqlDbType = SqlDbType.Structured;
cmd.ExecuteNonQuery();
}
If you had given more specific details in your question, I would have given a more specific answer.

Consider this approach, you don't need a for loop:
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection))
{
bulkCopy.DestinationTableName =
"dbo.BulkCopyDemoMatchingColumns";
try
{
// Write from the source to the destination.
bulkCopy.WriteToServer(ExistingSqlTableName);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}

If can deviate a little from the straight path of DataTable -> SQL table, it can also be done via a list of objects:
1) DataTable -> Generic list of objects
public static DataTable ConvertTo<T>(IList<T> list)
{
DataTable table = CreateTable<T>();
Type entityType = typeof(T);
PropertyDescriptorCollection properties = TypeDescriptor.GetProperties(entityType);
foreach (T item in list)
{
DataRow row = table.NewRow();
foreach (PropertyDescriptor prop in properties)
{
row[prop.Name] = prop.GetValue(item);
}
table.Rows.Add(row);
}
return table;
}
Source and more details can be found here. Missing properties will remain to their default values (0 for ints, null for reference types etc.)
2) Push the objects into the database
One way is to use EntityFramework.BulkInsert extension. An EF datacontext is required, though.
It generates the BULK INSERT command required for fast insert (user defined table type solution is much slower than this).
Although not the straight method, it helps constructing a base of working with list of objects instead of DataTables which seems to be much more memory efficient.

You can do this with a table value parameters.
Have a look at the following article:
http://www.codeproject.com/Articles/39161/C-and-Table-Value-Parameters

I would prefer user defined data type : it is super fast.
Step 1 : Create User Defined Table in Sql Server DB
CREATE TYPE [dbo].[udtProduct] AS TABLE(
[ProductID] [int] NULL,
[ProductName] [varchar](50) NULL,
[ProductCode] [varchar](10) NULL
)
GO
Step 2 : Create Stored Procedure with User Defined Type
CREATE PROCEDURE ProductBulkInsertion
#product udtProduct readonly
AS
BEGIN
INSERT INTO Product
(ProductID,ProductName,ProductCode)
SELECT ProductID,ProductName,ProductCode
FROM #product
END
Step 3 : Execute Stored Procedure from c#
SqlCommand sqlcmd = new SqlCommand("ProductBulkInsertion", sqlcon);
sqlcmd.CommandType = CommandType.StoredProcedure;
sqlcmd.Parameters.AddWithValue("#product", productTable);
sqlcmd.ExecuteNonQuery();
Possible Issue : Alter User Defined Table
Actually there is no sql server command to alter user defined type
But in management studio you can achieve this from following steps
1.generate script for the type.(in new query window or as a file)
2.delete user defied table.
3.modify the create script and then execute.

.NET / SQLite. Inserting parent/child record; how do I get the foreign key?

I use VS 2008 (C#) and SQLite via ADO.NET 2.0 Provider (SourceForce project).
The database used by application contains an "employees" (parent) and "employmentHistory" (children) datatables. "eploymentHistory" datatables is supposed to contain all the working contracts for any given employee from day 1 (e.g. a promotion will generate a new contract).
Unfortunately SQLite doesn't support foreign keys, so I need to take care of data consistency myself.
The employmentHistory database contains "id INTEGER PRIMARY KEY NOT NULL, employeeID INTEGER..." columns
So obviously, employeeID will refer to employess.ID, the primary key in the employees table. It identifies WHOSE contract it is.
I'm writing a method for adding a new employee. Everybody will have at least one contract (the first one), so adding a new employee is connected with adding the contract.
Here's the method:
internal static void AddNewEmployee(Employee e, ContractChange c)
{
SQLiteCommandBuilder builder = new SQLiteCommandBuilder(adapter);
var insert = new SQLiteCommand(connection);
insert.CommandText = #"INSERT INTO employees VALUES ( null, #firstName, #lastName, #phone, #mobile, null, null, ";
insert.CommandText+= #"#birthDate, #sex, #address, #nin, null, null); ";
insert.Parameters.AddWithValue("firstName", e.info.name);
insert.Parameters.AddWithValue("lastName", e.info.surname);
insert.Parameters.AddWithValue("phone", e.info.phone);
insert.Parameters.AddWithValue("mobile", e.info.mobile);
insert.Parameters.AddWithValue("birthDate", e.info.DOB);
insert.Parameters.AddWithValue("sex", e.info.sex);
insert.Parameters.AddWithValue("address", e.info.address);
insert.Parameters.AddWithValue("nin", e.info.NIN);
insert.CommandText += #"INSERT INTO employmentHistory VALUES ( null, null, #startDate, 'true', #position, #contractHrs, ";
insert.CommandText += #"#holidayAllowance, #comments, null); ";
insert.Parameters.AddWithValue("startDate", c.date);
insert.Parameters.AddWithValue("position", (int)c.role);
insert.Parameters.AddWithValue("contractHrs", (int)c.contractHrs);
insert.Parameters.AddWithValue("holidayAllowance", c.holidayAllowance);
insert.Parameters.AddWithValue("comments", c.comments);
DataTable employees = dataset.Tables[0];
var datarowEmp = employees.NewRow();
datarowEmp = e.ToDataRow(datarowEmp);
employees.Rows.Add(datarowEmp);
DataTable employmentHistory = dataset.Tables[1];
var datarowContract = employmentHistory.NewRow();
datarowContract = c.ToDataRow(datarowContract);
employmentHistory.Rows.Add(datarowContract);
adapter.UpdateCommand = insert;
adapter.InsertCommand = insert;
adapter.SelectCommand = new SQLiteCommand("SELECT * FROM employees; SELECT * FROM employmentHistory;", connection);
adapter.Update(dataset);
}
So I thought I would set the employeeID "manually" with a quick update. But - how do I know what ID was assigned to the employee I just added???
Amazingly (to me), even after SQLiteDataAdapter updates the data (I set a breakpoint at the very end of the method), this information is still not in the dataset. The field is still null. And yet SQLite does attribute the unique number at some point, as no exception is thrown and the employees are indeed getting primary keys (I saw the database in SQLite Database Browser).
So I can't do it automatically, and I can't do it manually - how am I supposed to enforce database consistency? :(

See last_insert_rowid
You can implement it in a single line:
cmd.CommandText = "INSERT INTO .. VALUES(..);SELECT last_insert_rowid() AS [ID]";
strId = cmd.ExecuteScalar().ToString()
Update:
You might also want to enclose the two inserts within a single transaction, so if one of them fails, you can rollback the entire insert, and prevent haveing corrupted data in your database.

select last_insert_rowid();