I have a DataTable and need the entire thing pushed to a Database table.
I can get it all in there with a foreach and inserting each row at a time. This goes very slow though since there are a few thousand rows.
Is there any way to do the entire datatable at once that might be faster?
The DataTable has less columns than the SQL table. the rest of them should be left NULL.
I discovered SqlBulkCopy is an easy way to do this, and does not require a stored procedure to be written in SQL Server.
Here is an example of how I implemented it:
// take note of SqlBulkCopyOptions.KeepIdentity , you may or may not want to use this for your situation.
using (var bulkCopy = new SqlBulkCopy(_connection.ConnectionString, SqlBulkCopyOptions.KeepIdentity))
{
// my DataTable column names match my SQL Column names, so I simply made this loop. However if your column names don't match, just pass in which datatable name matches the SQL column name in Column Mappings
foreach (DataColumn col in table.Columns)
{
bulkCopy.ColumnMappings.Add(col.ColumnName, col.ColumnName);
}
bulkCopy.BulkCopyTimeout = 600;
bulkCopy.DestinationTableName = destinationTableName;
bulkCopy.WriteToServer(table);
}
Since you have a DataTable already, and since I am assuming you are using SQL Server 2008 or better, this is probably the most straightforward way. First, in your database, create the following two objects:
CREATE TYPE dbo.MyDataTable -- you can be more speciifc here
AS TABLE
(
col1 INT,
col2 DATETIME
-- etc etc. The columns you have in your data table.
);
GO
CREATE PROCEDURE dbo.InsertMyDataTable
#dt AS dbo.MyDataTable READONLY
AS
BEGIN
SET NOCOUNT ON;
INSERT dbo.RealTable(column list) SELECT column list FROM #dt;
END
GO
Now in your C# code:
DataTable tvp = new DataTable();
// define / populate DataTable
using (connectionObject)
{
SqlCommand cmd = new SqlCommand("dbo.InsertMyDataTable", connectionObject);
cmd.CommandType = CommandType.StoredProcedure;
SqlParameter tvparam = cmd.Parameters.AddWithValue("#dt", tvp);
tvparam.SqlDbType = SqlDbType.Structured;
cmd.ExecuteNonQuery();
}
If you had given more specific details in your question, I would have given a more specific answer.
Consider this approach, you don't need a for loop:
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection))
{
bulkCopy.DestinationTableName =
"dbo.BulkCopyDemoMatchingColumns";
try
{
// Write from the source to the destination.
bulkCopy.WriteToServer(ExistingSqlTableName);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
If can deviate a little from the straight path of DataTable -> SQL table, it can also be done via a list of objects:
1) DataTable -> Generic list of objects
public static DataTable ConvertTo<T>(IList<T> list)
{
DataTable table = CreateTable<T>();
Type entityType = typeof(T);
PropertyDescriptorCollection properties = TypeDescriptor.GetProperties(entityType);
foreach (T item in list)
{
DataRow row = table.NewRow();
foreach (PropertyDescriptor prop in properties)
{
row[prop.Name] = prop.GetValue(item);
}
table.Rows.Add(row);
}
return table;
}
Source and more details can be found here. Missing properties will remain to their default values (0 for ints, null for reference types etc.)
2) Push the objects into the database
One way is to use EntityFramework.BulkInsert extension. An EF datacontext is required, though.
It generates the BULK INSERT command required for fast insert (user defined table type solution is much slower than this).
Although not the straight method, it helps constructing a base of working with list of objects instead of DataTables which seems to be much more memory efficient.
You can do this with a table value parameters.
Have a look at the following article:
http://www.codeproject.com/Articles/39161/C-and-Table-Value-Parameters
I would prefer user defined data type : it is super fast.
Step 1 : Create User Defined Table in Sql Server DB
CREATE TYPE [dbo].[udtProduct] AS TABLE(
[ProductID] [int] NULL,
[ProductName] [varchar](50) NULL,
[ProductCode] [varchar](10) NULL
)
GO
Step 2 : Create Stored Procedure with User Defined Type
CREATE PROCEDURE ProductBulkInsertion
#product udtProduct readonly
AS
BEGIN
INSERT INTO Product
(ProductID,ProductName,ProductCode)
SELECT ProductID,ProductName,ProductCode
FROM #product
END
Step 3 : Execute Stored Procedure from c#
SqlCommand sqlcmd = new SqlCommand("ProductBulkInsertion", sqlcon);
sqlcmd.CommandType = CommandType.StoredProcedure;
sqlcmd.Parameters.AddWithValue("#product", productTable);
sqlcmd.ExecuteNonQuery();
Possible Issue : Alter User Defined Table
Actually there is no sql server command to alter user defined type
But in management studio you can achieve this from following steps
1.generate script for the type.(in new query window or as a file)
2.delete user defied table.
3.modify the create script and then execute.
Related
I'm looking to do something simulair toward here: How do I insert multiple rows WITHOUT repeating the "INSERT INTO dbo.Blah" part of the statement?
except that in addition towards doing this in one query (faster then several dozen) I also want to do this parameterized as the input comes from the web.
Currently I have
foreach(string data in Scraper){
SqlConnection conn = new SqlConnection(WebConfigurationManager.AppSettings["ConnectionInfo"].ToString());
string query = "INSERT INTO DATABASE('web',#data)";
SqlCommand sqlCommand= new SqlCommand(query, conn);
sqlCommand.Parameters.AddWithValue("#data", data);
Command.executeNonQuery();
conn.close();
}
Which is a bit slugish (note the real example has a lot more colums but that would make things more confusing).
Since you are using c# and sql server 2008, you can use a table valued parameter to insert multiple rows to your database.
Here is a short description on how to do this:
First, you need to create a user defined table type:
CREATE TYPE MyTableType AS TABLE
(
Col1 int,
Col2 varchar(20)
)
GO
Then, you need to create a stored procedure that will accept this table type as a parameter
CREATE PROCEDURE MyProcedure
(
#MyTable dbo.MyTableType READONLY -- NOTE: table valued parameters must be Readonly!
)
AS
INSERT INTO MyTable (Col1, Col2)
SELECT Col1, Col2
FROM #MyTable
GO
Finally, execute this stored procedure from your c# code:
DataTable dt = new DataTable();
dt.Columns.Add("Col1", typeof(int));
dt.Columns.Add("Col2", typeof(string));
// Fill your data table here
using (var con = new SqlConnection("ConnectionString"))
{
using(var cmd = new SqlCommand("MyProcedure", con))
{
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add("#MyTable", SqlDbType.Structured).Value = dt;
con.Open();
cmd.ExecuteNonQuery();
}
}
You can make use of the SQL syntax:
INSERT INTO YOUR_TABLE (dataColumn) VALUES (data1),(data2),(data3)
So loop over your rows you wanna insert and append ",(datax)" to your query and also add the corresponding parameter.
Perhaps it helps.
I have this user table type in SQL Server:
CREATE TYPE [dbo].[ListNew] AS TABLE
(
[Id] [int] NOT NULL,
PRIMARY KEY CLUSTERED ([Id] ASC) WITH (IGNORE_DUP_KEY = OFF)
)
GO
And use this type in stored procedure parameter:
....
(#lstNew ListNew READONLY,
#UserName nvarchar(128))
AS
....
And using this stored procedure in ASP.NET MVC with this code:
List<int> lstNew = MyList.Select(o => o.Key).ToList();
List<XXXView> lstView = db.Database.SqlQuery<XXXView>("MyStoredProcedure #lstNew,#UserName",
new SqlParameter("#lstNew", lstNew),
new SqlParameter("#UserName", userName)).ToList();
but it's not working and get this error:
No mapping exists from object type System.Collections.Generic.List`1[[System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]] to a known managed provider native type.
I try without ListNew and used only username, it's working
Edit:
I use this code:
myParameter.SqlDbType = SqlDbType.Udt;
myParameter.UdtTypeName = "ListNew";
But I get the same warning
This is a solved problem and properly documented in - cough - the documentation.
YOu will need to define the table on the server side and then can pass in a table valued parameter.
https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql/table-valued-parameters
This runs down to using the SqlDbType Structured.
// Configure the command and parameter.
SqlCommand insertCommand = new SqlCommand(sqlInsert, connection);
SqlParameter tvpParam = insertCommand.Parameters.AddWithValue("#tvpNewCategories", addedCategories);
tvpParam.SqlDbType = SqlDbType.Structured;
tvpParam.TypeName = "dbo.CategoryTableType";
You CAN use a DataTable, but then you introduce the most overhead approach possible as object model - or you just use...
https://forums.asp.net/t/1845039.aspx?Table+Value+Parameter+Use+With+C+without+using+DataTable
Basically you transform your data into SqlDataRecords and pass them in. Needs some metadata - but generally this can be generalized and fits in below a page of code. The link has the code (which I can not copy here due to - well - it not being MY code).
I always use XML to pass these type of DATA to sqlserver sproc.
When you pass your XML to sproc you can use something like this to have it as a table in your sproc:
(commented lines are my XML's structure. manipulate it for your own use.)
EXEC sp_xml_preparedocument #XmlHandle output,#FieldPermissionAccessXML
--'<FieldPermissions>
--<FieldPermission FieldName="" RoleID="1" Access="1" />
--<FieldPermission FieldName="" RoleID="2" Access="1" />'
--select * from JMP_FieldPermissions
INSERT INTO #FieldPermissionsTable
SELECT *--ID,Value, Value2, Navi_User
FROM OPENXML (#XmlHandle, './Row_ID/Elements')
WITH (TE_ID VARCHAR(MAX) '#ID',
Value VARCHAR(MAX) '#Value',
Value2 VARCHAR(MAX) '#Value2',
NAVI_USER VARCHAR(MAX) '#Navi_User'
)
Create the DataTable variable in c#, put the data in the DataTable, pass it to sp:
List<int> lstNew = MyList.Select(o => o.Key).ToList();
DataTable lstNewTable = new ListNew();
foreach (var id in lstNew )
{
lstNewTable.Rows.Add(id);
}
List<XXXView> lstView = db.Database.SqlQuery<XXXView>("MyStoredProcedure #lstNew,#UserName",new SqlParameter("#lstNew", lstNewTable),
new SqlParameter("#UserName", userName)).ToList();
Several answers were close but none gave a full working model. #Alison Niu just needs to add a column & name to populate the DataTable. #Saurabh Gadani the reflection is very flexible but the Props are Null. Also the cmd.Parameters needs some special values set for Sql to find the table definition. I am grateful for these answers that got me closer to a solution.
Populate a DataTable from a List of Integers
private DataTable ListInt_ToTable(List<int> tableThese)
{
DataTable lstNewTable = new DataTable();
//Columns
lstNewTable.Columns.Add("ID", typeof(int));
//Values
foreach (var id in tableThese)
{
lstNewTable.Rows.Add(id);
}
return lstNewTable;
}
The TYPE defines the table layout for SQL to receive the parameter. It can be written within the Stored Procedure but this is how I prefer to keep it in the application that built the data:
public static string SqlCmd_spStackOverflow_Loaded_ListIDs = #"
IF TYPE_ID(N'IdLoadedTableType') IS NULL
Begin
CREATE TYPE dbo.IdLoadedTableType
AS TABLE ( ID INT );
End
EXEC dbo.spStackOverflow_Loaded_ListIDs
#TblIds
";
Sending the Sql to run the Stored Procedure. Notice the parameters **
DataTable mappingTbl = ListInt_ToTable(mappingData);
using (SqlConnection sqlConnection = new SqlConnection(ConnectionString))
{
try
{
DataTable results = new DataTable();
SqlDataAdapter sqlDataAdapter = new SqlDataAdapter(SqlCmd_spStackOverflow_Loaded_ListIDs, sqlConnection);
//apply parameters
//NOT VIA sqlDataAdapter.SelectCommand.Parameters.AddWithValue(...);
var specialParm = new SqlParameter(); // **
specialParm.ParameterName = "#TblIds"; // acts like a Declare in SQL
specialParm.Value = (object)mappingTbl; // sets the data valaes
specialParm.SqlDbType = SqlDbType.Structured; // unique to User-Defined Table parameters
specialParm.TypeName = "dbo.IdLoadedTableType"; // refers to created type
sqlDataAdapter.SelectCommand.Parameters.Add(specialParm); // done
sqlDataAdapter.Fill(results);
return results;
}
catch (Exception ex)
{
throw new Exception("An error occurred while executing a SQL Read statement with Parameters. SQL Statement: " + SqlCmd_spStackOverflow_Loaded_ListIDs + " Exception: " + ex.Message);
}
}
This model is working in my solution, except I use long in C# & BigInt in sql. Please improve options as you find something.
You can not pass any generic type list as SP parameter, must have to pass it as DataTable instead of List.
Here is an eaxample:
DataTable mappingTbl = ListToDataTable(mappingData);
SqlConnection con = new SqlConnection(conStr);
con.Open();
SqlCommand cmd = new SqlCommand(StoredProcedure.GetSavedFormList, con);
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.AddWithValue("#UDT_EBFromMappingTable", mappingTbl);
public DataTable ListToDataTable<T>(List<T> items)
{
DataTable dataTable = new DataTable(typeof(T).Name);
//Get all the properties
PropertyInfo[] Props = typeof(T).GetProperties(BindingFlags.Public | BindingFlags.Instance);
foreach (PropertyInfo prop in Props)
{
//Defining type of data column gives proper data table
var type = (prop.PropertyType.IsGenericType && prop.PropertyType.GetGenericTypeDefinition() == typeof(Nullable<>) ? Nullable.GetUnderlyingType(prop.PropertyType) : prop.PropertyType);
//Setting column names as Property names
dataTable.Columns.Add(prop.Name, type);
}
foreach (T item in items)
{
var values = new object[Props.Length];
for (int i = 0; i < Props.Length; i++)
{
//inserting property values to datatable rows
values[i] = Props[i].GetValue(item, null);
}
dataTable.Rows.Add(values);
}
//put a breakpoint here and check datatable
return dataTable;
}
May this helps you. :)
I have this code in C#, but I need it to select all columns EXCEPT the first column of the table (the identity column), so that when I insert the data into an identical table in a different database, the destination database assigns its own identity column values:
SqlCommand commandSourceData = new SqlCommand($"SELECT * FROM dbo.{tableName};", sourceConnection);
SqlDataReader reader = commandSourceData.ExecuteReader();
Is there a way to do this?
If you want a generic solution for every column in your database you can use this kind of code
public string GetColumnsWithoutIdentity(string tableName, SqlConnection con)
{
SqlDataAdapter da = new SqlDataAdapter($"SELECT * FROM dbo.{tableName} where 1=0", con);
DataTable dt = new DataTable();
da.FillSchema(dt, SchemaType.Source);
var cols = dt.Columns.Cast<DataColumn>().Where(x => !x.AutoIncrement).Select(x => x.ColumnName);
return string.Join(",", cols);
}
Now you can use the returned string to build an Sql statement without the autoincrement column.
Notice that this code is vulnerable to Sql Injection. You should be absolutely sure that the tableName parameter used to build the first query is not typed directly by your user. Let it choose from a whitelist (readonly) of predefined tables (and also this is not 100% safe)
Another drawback is the fact that you need to hit the database two times. Once to get the schema with the info about the AutoIncrement column and one to fill the datatable after that.
i would like to know what is the standard/best way of doing the following:
i have a form web app in asp.net and using C#
the user will enter data into the form and click INSERT and it will insert data into 4 different tables.
the fields are:
primarykey, animal, street, country
the form allows for multiple animals, multiple streets and multiple countries per primarykey. so when i have data like this:
[1],[rhino,cat,dog],[luigi st, paul st], [russia,israel]
i need it inserted into tables like this:
table1:
1,rhino
1,cat
1,dog
table2:
1,luigi st
1, paul st
table3:
1,russia
1,israel
questions
I'm at a total loss on how to do this. if i just had one table and one set of data per primary key i would just use the InsertQuery and do it this way, but since it is multiple tables i don't know how to do this??
what control(s) should i use in order to allow user to input multiple values? currently i am just using textboxes and thinking of separating the entries by semi colons, but that's probably not the right way.
I wanted to recommend that you take advantage of the new multirow insert statement in SQL 2008 so that you can just pass a sql statement like this:
INSERT INTO table1(id,animal_name) values (1,cat),(1,dog),(1,horse)...
To your SqlCommand but I don't know how to build a statement like that w/o risking being victim of a SQL Injection Attack.
Another alternative is to define data table types in your sql database:
And then construct a DataTable in C# that matches your datatable type definition:
DataTable t = new DataTable();
t.Columns.Add("id");
t.Columns.Add("animal_name");
foreach(var element in your animals_list)
{
DaraRow r = t.NewRow();
r.ItemArray = new object[] { element.id, element.animal_name };
t.Rows.Add(r);
}
// Assumes connection is an open SqlConnection.
using (connection)
{
// Define the INSERT-SELECT statement.
string sqlInsert = "INSERT INTO dbo.table1 (id, animal_name) SELECT nc.id, nc.animal_name FROM #animals AS nc;"
// Configure the command and parameter.
SqlCommand insertCommand = new SqlCommand(sqlInsert, connection);
SqlParameter tvpParam = insertCommand.Parameters.AddWithValue("#animals", t);
tvpParam.SqlDbType = SqlDbType.Structured;
tvpParam.TypeName = "dbo.AnimalTable";
// Execute the command.
insertCommand.ExecuteNonQuery();
}
Read more here.
Or if you are familiar with Stored Procedures, same as previous suggestion but having the stored procedure receive the DataTable t as parameter.
If none of the above work for you, create a SqlTranscation from the Connection object and iterate through each row of each data set inserting the record in the appropriate table and finally commit the transaction. Example here.
Use Checkboxes on the front end. Have a service/repository to save the user data. Something like the following:
public void UpdateUserAnimals(Guid userId, string[] animals)
{
using (SqlConnection conn = new SqlConnection("connectionstring..."))
{
using (SqlCommand cmd = new SqlCommand("Insert Into UserAnimals(UserId, Animals) values (#UserId, #Animal)"))
{
conn.Open();
cmd.Parameters.AddWithValue("#UserId", userId);
foreach(string animal in animals)
{
cmd.Parameters.AddWithValue("#Animal", animal);
cmd.ExecuteNonQuery();
}
}
}
}
There are more complex solutions, but this is a simple one.
I looked at the SQLTeam website but now am having a new problem cause I have an IDENTITY column and their example does not.
I have SQL Server 2008 and VS 2008. I am trying to execute the InsertPIF SP using C# and a table UDT, but am getting an exception. I have looked at a SQLTeam website
example with a Table UDT, but their example doesn't have an identity column like mine does.
However, their example table doesn't have an identity column like mine.
What I am trying to do should be simple. I have tried many different variations of this, but essentially:
ALTER PROCEDURE [dbo].[InsertPIF] #InputFileParam parseInputFile READONLY
AS
INSERT dbo.ParentTable SELECT strRow FROM #InputFileParam
GO
where
CREATE TABLE [dbo].[ParentTable](
[ParentID] [int] IDENTITY(1,1) NOT NULL,
[strInput] [varchar](8000) NULL,
PRIMARY KEY NONCLUSTERED
(...)
and type:
CREATE TYPE [dbo].[parseInputFile] AS TABLE(
[NumCols] [int] IDENTITY(1,1) NOT NULL,
[strRow] [varchar](500) NOT NULL,
PRIMARY KEY CLUSTERED
(...)
What I am trying to do is to execute the above SP to insert new records into the ParentTable. Example:
INSERT INTO ParentTable(strInput) VALUES ('A3|BB|C|DDD')
INSERT INTO ParentTable(strInput) VALUES ('A4|GOB|BLDY|GOOK')
INSERT INTO ParentTable(strInput) VALUES ('A5|Hello|My|Darling')
Here is more of my C# code prior to the AppendData section:
DataTable dt = new DataTable();
String[] header = myStringArray[0].Split('|');
DataRow dr = dt.NewRow();
DataColumn dc = new DataColumn();
dc.DataType = typeof(Int32);
dc.ColumnName = “NumCols”;
dc.AutoIncrement=true;
dt.Columns.Add(dc);
DataColumn dc2 = new DataColumn();
dc2.DataType = typeof(string);
dc2.ColumnName = “strRow”;
dt.Columns.Add(dc2);
dr["NumCols"] = 1;
dr["strRow"] = "AA|BBB|CC|DD|E";
dt.Rows.Add(dr);
...
SqlConnection conn = new SqlConnection(connString);
SqlCommand cmd = conn.CreateCommand();
cmd.CommandText = "dbo.InsertPIF";
cmd.CommandType = CommandType.StoredProcedure;
cmd.Connection = conn;
SqlParameter param = cmd.Parameters.AddWithValue("#InputFileParam", SqlDbType.Structured);
param.Value = dt;
conn.Open();
RowsAffected = cmd.ExecuteNonQuery();
I set identity to be turned on for the ParentTable, but it seems like I have to execute this each session, so I'm not sure how to remove identity requirement or else to make identity remain on. Exception:
INSERT into an identity column not allowed on table variables. The data for table-valued parameter '#InputFileParam' doesn't conform to the table type of the parameter.
How do I execute this SP?
I think Roland is on the right track - the IDENTITY columns definitely are the problem. Since your ultimate target table already has an IDENTITY column which will be filled automagically for you, I would recommend not specifying the same column in your user-defined table type (and also do not declare a primary key on the table type):
CREATE TYPE [dbo].[parseInputFile] AS TABLE
([strRow] [varchar](500) NOT NULL)
Then in your C# code, just set up a DataTable with a single column:
DataTable dt = new DataTable();
DataColumn dc = new DataColumn();
dc.DataType = typeof(string);
dc.ColumnName = “strRow”;
dt.Columns.Add(dc);
and then add your columns to your DataTable "dt" - with just the one string "strRow".
In your stored proc that then gets called with your user-defined table type, just simply insert your strings:
ALTER PROCEDURE [dbo].[InsertPIF] #InputFileParam parseInputFile READONLY
AS
INSERT INTO dbo.ParentTable(strInput)
SELECT strRow FROM #InputFileParam
GO
That should do it, I hope! (haven't had the time to test this myself)
Marc
Okay, I haven't actually worked with TVPs yet, so I could be wrong--but from my reading, & general SS background, I think your problem is this:
Your permanent table has an Identity column, so SS wants to handle that for you. You can set Identity Insert on, but that's not a good idea....
One thing to try is to specify the column to insert in your SP:
ALTER PROCEDURE [dbo].[InsertPIF] #InputFileParam parseInputFile READONLY
AS
INSERT dbo.ParentTable (strInput) SELECT strRow FROM #InputFileParam
GO
The other thing to look at, as Tim Lentine suggested (while I was writing this answer), is whether or not you're actually getting to the server. Since you've defined the NumCols column in your DataTable as AutoIncrement, it could be that you're getting your exception in the C# code, at:
...
dr["NumCols"] = 1;
...
< edit>
Don't know what I was thnking when I clicked Submit...
If that's the case, try leaving out the AutoIncrement when you creat your DataTable. If the column data types are consistent, this property shouldn't matter when lining up the TVP as being "the same" as the SS Type.
< /edit>
Could the error be coming from the parseInputFile table instead? NumCols is defined as an identify field, but in the C# code you are setting the value of the record to 1.