Develop Reference

Bulk Copy Commit Rows Without Errors

I have a process that takes a lists and inserts it into a database using SQL bulk copy because of how particularly large this list can be. It works fine, checks constraints and all which is perfect. The problem is, if I have 10,000 records and one of those records has an error, I still want to commit the other 9,999. Is there a way to do this other than manually checking each constraint before SQL bulk copy or inserting one at a time? Seems tedious and slow which kind of defeats the point. Thanks.
var copy = new SqlBulkCopy(ConfigurationManager.ConnectionStrings["constr"].ConnectionString, SqlBulkCopyOptions.CheckConstraints)
{
DestinationTableName = obj.TableName
};
var table = new DataTable();
copy.WriteToServer(table);

Without setting a batch size to 1 (which would defeat the purpose of the bulk copy) or pre-checking the data before the copy the normal way around this issue is you copy in to a temporary table with the same schema as your target table but with no constraints, remove the rows that would violate the constraints on insert, then do a normal insert from the temp table in to your live table.
const string _createTableString = "Create table #temp (/* SNIP */)";
const string _insertTableString = #"
declare #sql nvarchar(2000)
set #sql = N'INSERT INTO ' + QUOTENAME(#tableName) + N' SELECT * from #temp'
exec sp_executesql #sql";
using (var connection = new SqlConnection(ConfigurationManager.ConnectionStrings["constr"].ConnectionString))
{
connection.Open();
using (var command = new SqlCommand(_createTableString, connection))
{
command.ExecuteNonQuery();
}
using (var copy = new SqlBulkCopy(connection))
{
copy.DestinationTableName = "#temp";
copy.WriteToServer(table);
}
using (var command = new SqlCommand(_insertTableString, connection))
{
command.Parameters.AddWithValue("#tableName", obj.TableName)
command.ExecuteNonQuery();
}
}
Note the use of QUOTENAME to make sure that no SQL injections can sneak in via the name of the table passed in to obj.TableName.

SqlBulkCopy ColumnMapping Error

My goal is to copy generic tables from one database to another. I would like to have it copy the data as is and it would be fine to either delete whatever is in the table or to add to it with new columns if there are new columns. The only thing I may want to change is to add something for versioning which can be done in a seperate part of the query.
Opening the data no problem but when I try a bulk copy but it is failing. I have gone though several posts and the closest thing is this one:
SqlBulkCopy Insert with Identity Column
I removed the SqlBulkCopyOptions.KeepIdentity from my code but it still is throwing
"The given ColumnMapping does not match up with any column in the source or destination" error
I have tried playing with the SqlBulkCopyOptions but so far no luck.
Ideas?
public void BatchBulkCopy(string connectionString, DataTable dataTable, string DestinationTbl, int batchSize)
{
// Get the DataTable
DataTable dtInsertRows = dataTable;
using (SqlBulkCopy sbc = new SqlBulkCopy(connectionString))
{
sbc.DestinationTableName = DestinationTbl;
// Number of records to be processed in one go
sbc.BatchSize = batchSize;
// Finally write to server
sbc.WriteToServer(dtInsertRows);
}
}

If I could suggest another approach, I would have a look at the SMO (SQL Server Management Objects) library to perform such tasks.
You can find an interesting article here.
Using SMO, you can perform tasks in SQL Server, such a bulk copy, treating tables, columns and databases as objects.
Some time ago, I used SMO in a small open source application I developed, named SQLServerDatabaseCopy.
To copy the data from table to table, I created this code (the complete code is here):
foreach (Table table in Tables)
{
string columnsTable = GetListOfColumnsOfTable(table);
string bulkCopyStatement = "SELECT {3} FROM [{0}].[{1}].[{2}]";
bulkCopyStatement = String.Format(bulkCopyStatement, SourceDatabase.Name, table.Schema, table.Name, columnsTable);
using (SqlCommand selectCommand = new SqlCommand(bulkCopyStatement, connection))
{
LogFileManager.WriteToLogFile(bulkCopyStatement);
SqlDataReader dataReader = selectCommand.ExecuteReader();
using (SqlConnection destinationDatabaseConnection = new SqlConnection(destDatabaseConnString))
{
if (destinationDatabaseConnection.State == System.Data.ConnectionState.Closed)
{
destinationDatabaseConnection.Open();
}
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(destinationDatabaseConnection))
{
bulkCopy.DestinationTableName = String.Format("[{0}].[{1}]", table.Schema, table.Name);
foreach (Column column in table.Columns)
{
//it's not needed to perfom a mapping for computed columns!
if (!column.Computed)
{
bulkCopy.ColumnMappings.Add(column.Name, column.Name);
}
}
try
{
bulkCopy.WriteToServer(dataReader);
LogFileManager.WriteToLogFile(String.Format("Bulk copy successful for table [{0}].[{1}]", table.Schema, table.Name));
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.StackTrace);
}
finally
{
//closing reader
dataReader.Close();
}
}
}
}
}
As you can see, you have to add the ColumnMappings to the BulkCopy object for each column, because you have to define which column of source table must be mapped to a column of destination table. This is the reason of your error that says: The given ColumnMapping does not match up with any column in the source or destination.

I would add some validation to this to check what columns your source and destination tables have in common.
This essentially queries the system views (I have assumed SQL Server but this will be easily adaptable for other DBMS), to get the column names in the destination table (excluding identity columns), iterates over these and if there is a match in the source table adds the column mapping.
public void BatchBulkCopy(string connectionString, DataTable dataTable, string DestinationTbl, int batchSize)
{
using (SqlBulkCopy sbc = new SqlBulkCopy(connectionString))
{
sbc.DestinationTableName = DestinationTbl;
string sql = "SELECT name FROM sys.columns WHERE is_identity = 0 AND object_id = OBJECT_ID(#table)";
using (var connection = new SqlConnection(connectionString))
using (var command = new SqlCommand(sql, connection))
{
command.Parameters.AddWithValue("#table", DestinationTbl);
connection.Open();
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
var column = reader.GetString(0);
if (dataTable.Columns.Contains(column))
{
sbc.ColumnMappings.Add(column, column);
}
}
}
}
// Number of records to be processed in one go
sbc.BatchSize = batchSize;
// Finally write to server
sbc.WriteToServer(dataTable);
}
}
This could still get invalid cast errors as there is no data type check, but should get you started for a generic method.

You can add
sbc.ColumnMappings.Add(0, 0);
sbc.ColumnMappings.Add(1, 1);
sbc.ColumnMappings.Add(2, 2);
sbc.ColumnMappings.Add(3, 3);
sbc.ColumnMappings.Add(4, 4);
before executing
sbc.WriteToServer(dataTable);
Thank you !!

How to count columns of SQL Server 2008's table using C#?

I am developing an application in C# in Visual Studio 2008. I connected a SQL Server 2008 database with it.
I want to count the number of columns so that I can loop around them to get the particular data.
I can figure it out columns by going to the database but I am joing 4-5 tables in my programs so I want to know if I can count the columns.
Can anyone help me in this?
Thank you
Shyam

select count(*) from INFORMATION_SCHEMA.columns where TABLE_NAME = 'YourTableName'

Something like this ?
SELECT COUNT(*)
FROM sys.columns c
INNER JOIN sys.tables t ON c.object_id = t.object_id
WHERE t.name = 'yourTable'
See this page provided wy TaronPro to know how to retrieve the result.

If you are using SQLConnection object to connect to DB, use its GetSchema method to get list of all columns without querying.
using (SqlConnection connection = new SqlConnection(connectionString))
{
// Connect to the database then retrieve the schema information.
connection.Open();
DataTable table = connection.GetSchema("Tables");
..
..
..
If you want to know columns for specific owner, table or table type, use restriction within GetSchema method.
string[] restrictions = new string[4];
restrictions[1] = "dbo";
DataTable table = connection.GetSchema("Tables", restrictions);
for more information refer this link.

What I did in a similar situation is that when I executed the query I retrieved all the data into a DataSet.
When I got the DataSet I opened the first table (ds.Tables[0]). Obviously you check first for existance.
When you have the table then its as simple as performing a
dt.Columns.Count;
In summary DS.Tables[0].Columns.Count
To find a specific column by name you loop through and find the one that
for (z=0; z < dt.Columns.Count; z++)
{
// check to see if the column name is the required name passed in.
if (dt.Columns[z].ColumnName == fieldName)
{
// If the column was found then retrieve it
//dc = dt.Columns[z];
// and stop looking the rest of the columns
requiredColumn = z;
break;
}
}
Then to find the data you need I would then loop through the rows of the table and get the field for that column...
ie...
string return = dr.Field<string>(requiredColumn);
Probalby not the best way of doing it but it works. Obviously if the data contained in the field is not string then you need to pass the appropriate type...
dr.Field<decimal>(requiredColumn)
dr.Field<int>(requiredColumn)
etc
Rgds
George

The reader itself gives you the number of columns. This is useful when you don't want to know the number rows of a specific table or view but from an ad-hoc query.
You can dump the columns like this
string sql = "SELECT * FROM my query";
SqlCommand cmd = new SqlCommand(sql, connection);
using (SqlDataReader reader = cmd.ExecuteReader()) {
while (reader.Read()) {
for (int i = 0; i < reader.FieldCount; i++) {
Console.WriteLine("{0} = {1}",
reader.GetName(i),
reader.IsDBNull(i) ? "NULL" : reader.GetValue(i));
}
Console.WriteLine("---------------");
}
}

you can use Microsoft.SqlServer.Management.Smo namespace to get the number of columns in a specified table as follows
1 . add Microsoft.SqlServer.Management.Smo dll in your project and use the namespace Microsoft.SqlServer.Management.Smo
2 . write the follwing code
private int colCount()
{
Server server=new Server(".\\SQLEXPRESS");
Database database=Server.Databases["your database name"];
Table table=database.Tables["your table name"];
return (table.Columns.Count);
}

Bulk query insert in c#

I am inserting record using bulk query from source table to destination table. Source table have 10,000 records. Suppose source table have a column sid int, sname varchar(60) and destination column have sid int, sname varchar(30).
Now I was not able to insert all record successfully, as length problem in source sname and destination sname. As only few rows have a problem.
Now my question is that is there any way to insert record in destination table using bulk insert so that correct record is inserted and incorrect record is not inserted.
I am using c# 3.5
code which I am using
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(constring, SqlBulkCopyOptions.UseInternalTransaction))
{
try
{
//Edit mapping.
bulkCopy.ColumnMappings.Add("AddressID", "AddressID");
bulkCopy.ColumnMappings.Add("AddressLine1", "AddressLine1");
bulkCopy.ColumnMappings.Add("City", "City");
//specify destination table.
bulkCopy.DestinationTableName = "[Address]";
bulkCopy.BatchSize = 100;
bulkCopy.NotifyAfter = 100;
// bulkCopy.SqlRowsCopied += new SqlRowsCopiedEventHandler(bulkCopy_SqlRowsCopied);
bulkCopy.WriteToServer(table);
}
catch (Exception ex)
{
richTextBox1.AppendText("\n\n" + ex.Message);
}
}
Thanks.

Are you utilizing BULK INSERT through SQL Server, or are you using the System.Data.SqlClient.SqlBulkCopy class?
If you are using the BULK INSERT T-SQL, then you can just set MAXERRORS equal to whatever your desireble threshold is (the default is 10, which is why your `BULK INSERT is probably canceling out).
BULK INSERT YourDb.YourSchema.YourTable
FROM 'c:\YourFile.txt'
WITH
(
FIELDTERMINATOR =' |',
ROWTERMINATOR =' |\n',
MAXERRORS = 1000 -- or however many rows you think are exceeding the 30 char limit
)
EDIT
After seeing your are using the SqlBulkInsert class, I think your best bet would be to modify the DataTable before calling the SqlBulkCopy.WriteToServer() method:
foreach(DataRow row in table.Rows)
{
if (row["YourColumnThatExceeds30Chars"].ToString().Length > 30)
row["YourColumnThatExceeds30Chars"] =
row["YourColumnThatExceeds30Chars"].ToString().Substring(0, 30);
}

Getting the Last Insert ID with SQLite.NET in C#

I have a simple problem with a not so simple solution... I am currently inserting some data into a database like this:
kompenzacijeDataSet.KompenzacijeRow kompenzacija = kompenzacijeDataSet.Kompenzacije.NewKompenzacijeRow();
kompenzacija.Datum = DateTime.Now;
kompenzacija.PodjetjeID = stranka.id;
kompenzacija.Znesek = Decimal.Parse(tbZnesek.Text);
kompenzacijeDataSet.Kompenzacije.Rows.Add(kompenzacija);
kompenzacijeDataSetTableAdapters.KompenzacijeTableAdapter kompTA = new kompenzacijeDataSetTableAdapters.KompenzacijeTableAdapter();
kompTA.Update(this.kompenzacijeDataSet.Kompenzacije);
this.currentKompenzacijaID = LastInsertID(kompTA.Connection);
The last line is important. Why do I supply a connection? Well there is a SQLite function called last_insert_rowid() that you can call and get the last insert ID. Problem is it is bound to a connection and .NET seems to be reopening and closing connections for every dataset operation. I thought getting the connection from a table adapter would change things. But it doesn't.
Would anyone know how to solve this? Maybe where to get a constant connection from? Or maybe something more elegant?
Thank you.
EDIT:
This is also a problem with transactions, I would need the same connection if I would want to use transactions, so that is also a problem...

Using C# (.net 4.0) with SQLite, the SQLiteConnection class has a property LastInsertRowId that equals the Primary Integer Key of the most recently inserted (or updated) element.
The rowID is returned if the table doesn't have a primary integer key (in this case the rowID is column is automatically created).
See https://www.sqlite.org/c3ref/last_insert_rowid.html for more.
As for wrapping multiple commands in a single transaction, any commands entered after the transaction begins and before it is committed are part of one transaction.
long rowID;
using (SQLiteConnection con = new SQLiteConnection([datasource])
{
SQLiteTransaction transaction = null;
transaction = con.BeginTransaction();
... [execute insert statement]
rowID = con.LastInsertRowId;
transaction.Commit()
}

select last_insert_rowid();
And you will need to execute it as a scalar query.
string sql = #"select last_insert_rowid()";
long lastId = (long)command.ExecuteScalar(sql); // Need to type-cast since `ExecuteScalar` returns an object.

last_insert_rowid() is part of the solution. It returns a row number, not the actual ID.
cmd = CNN.CreateCommand();
cmd.CommandText = "SELECT last_insert_rowid()";
object i = cmd.ExecuteScalar();
cmd.CommandText = "SELECT " + ID_Name + " FROM " + TableName + " WHERE rowid=" + i.ToString();
i = cmd.ExecuteScalar();

I'm using Microsoft.Data.Sqlite package and I do not see a LastInsertRowId property. But you don't have to create a second trip to database to get the last id. Instead, combine both sql statements into a single string.
string sql = #"
insert into MyTable values (null, #name);
select last_insert_rowid();";
using (var cmd = conn.CreateCommand()) {
cmd.CommandText = sql;
cmd.Parameters.Add("#name", SqliteType.Text).Value = "John";
int lastId = Convert.ToInt32(cmd.ExecuteScalar());
}

There seems to be answers to both Microsoft's reference and SQLite's reference and that is the reason some people are getting LastInsertRowId property to work and others aren't.
Personally I don't use an PK as it's just an alias for the rowid column. Using the rowid is around twice as fast as one that you create. If I have a TEXT column for a PK I still use rowid and just make the text column unique. (for SQLite 3 only. You need your own for v1 & v2 as vacuum will alter rowid numbers)
That said, the way to get the information from a record in the last insert is the code below. Since the function does a left join to itself I LIMIT it to 1 just for speed, even if you don't there will only be 1 record from the main SELECT statement.
SELECT my_primary_key_column FROM my_table
WHERE rowid in (SELECT last_insert_rowid() LIMIT 1);

The SQLiteConnection object has a property for that, so there is not need for additional query.
After INSERT you just my use LastInsertRowId property of your SQLiteConnection object that was used for INSERT command.
Type of LastInsertRowId property is Int64.
Off course, as you already now, for auto increment to work the primary key on table must be set to be AUTOINCREMENT field, which is another topic.

database = new SQLiteConnection(databasePath);
public int GetLastInsertId()
{
return (int)SQLite3.LastInsertRowid(database.Handle);
}

# How about just running 2x SQL statements together using Execute Scalar?
# Person is a object that has an Id and Name property
var connString = LoadConnectionString(); // get connection string
using (var conn = new SQLiteConnection(connString)) // connect to sqlite
{
// insert new record and get Id of inserted record
var sql = #"INSERT INTO People (Name) VALUES (#Name);
SELECT Id FROM People
ORDER BY Id DESC";
var lastId = conn.ExecuteScalar(sql, person);
}

In EF Core 5 you can get ID in the object itself without using any "last inserted".
For example:
var r = new SomeData() { Name = "New Row", ...};
dbContext.Add(r);
dbContext.SaveChanges();
Console.WriteLine(r.ID);
you would get new ID without thinking of using correct connection or thread-safety etc.

If you're using the Microsoft.Data.Sqlite package, it doesn't include a LastInsertRowId property in the SqliteConnection class, but you can still call the last_insert_rowid function by using the underlying SQLitePCL library. Here's an extension method:
using Microsoft.Data.Sqlite;
using SQLitePCL;
public static long GetLastInsertRowId(this SqliteConnection connection)
{
var handle = connection.Handle ?? throw new NullReferenceException("The connection is not open.");
return raw.sqlite3_last_insert_rowid(handle);
}