I'm using SqlBulkCopy against two SQL Server 2008 with different sets of columns (going to move some data from prod server to dev). So want to skip some columns not yet existed / not yet removed.
How can I do that? Some trick with ColumnMappings?
Edit:
I do next:
DataTable table = new DataTable();
using (var adapter = new SqlDataAdapter(sourceCommand))
{
adapter.Fill(table);
}
table.Columns
.OfType<DataColumn>()
.ForEach(c => bulk.ColumnMappings.Add(
new SqlBulkCopyColumnMapping(c.ColumnName, c.ColumnName)));
bulk.WriteToServer(table)
and get:
The given ColumnMapping does not match up with any column in the source or destination.
DataTable table = new DataTable();
using (var adapter = new SqlDataAdapter(sourceCommand))
{
adapter.Fill(table);
}
using (SqlBulkCopy bulk = new SqlBulkCopy(targetConnection, SqlBulkCopyOptions.KeepIdentity, null) { DestinationTableName = tableName })
{
foreach (string columnName in GetMapping(stringSource, stringTarget, tableName))
{
bulk.ColumnMappings.Add(new SqlBulkCopyColumnMapping(columnName, columnName));
}
targetConnection.Open();
bulk.WriteToServer(table);
}
private static IEnumerable<string> GetMapping(string stringSource, string stringTarget, string tableName)
{
return Enumerable.Intersect(
GetSchema(stringSource, tableName),
GetSchema(stringTarget, tableName),
StringComparer.Ordinal); // or StringComparer.OrdinalIgnoreCase
}
private static IEnumerable<string> GetSchema(string connectionString, string tableName)
{
using (SqlConnection connection = new SqlConnection(connectionString))
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = "sp_Columns";
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add("#table_name", SqlDbType.NVarChar, 384).Value = tableName;
connection.Open();
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
yield return (string)reader["column_name"];
}
}
}
}
When SqlBulkCopyColumnMapping is used, only columns for which mappings are created will be copied.
If you do not create a mapping for a column, it will be ignored by the copy process.
You can see this in the demo code here - the sample source table in the AdventureWorks demo database contains more columns than are mapped or copied.
EDIT
It's difficult to be certain without more information about the database schema, but at a guess the issue is with this statement:
new SqlBulkCopyColumnMapping(c.ColumnName, c.ColumnName)
From your description, it sounds like not all the columns in the source table exist in the destination table. You need a filter in your SqlBulkCopyColumnMapping construction loop to skip any columns which do not exist in the destination.
My C# is not good enough to give a example which I'm confident will work, but in pseudocode it would be
foreach column c in sourcetable
{
if c.ColumnName exists in destination_table.columns
{
new SqlBulkCopyColumnMapping(c.ColumnName, c.ColumnName)
}
}
(I'm sure it's possible to convert this to a lambda expression)
Note that this is not particularly robust in the scenario where the column names match but the datatypes are incompatible.
Ed Harper, this is what it looks like without pseudo code
(in this case from DataTable dt (fully defined) to an existing table in the db:
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connectionString))
{
bulkCopy.DestinationTableName = "dbo.DepartmentsItems";
// Write from the source to the destination.
foreach (DataColumn c in dt.Columns)
{
bulkCopy.ColumnMappings.Add(c.ColumnName, c.ColumnName);
}
bulkCopy.WriteToServer(dt);
return dt.Rows.Count;
}
try this:SqlBulkCopyColumnMapping Class
Hope you are looking for the same
Related
This questions had been posted several times (1,2,3,4), but I did not find one that would apply to my case.
I have a Dictionary with the structure:
public Dictionary<Int32, PhaseMag> Data;
Where PhaseMag:
public struct PhaseMag
{
public Single Magnitude;
public Single Phase;
}
Each Key value will contain two 2 values (Mag. and Phase). If you prefer to see an image.
I need to store all the content of this Dictionary in a unique Table in the DB (Microsoft SQL). Each line of the Dictionary should become one line of my DB Table. My final table will contain 3 fields, 1) Key 2) Mag and 3) Phase. For example, if I have 30 Keys, my table will contain 30 lines, one for each key.
My ideas:
Create a foreach loop based on each Key and create an insert into to the DB (one for Key->Mag and other for Key->Phase). But I don't think that this will be the best approach, especially, because my dictionary contains several thousand of lines.
So, what should be my approach to do this? I simply need to save my Dictionary into the DB, which each line of the Dic. will be one line from the DB.
Assume you're using SQL Server, here're two of many options you have:
Bulk insert, it's recommended.
public void BulkWrite(Dictionary<Int32, PhaseMag> data)
{
var dataTable = new DataTable();
dataTable.Columns.Add(new DataColumn { DataType = typeof(int), ColumnName = "Key" });
dataTable.Columns.Add(new DataColumn { DataType = typeof(Single), ColumnName = "Magnitude" });
dataTable.Columns.Add(new DataColumn { DataType = typeof(Single), ColumnName = "Phase" });
foreach (var x in data)
{
var r = dataTable.NewRow();
dataTable.Rows.Add(r);
r[0] = x.Key;
r[1] = x.Value.Magnitude;
r[2] = x.Value.Phase;
}
using (var conn = new SqlConnection("conneciton string"))
{
conn.Open();
using (var bulkCopy = new SqlBulkCopy(conn))
{
bulkCopy.BatchSize = 4000;
bulkCopy.DestinationTableName = "YorTableName";
bulkCopy.WriteToServer(dataTable);
}
}
}
Multiple inline query, executed as batch. As all your data are number, so low risk of SQL injection even using inline query.
public void InlineQueryWrite(Dictionary<Int32, PhaseMag> data)
{
using (var conn = new SqlConnection("conneciton string"))
{
conn.Open();
foreach (var bulk in data.Select((d, i) => new {d, i}).GroupBy(x => x.i % 10))
{
var sb = new StringBuilder();
foreach (var x in bulk)
{
sb.AppendFormat("Insert Into Your_Table (Key, Magnitude, Phase) Values ({0},{1},{2});", x.d.Key, x.d.Value.Magnitude, x.d.Value.Phase);
}
using (var command = conn.CreateCommand())
{
command.CommandText = sb.ToString();
command.ExecuteNonQuery();
}
}
}
}
I haven't ran/tested through the code, but they should work.
I have a SQL Server stored procedure that I run two select statements. I can easily return one select statement and store it in a DataTable but how do I use two?
How can I set my variable countfromfirstselectstatement equal to the count returned from my first select statement and how can I set my variable countfromsecondselectstatement equal to the count returned from the second select.
private void ReturnTwoSelectStatements()
{
DataSet dt112 = new DataSet();
using (var con = new SqlConnection(connectionString))
{
using (var cmd = new SqlCommand("Return2SelectStatements", con))
{
using (var da = new SqlDataAdapter(cmd))
{
cmd.CommandType = CommandType.StoredProcedure;
da.Fill(dt112);
}
}
}
DataTable table1 = dt112.Tables[0];
DataTable table2 = dt112.Tables[1];
string countfromFirstSelectStatement = table1.Rows.Count().ToString();
string countfromSecondSelectStatement = table2.Rows.Count().ToString();
//I only want to iterate the data from the 1st select statement
foreach (DataRow row in dt112.Rows)
{
}
}
You could also directly use the DbDataReader (cmd.ExecuteReader()), which gives you NextResult to advance to the next result set. DataTable.Load allows you to load rows from the data reader to the table.
DbDataAdapter is really a bit of an overkill for just reading data into a data table. It's designed to allow the whole CRUD-breadth of operation for controls that are abstracted away from the real source of data, which isn't really your case.
Sample:
using (var reader = cmd.ExecuteReader())
{
dataTable1.Load(reader);
if (!reader.NextResult()) throw SomethingWhenTheresNoSecondResultSet();
dataTable2.Load(reader);
}
Fill a Dataset with both select statements and access it:
....
DataSet dataSet = new DataSet();
da.Fill(dataSet);
....
DataTable table1 = dataSet.Tables[0];
DataTable table2 = dataSet.Tables[1];
string countfromFirstSelectStatement = table1.Rows.Count.ToString();
string countfromSecondSelectStatement = table2.Rows.Count.ToString();
I am migrating my program from Microsoft SQL Server to MySQL. Everything works well except one issue with bulk copy.
In the solution with MS SQL the code looks like this:
connection.Open();
SqlBulkCopy bulkCopy = new SqlBulkCopy(connection);
bulkCopy.DestinationTableName = "testTable";
bulkCopy.WriteToServer(rawData);
Now I try to do something similar for MySQL. Because I think there would be bad performance I don't want to write the DataTable to a CSV file and do the insert from there with the MySqlBulkLoader class.
Any help would be highly appreciated.
Because I think there would be bad performance I don't want to write the DataTable to a CSV file and do the insert from there with the MySqlBulkLoader class.
Don't rule out a possible solution based on unfounded assumptions. I just tested the insertion of 100,000 rows from a System.Data.DataTable into a MySQL table using a standard MySqlDataAdapter#Update() inside a Transaction. It consistently took about 30 seconds to run:
using (MySqlTransaction tran = conn.BeginTransaction(System.Data.IsolationLevel.Serializable))
{
using (MySqlCommand cmd = new MySqlCommand())
{
cmd.Connection = conn;
cmd.Transaction = tran;
cmd.CommandText = "SELECT * FROM testtable";
using (MySqlDataAdapter da = new MySqlDataAdapter(cmd))
{
da.UpdateBatchSize = 1000;
using (MySqlCommandBuilder cb = new MySqlCommandBuilder(da))
{
da.Update(rawData);
tran.Commit();
}
}
}
}
(I tried a couple of different values for UpdateBatchSize but they didn't seem to have a significant impact on the elapsed time.)
By contrast, the following code using MySqlBulkLoader took only 5 or 6 seconds to run ...
string tempCsvFileSpec = #"C:\Users\Gord\Desktop\dump.csv";
using (StreamWriter writer = new StreamWriter(tempCsvFileSpec))
{
Rfc4180Writer.WriteDataTable(rawData, writer, false);
}
var msbl = new MySqlBulkLoader(conn);
msbl.TableName = "testtable";
msbl.FileName = tempCsvFileSpec;
msbl.FieldTerminator = ",";
msbl.FieldQuotationCharacter = '"';
msbl.Load();
System.IO.File.Delete(tempCsvFileSpec);
... including the time to dump the 100,000 rows from the DataTable to a temporary CSV file (using code similar to this), bulk-loading from that file, and deleting the file afterwards.
Similar to SqlBulkCopy, we have MySqlBulkCopy for Mysql.
here is the example how to use it.
public async Task<bool> MySqlBulCopyAsync(DataTable dataTable)
{
try
{
bool result = true;
using (var connection = new MySqlConnector.MySqlConnection(_connString + ";AllowLoadLocalInfile=True"))
{
await connection.OpenAsync();
var bulkCopy = new MySqlBulkCopy(connection);
bulkCopy.DestinationTableName = "yourtable";
// the column mapping is required if you have a identity column in the table
bulkCopy.ColumnMappings.AddRange(GetMySqlColumnMapping(dataTable));
await bulkCopy.WriteToServerAsync(dataTable);
return result;
}
}
catch (Exception ex)
{
throw;
}
}
private List<MySqlBulkCopyColumnMapping> GetMySqlColumnMapping(DataTable dataTable)
{
List<MySqlBulkCopyColumnMapping> colMappings = new List<MySqlBulkCopyColumnMapping>();
int i = 0;
foreach (DataColumn col in dataTable.Columns)
{
colMappings.Add(new MySqlBulkCopyColumnMapping(i, col.ColumnName));
i++;
}
return colMappings;
}
You can ignore the column mapping if you don't have any identity column in your table.
If you have identity column then you have to use the column mapping otherwise it won't insert any records in the table
It will just give message like "x rows were copied but only 0 rows were inserted".
This class i available in the below library
Assembly MySqlConnector, Version=1.0.0.0
Using any of BulkOperation NuGet-package, you can easily have this done.
Here is an example using the package from https://www.nuget.org/packages/Z.BulkOperations/2.14.3/
MySqlConnection conn = DbConnection.OpenConnection();
DataTable dt = new DataTable("testtable");
MySqlDataAdapter da = new MySqlDataAdapter("SELECT * FROM testtable", conn);
MySqlCommandBuilder cb = new MySqlCommandBuilder(da);
da.Fill(dt);
instead of using
......
da.UpdateBatchSize = 1000;
......
da.Update(dt)
just following two lines
var bulk = new BulkOperation(conn);
bulk.BulkInsert(dt);
will take only 5 seconds to copy the whole DataTable into MySQL without first dumping the 100,000 rows from the DataTable to a temporary CSV file.
My table contains 10 columns. I need to insert a list using c#.
I have stored the details of multiple members, for each count its has to insert the consecutive details in the same row.
if (members.Count >= 1)
{
foreach (Members myList in members)
{
Command.Parameters.Add("first", SqlDbType.VarChar).Value = myList.first;
Command.Parameters.Add("last", SqlDbType.VarChar).Value = myList.last;
Command.Parameters.Add("age", SqlDbType.VarChar).Value = myList.age;
}
}
Example : for count=1 the table looks like
"fName1","lName1",21
for count=2 the table looks like
"fName1","lName1",21,"fname2","lName2",21
please help on this.
The coding style looks ambiguous. Your foreach loop runs for - 'Members' in members. It makes hard to understand what are trying to do. Let me suggest you to refactor your code and let the class name be 'Member'. You can put members in db with ADO.Net (there are other ways too) as follows -
using (SqlConnection connection = new SqlConnection(connectionString))
{
using (SqlCommand command = connection.CreateCommand())
{
//select just schema of the table.
command.CommandText = "select * from members where 1=2;";
using (SqlDataAdapter adapter = new SqlDataAdapter(command))
{
using (SqlCommandBuilder builder = new SqlCommandBuilder(adapter))
{
using (DataTable dt = new DataTable())
{
foreach (Member item in memebers)
{
DataRow row = dt.NewRow();
row.SetField<string>("", item.FirstName);
row.SetField<string>("", item.LastName);
row.SetField<int>("", item.Age);
//
// number of SetField should be equal to number of selected columns.
//
dt.Rows.Add(row);
}
adapter.Update(dt);
}
}
}
}
}
I want to fetch all the column names for specific table..
I am using msaccess and C# .net 2008.
You can fetch schema information for a given query through OleDb using the SchemaOnly CommandBehavior and the GetSchemaTable method, as follows:
var conStr = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=NWIND.mdb";
using (var con = new OleDbConnection(conStr))
{
con.Open();
using (var cmd = new OleDbCommand("select * from Suppliers", con))
using (var reader = cmd.ExecuteReader(CommandBehavior.SchemaOnly))
{
var table = reader.GetSchemaTable();
var nameCol = table.Columns["ColumnName"];
foreach (DataRow row in table.Rows)
{
Console.WriteLine(row[nameCol]);
}
}
}
A variant of bubi's method for a specific table:
public List<string> GetTableColumnNames(string tableName)
{
var conStr = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=NWIND.mdb";
using (var connection = new OleDbConnection(conStr))
{
connection.Open();
var schemaTable = connection.GetOleDbSchemaTable(
OleDbSchemaGuid.Columns,
new Object[] { null, null, tableName });
if (schemaTable == null)
return null;
var columnOrdinalForName = schemaTable.Columns["COLUMN_NAME"].Ordinal;
return (from DataRow r in schemaTable.Rows select r.ItemArray[columnOrdinalForName].ToString()).ToList();
}
}
Of course first you might want to check if the table actually exists before getting its column names:
public bool TableExists(string tableName)
{
var conStr = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=NWIND.mdb";
using (var connection = new OleDbConnection(conStr))
{
connection.Open();
var tables = connection.GetSchema("Tables");
var tableExists = false;
for (var i = 0; i < tables.Rows.Count; i++)
{
tableExists = String.Equals(tables.Rows[i][2].ToString(),
tableName,
StringComparison.CurrentCultureIgnoreCase);
if (tableExists)
break;
}
return tableExists;
}
}
This retrieves all the columns of all tables and views
DataTable schemaTable = ((OleDbConnection)jetConnection).GetOleDbSchemaTable(
System.Data.OleDb.OleDbSchemaGuid.Columns,
new object[] { null, null, null, null });
I found this article while trying to build a C# application to migrate an Access database. The database I'm migrating is an Access 2007/2010 file with .accdb extension.
If you use this code on a table that has Memo or Attachment columns (available in accdb files), it will return the type of these columns as string (wchar).
I had trouble finding much information about how to deal with these types of columns, so I wanted to provide a link to the article that helped me figure out how to handle them:
https://social.msdn.microsoft.com/Forums/vstudio/en-US/d15606f9-f38d-4a1b-8ce3-000c558e79c5
I took the bottom example in that thread and converted it to C#. I did have to add this using statement to the module to avoid having to edit all of the references to "AccessDao":
using AccessDao = Microsoft.Office.Interop.Access.Dao;
My apologies for tacking onto an old thread, but I used this thread as a starting point for writing my code and didn't realize this gotcha right away.
Here's code to get the column names in the order they appear in the Access table. The examples in the other answers here return the column names in alphabetical order (at least for me... using the Microsoft Access Database Engine 2016 Redistributable and .NET Core 3.1).
Based on qnaninf's code example:
var schemaTable = conn.GetOleDbSchemaTable(OleDbSchemaGuid.Columns, new object[] { null, null, tableName });
var columnOrdinalForName = schemaTable.Columns["COLUMN_NAME"].Ordinal;
var columnOrdinalForOrdinal = schemaTable.Columns["ORDINAL_POSITION"].Ordinal;
var rows = schemaTable.Rows;
var columns = from DataRow r in schemaTable.Rows
orderby r.ItemArray[columnOrdinalForOrdinal]
select new
{
Ordinal = r.ItemArray[columnOrdinalForOrdinal].ToString(),
ColumnName = r.ItemArray[columnOrdinalForName].ToString()
};
You can get the column names in Vb.net and Oledb from MS access database as follows.
'In Vb.net with OleDb
Dim adapter As new OleDb.OleDbDataAdapter
Dim ds As New DataSet
cmd.CommandText = "select * from table_name where 1=2"
adapter.SelectCommand = cmd
adapter.Fill(ds)
adapter.Dispose()
cmd.Dispose()
For Each dr In ds.Tables(0).Columns
ComboBox1.Items.Add(dr.ToString) 'The Column name will come in this combobox
Next