This questions had been posted several times (1,2,3,4), but I did not find one that would apply to my case.
I have a Dictionary with the structure:
public Dictionary<Int32, PhaseMag> Data;
Where PhaseMag:
public struct PhaseMag
{
public Single Magnitude;
public Single Phase;
}
Each Key value will contain two 2 values (Mag. and Phase). If you prefer to see an image.
I need to store all the content of this Dictionary in a unique Table in the DB (Microsoft SQL). Each line of the Dictionary should become one line of my DB Table. My final table will contain 3 fields, 1) Key 2) Mag and 3) Phase. For example, if I have 30 Keys, my table will contain 30 lines, one for each key.
My ideas:
Create a foreach loop based on each Key and create an insert into to the DB (one for Key->Mag and other for Key->Phase). But I don't think that this will be the best approach, especially, because my dictionary contains several thousand of lines.
So, what should be my approach to do this? I simply need to save my Dictionary into the DB, which each line of the Dic. will be one line from the DB.
Assume you're using SQL Server, here're two of many options you have:
Bulk insert, it's recommended.
public void BulkWrite(Dictionary<Int32, PhaseMag> data)
{
var dataTable = new DataTable();
dataTable.Columns.Add(new DataColumn { DataType = typeof(int), ColumnName = "Key" });
dataTable.Columns.Add(new DataColumn { DataType = typeof(Single), ColumnName = "Magnitude" });
dataTable.Columns.Add(new DataColumn { DataType = typeof(Single), ColumnName = "Phase" });
foreach (var x in data)
{
var r = dataTable.NewRow();
dataTable.Rows.Add(r);
r[0] = x.Key;
r[1] = x.Value.Magnitude;
r[2] = x.Value.Phase;
}
using (var conn = new SqlConnection("conneciton string"))
{
conn.Open();
using (var bulkCopy = new SqlBulkCopy(conn))
{
bulkCopy.BatchSize = 4000;
bulkCopy.DestinationTableName = "YorTableName";
bulkCopy.WriteToServer(dataTable);
}
}
}
Multiple inline query, executed as batch. As all your data are number, so low risk of SQL injection even using inline query.
public void InlineQueryWrite(Dictionary<Int32, PhaseMag> data)
{
using (var conn = new SqlConnection("conneciton string"))
{
conn.Open();
foreach (var bulk in data.Select((d, i) => new {d, i}).GroupBy(x => x.i % 10))
{
var sb = new StringBuilder();
foreach (var x in bulk)
{
sb.AppendFormat("Insert Into Your_Table (Key, Magnitude, Phase) Values ({0},{1},{2});", x.d.Key, x.d.Value.Magnitude, x.d.Value.Phase);
}
using (var command = conn.CreateCommand())
{
command.CommandText = sb.ToString();
command.ExecuteNonQuery();
}
}
}
}
I haven't ran/tested through the code, but they should work.
Related
I'm trying to create code that adds column 'ExternalId' to almost every table in foreach loop with some exceptions. If table schema name equals "Scheduler" or table name starts with "users" dont add column.
First this I probably need is a list of schemas and tables.
public class AddExternalIdColumnsToManyTables : Migration
{
public override void Up()
{
// schema name example
// Scheduler.Job
// dbo.users_logins
var schemaAndTableNames = new List<string>();
foreach (var item in schemaAndTableNames)
{
if (item.StartsWith("Scheduler"))
{
continue;
}
if (item.Split('.')[1].StartsWith("users"))
{
continue;
}
Alter.Table(item.Split('.')[1]).AddColumn("ExternalId").AsInt32().Nullable();
}
If you're asking how to get table names:
MigrationBase class (Migration's parent) has ConnectionsString property. You can easily get it and write ADO.net query to get table names. Simple example:
var builder = new SqlConnectionStringBuilder(ConnectionString);
var query = "SELECT TABLE_NAME" +
"FROM INFORMATION_SCHEMA.TABLES" +
"WHERE TABLE_TYPE = 'BASE TABLE' AND " +
$"TABLE_CATALOG = '{builder.InitialCatalog}'"
using (var connection = new SqlConnection(builder.ConnectionString))
{
connection.Open();
var command = new SqlCommand(query, connection);
using (var sqlReader = command.ExecuteReader(CommandBehavior.KeyInfo))
{
var schemaTable = sqlReader.GetSchemaTable();
foreach (DataRow row in schemaTable.Rows)
{
///your table names
}
}
connection.Close();
}
I want to perform insert into database but in different Tables. Each record i have in my file is assigned with the table Identifiers which will be used as a key. For now what i have done is to make it a generic approach, I have created one more file.txt in which i have specified all the identifiers with the table names. and stored it into a dictionary of string, string.
So, I have example some thing like below in my dictionary:
Table Identifier, Table Name
Table Identifier, Table Name
Then I created another dictionary of string as key (Note: I have used the previous dictionary value as a key into this dictionary) and list of string as values to get the column names of the table name.
So, No I have some thing like this below sample data into my dictionary:
Table Name, Column Names IEnumerable
Table Name, Column Names IEnumerable
Then,
The datafile.txt which contains the data as pipe delimited, I split them and saved into a List of KVP of string and List of String. As I mentioned before, I have table identifiers in my data files. So i used them as key in my KVP and store the split values into List of string.
So, No I have some thing like this below sample data into my List of KVP of string , List of string:
Table Identifier, IEnumerable Values
Table Identifier, IEnumerable Values
Now doing half of the work, I am stuck into my FINAL ISSUE:
Now I have All the identifers , Table Names, Column Names and Values with me into the dictionary and List. And only thing to do is to match and merge the records and DUMP it!
For matching: I have thought to match the List of KVPs key with the dictionarys key and then use the values as a key to get the Column Names.
Expected Scenario Image:
List [[List< Column Names>] , [List of String Values]]
MY CODE:
DataTable dt = null;
SqlConnection cn = null;
SqlDataReader dataReader = null;
SqlBulkCopy bulkInsert = null;
StreamReader reader = null;
string path = string.Empty;
public void test()
{
string TableIdentiferFilepath = HttpContext.Current.Server.MapPath("/testfile/TableIdentifer.txt");
Dictionary<string, string> TableIdentifer_TableName = null;
Dictionary<string, List<string>> Table_Name_ColumnName = null;
using (reader = new StreamReader(TableIdentiferFilepath))
{
TableIdentifer_TableName = new Dictionary<string, string>();
Table_Name_ColumnName = new Dictionary<string, List<string>>();
while (!reader.EndOfStream)
{
string[] curr = reader.ReadLine().Split(new string[] { ",", "\r\n" }, StringSplitOptions.None);
TableIdentifer_TableName.Add(curr[0], curr[1]);
using (cn = new SqlConnection(ConString.Connection.conn))
{
cn.Open();
if (cn.State == ConnectionState.Open)
{
string query = string.Format("select column_name from information_schema.columns where table_name = '{0}' order by ordinal_position", curr[1].ToString());
using (SqlCommand cmd = new SqlCommand(query))
{
using (SqlDataAdapter da = new SqlDataAdapter(query, cn))
{
using (dt = new DataTable())
{
da.Fill(dt);
List<string> dataColumns = dt.AsEnumerable().Select(r => r.Field<string>("column_name")).ToList();
Table_Name_ColumnName.Add(curr[1], dataColumns);
}
}
}
}
}
}
}
string path = HttpContext.Current.Server.MapPath("/TextFile/DataSample.txt");
List<KeyValuePair<string, List<string>>> KVPValues = new List<KeyValuePair<string, List<string>>>();
using (reader = new StreamReader(path))
{
while (!reader.EndOfStream)
{
string[] arr = reader.ReadLine().Split(new string[] { "|", "\r\n" }, StringSplitOptions.None);
var collValues = new List<string>();
KVPValues.Add(new KeyValuePair<string, List<string>>(arr[0], arr.Skip(1).AsEnumerable().ToList()));
foreach (var item in TableIdentifer_TableName)
{
foreach (var item2 in Table_Name_ColumnName.Where(c => c.Key == item.Value))
{
var curr_val = item2.Value;
var currKey = KVPValues.Where(p => p.Key == item.Key).ToList();
}
}
}
}
}
This is a BIG PICTURE! Hopefully now people will understand what i want to achieve.
My table contains 10 columns. I need to insert a list using c#.
I have stored the details of multiple members, for each count its has to insert the consecutive details in the same row.
if (members.Count >= 1)
{
foreach (Members myList in members)
{
Command.Parameters.Add("first", SqlDbType.VarChar).Value = myList.first;
Command.Parameters.Add("last", SqlDbType.VarChar).Value = myList.last;
Command.Parameters.Add("age", SqlDbType.VarChar).Value = myList.age;
}
}
Example : for count=1 the table looks like
"fName1","lName1",21
for count=2 the table looks like
"fName1","lName1",21,"fname2","lName2",21
please help on this.
The coding style looks ambiguous. Your foreach loop runs for - 'Members' in members. It makes hard to understand what are trying to do. Let me suggest you to refactor your code and let the class name be 'Member'. You can put members in db with ADO.Net (there are other ways too) as follows -
using (SqlConnection connection = new SqlConnection(connectionString))
{
using (SqlCommand command = connection.CreateCommand())
{
//select just schema of the table.
command.CommandText = "select * from members where 1=2;";
using (SqlDataAdapter adapter = new SqlDataAdapter(command))
{
using (SqlCommandBuilder builder = new SqlCommandBuilder(adapter))
{
using (DataTable dt = new DataTable())
{
foreach (Member item in memebers)
{
DataRow row = dt.NewRow();
row.SetField<string>("", item.FirstName);
row.SetField<string>("", item.LastName);
row.SetField<int>("", item.Age);
//
// number of SetField should be equal to number of selected columns.
//
dt.Rows.Add(row);
}
adapter.Update(dt);
}
}
}
}
}
I'm using SqlBulkCopy against two SQL Server 2008 with different sets of columns (going to move some data from prod server to dev). So want to skip some columns not yet existed / not yet removed.
How can I do that? Some trick with ColumnMappings?
Edit:
I do next:
DataTable table = new DataTable();
using (var adapter = new SqlDataAdapter(sourceCommand))
{
adapter.Fill(table);
}
table.Columns
.OfType<DataColumn>()
.ForEach(c => bulk.ColumnMappings.Add(
new SqlBulkCopyColumnMapping(c.ColumnName, c.ColumnName)));
bulk.WriteToServer(table)
and get:
The given ColumnMapping does not match up with any column in the source or destination.
DataTable table = new DataTable();
using (var adapter = new SqlDataAdapter(sourceCommand))
{
adapter.Fill(table);
}
using (SqlBulkCopy bulk = new SqlBulkCopy(targetConnection, SqlBulkCopyOptions.KeepIdentity, null) { DestinationTableName = tableName })
{
foreach (string columnName in GetMapping(stringSource, stringTarget, tableName))
{
bulk.ColumnMappings.Add(new SqlBulkCopyColumnMapping(columnName, columnName));
}
targetConnection.Open();
bulk.WriteToServer(table);
}
private static IEnumerable<string> GetMapping(string stringSource, string stringTarget, string tableName)
{
return Enumerable.Intersect(
GetSchema(stringSource, tableName),
GetSchema(stringTarget, tableName),
StringComparer.Ordinal); // or StringComparer.OrdinalIgnoreCase
}
private static IEnumerable<string> GetSchema(string connectionString, string tableName)
{
using (SqlConnection connection = new SqlConnection(connectionString))
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = "sp_Columns";
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add("#table_name", SqlDbType.NVarChar, 384).Value = tableName;
connection.Open();
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
yield return (string)reader["column_name"];
}
}
}
}
When SqlBulkCopyColumnMapping is used, only columns for which mappings are created will be copied.
If you do not create a mapping for a column, it will be ignored by the copy process.
You can see this in the demo code here - the sample source table in the AdventureWorks demo database contains more columns than are mapped or copied.
EDIT
It's difficult to be certain without more information about the database schema, but at a guess the issue is with this statement:
new SqlBulkCopyColumnMapping(c.ColumnName, c.ColumnName)
From your description, it sounds like not all the columns in the source table exist in the destination table. You need a filter in your SqlBulkCopyColumnMapping construction loop to skip any columns which do not exist in the destination.
My C# is not good enough to give a example which I'm confident will work, but in pseudocode it would be
foreach column c in sourcetable
{
if c.ColumnName exists in destination_table.columns
{
new SqlBulkCopyColumnMapping(c.ColumnName, c.ColumnName)
}
}
(I'm sure it's possible to convert this to a lambda expression)
Note that this is not particularly robust in the scenario where the column names match but the datatypes are incompatible.
Ed Harper, this is what it looks like without pseudo code
(in this case from DataTable dt (fully defined) to an existing table in the db:
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connectionString))
{
bulkCopy.DestinationTableName = "dbo.DepartmentsItems";
// Write from the source to the destination.
foreach (DataColumn c in dt.Columns)
{
bulkCopy.ColumnMappings.Add(c.ColumnName, c.ColumnName);
}
bulkCopy.WriteToServer(dt);
return dt.Rows.Count;
}
try this:SqlBulkCopyColumnMapping Class
Hope you are looking for the same
I want to fetch all the column names for specific table..
I am using msaccess and C# .net 2008.
You can fetch schema information for a given query through OleDb using the SchemaOnly CommandBehavior and the GetSchemaTable method, as follows:
var conStr = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=NWIND.mdb";
using (var con = new OleDbConnection(conStr))
{
con.Open();
using (var cmd = new OleDbCommand("select * from Suppliers", con))
using (var reader = cmd.ExecuteReader(CommandBehavior.SchemaOnly))
{
var table = reader.GetSchemaTable();
var nameCol = table.Columns["ColumnName"];
foreach (DataRow row in table.Rows)
{
Console.WriteLine(row[nameCol]);
}
}
}
A variant of bubi's method for a specific table:
public List<string> GetTableColumnNames(string tableName)
{
var conStr = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=NWIND.mdb";
using (var connection = new OleDbConnection(conStr))
{
connection.Open();
var schemaTable = connection.GetOleDbSchemaTable(
OleDbSchemaGuid.Columns,
new Object[] { null, null, tableName });
if (schemaTable == null)
return null;
var columnOrdinalForName = schemaTable.Columns["COLUMN_NAME"].Ordinal;
return (from DataRow r in schemaTable.Rows select r.ItemArray[columnOrdinalForName].ToString()).ToList();
}
}
Of course first you might want to check if the table actually exists before getting its column names:
public bool TableExists(string tableName)
{
var conStr = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=NWIND.mdb";
using (var connection = new OleDbConnection(conStr))
{
connection.Open();
var tables = connection.GetSchema("Tables");
var tableExists = false;
for (var i = 0; i < tables.Rows.Count; i++)
{
tableExists = String.Equals(tables.Rows[i][2].ToString(),
tableName,
StringComparison.CurrentCultureIgnoreCase);
if (tableExists)
break;
}
return tableExists;
}
}
This retrieves all the columns of all tables and views
DataTable schemaTable = ((OleDbConnection)jetConnection).GetOleDbSchemaTable(
System.Data.OleDb.OleDbSchemaGuid.Columns,
new object[] { null, null, null, null });
I found this article while trying to build a C# application to migrate an Access database. The database I'm migrating is an Access 2007/2010 file with .accdb extension.
If you use this code on a table that has Memo or Attachment columns (available in accdb files), it will return the type of these columns as string (wchar).
I had trouble finding much information about how to deal with these types of columns, so I wanted to provide a link to the article that helped me figure out how to handle them:
https://social.msdn.microsoft.com/Forums/vstudio/en-US/d15606f9-f38d-4a1b-8ce3-000c558e79c5
I took the bottom example in that thread and converted it to C#. I did have to add this using statement to the module to avoid having to edit all of the references to "AccessDao":
using AccessDao = Microsoft.Office.Interop.Access.Dao;
My apologies for tacking onto an old thread, but I used this thread as a starting point for writing my code and didn't realize this gotcha right away.
Here's code to get the column names in the order they appear in the Access table. The examples in the other answers here return the column names in alphabetical order (at least for me... using the Microsoft Access Database Engine 2016 Redistributable and .NET Core 3.1).
Based on qnaninf's code example:
var schemaTable = conn.GetOleDbSchemaTable(OleDbSchemaGuid.Columns, new object[] { null, null, tableName });
var columnOrdinalForName = schemaTable.Columns["COLUMN_NAME"].Ordinal;
var columnOrdinalForOrdinal = schemaTable.Columns["ORDINAL_POSITION"].Ordinal;
var rows = schemaTable.Rows;
var columns = from DataRow r in schemaTable.Rows
orderby r.ItemArray[columnOrdinalForOrdinal]
select new
{
Ordinal = r.ItemArray[columnOrdinalForOrdinal].ToString(),
ColumnName = r.ItemArray[columnOrdinalForName].ToString()
};
You can get the column names in Vb.net and Oledb from MS access database as follows.
'In Vb.net with OleDb
Dim adapter As new OleDb.OleDbDataAdapter
Dim ds As New DataSet
cmd.CommandText = "select * from table_name where 1=2"
adapter.SelectCommand = cmd
adapter.Fill(ds)
adapter.Dispose()
cmd.Dispose()
For Each dr In ds.Tables(0).Columns
ComboBox1.Items.Add(dr.ToString) 'The Column name will come in this combobox
Next