Develop Reference

Populate database table from CSV

I am trying to populate a datatable reading values from a csv file. The format of the datatable should match a corresponding table in a database.
The csv file has very many columns (~80) so I don't want to type out everything. The names of the columns in the csv file don't match the names of the columns in the db exactly. Also, two additional columns, with data not present in the csv have to be added manually.
The problem is to convert the string data from the csv-file to the correct type in the datatable.
Currently I have
I read the table template from the database and use this to create my new datatable.
I create a map that maps the column positions from csv file to the column positions in the database.
I try to insert the value from the csv file into the datatable. This is where my code fails, because the data is of the incorrect type. As stated above, since there are so many different columns, I dont want to do the conversion manually but rather infer the type from the table template. Also, some columns can contain null values.
My code
public static DataTable ReadAssets(string strFilePath, DateTime reportingDate, Enums.ReportingBases reportingBasis, char sep=',')
{
//Reads the table template from the database
DataTable dt = DbInterface.Db.GetTableTemplate("reports.Assets");
var dbColumnNames = dt.Columns.Cast<DataColumn>().Select(x => x.ColumnName).ToList();
//These columns are not present in the csv data and so they have to be added manually
int posReportingDate = dbColumnNames.IndexOf("ReportingDate");
int posReportingBasis = dbColumnNames.IndexOf("ReportingBasis");
//read the csv and populate the table
using (StreamReader sr = new (strFilePath))
{
string[] csvColumnNames = sr.ReadLine().Split(sep);
//creates an <int, int> dictionary that maps the columns
var columnMap = CreateColumnMap(dbColumnNames.ToArray(), csvColumnNames);
while (!sr.EndOfStream)
{
string[] csvRow = sr.ReadLine().Split(sep);
DataRow dr = dt.NewRow();
dr[posReportingDate] = reportingDate;
dr[posReportingBasis] = reportingBasis.ToString();
foreach(var posPair in columnMap)
{
//This is where the code fails.... I need a conversion to the correct type here.
dr[posPair.Value] = csvRow[posPair.Key];
}
dt.Rows.Add(dr);
}
}
return dt;
}

I maintain a couple libraries that can help with this scenario: Sylvan.Data and Sylvan.Data.Csv. They are both open-source, MIT licensed, and available on nuget.org. My library allows applying a schema to CSV data, and attaching extra columns. Doing this allows using the SqlBulkCopy to efficiently load the data directly into the database. My CSV parser also happens to be the fastest in the .NET ecosystem.
As an example, given the following target SQL table:
create table MyTable (
Name varchar(32),
Value int,
ValueDate datetime,
InsertDate datetime,
RowNum int
)
A CSV file, data.csv, containing the following:
a,b,c
a,1,2022-01-01
b,2,2022-01-02
Here is a complete C# 6 sample program that will bulk copy CSV data along with "extra" columns into a data base table.
using Sylvan.Data; // v0.1.1
using Sylvan.Data.Csv; // v1.1.11
using System.Data.SqlClient;
const string SourceCsvFile = "data.csv";
const string TargetTableName = "MyTable";
var conn = new SqlConnection();
conn.ConnectionString = new SqlConnectionStringBuilder
{
DataSource = ".",
InitialCatalog = "Test",
IntegratedSecurity = true
}.ConnectionString;
conn.Open();
// read schema for the target table
var cmd = conn.CreateCommand();
cmd.CommandText = $"select top 0 * from {TargetTableName}";
var reader = cmd.ExecuteReader();
var schema = reader.GetColumnSchema();
reader.Close();
// apply the database schema to the CSV data
var opts = new CsvDataReaderOptions { Schema = new CsvSchema(schema) };
var csvReader = CsvDataReader.Create(SourceCsvFile, opts);
// attach additional external columns to the CSV data
var data = csvReader.WithColumns(
new CustomDataColumn<DateTime>("ImportDate", r => DateTime.UtcNow),
new CustomDataColumn<int>("RowNum", r => csvReader.RowNumber)
);
// bulk copy the data into the target table
var bc = new SqlBulkCopy(conn);
bc.DestinationTableName = TargetTableName;
bc.WriteToServer(data);
Hopefully you find this to be an elegant solution.

InvalidOperationException while trying bulkInsert the datatable into SQL Server

I got data from text file into datatable and now when I am trying to insert that data into a SQL Server 2008 database, I get the following error:
InvalidOperationException: String or binary data would be truncated
I cannot get the source of error ie which record is throwing this error.
The code is as below
for (int i = 0; i < dt.Columns.Count; i++)
{
if (i == 159)
{
}
bulkCopy.ColumnMappings.Add(dt.Columns[i].ColumnName,DestTable.Columns[i].ColumnName);
}
bulkCopy.BulkCopyTimeout = 600;
bulkCopy.DestinationTableName = "dbo.TxtFileInfo";
bulkCopy.WriteToServer(dt);
I have the datatable in the dt variable. And matched columns for both datatable created from text file and also the empty table created in database to add the values to it.
I have copied all records from text file into datatable using below code.
while (reader.Read())
{
int count1 = reader.FieldCount;
for (int i = 0; i < count1; i++)
{
string value = reader[i].ToString();
list.Add(value);
}
dt.Rows.Add(list.ToArray());
list.Clear();
}
I have got proper records from the text file. Also the number of columns are equal. My database table TextToTable has all columns of datatype nvarchar(50) and I am fetching each record as string from text file. But during bulk insert the error shown is
Cannot convert string to nvarchar(50)

Seems you are trying to insert data which has less length in DB (for example: you have data length 20 but in DB accept only 10 )
To check data which is coming from text file. Have if condition to check length of data in your code and have break point to troubleshoot.
If yes then increase the length of column in DB.
alter tablename
alter column columnname varchar(xxx)

How can I insert all rows from one table into another table with Linq-to-SQL?

I have two tables in SQL Server.
Table one contains these columns:
1-id
2-name
3-family
4-address
and table two contains these columns:
1-id
2-name
In table one I have 100000 rows and read all record with this query:
var query = (from p in datacontext.table1
select p).toArray();
I want insert all data from up query into the table2, now I use this method:
for(int i = 0; i < query.count(); i++) {
table2 beh = new tabl2();
beh.name = query[0].name;
datacontext.table2.insertonsubmit(beh);
datacontext.submitchange();
}
Is there another way? Thanks.

Making use of Linq to SQL to insert record one by one will take lot of time. Instead of that I Suggest make use of Bulk insert so that your data get insert in one go in less amount of time for that you can make use of of DataTable and OpenXML. for that tutorial is : Bulk Insertion of Data Using C# DataTable and SQL server OpenXML function
or use this
SqlBulkCopy.WriteToServer Method (DataTable)

Try
var bulkCopy = new SqlBulkCopy(connection);
bulkCopy.DestinationTableName = "table2";
bulkCopy.ColumnMappings.Add("Name", "Name");
using (var reader = new EntityDataReader<Table1>(query))
{
bulkCopy.WriteToServer(reader);
}
EntityDataReader

Using Bulk insert in System.Data.SqlClient data get insert in to db in less amount of time using the help of datatable
DataTable dt = getData();
SqlBulkCopyOptions options = SqlBulkCopyOptions.Default;
using (SqlBulkCopy sqlBulkCopy = new SqlBulkCopy(sqlconnection, options, null))
{
dt.Columns.Add("Id", typeof(long)).SetOrdinal(0);
dt = AddDefaultColumn(dt);
sqlBulkCopy.BulkCopyTimeout = 300;
sqlBulkCopy.DestinationTableName = "tableName";
sqlBulkCopy.WriteToServer(dt);
}

Say your table name is table1 and table2 with the columns of id and description you can use
INSERT INTO table2 (id, description)
SELECT table2.id, table2.description
FROM table1;
furthermore you can add a where
INSERT INTO table2 (id, description)
SELECT table2.id, table2.description
FROM table1 where table1.id =1;
you can visit this link for more info https://technet.microsoft.com/en-us/library/ms188263%28v=sql.105%29.aspx

Autodetecting Oracle data types

I'm trying to fetch data from multiple tables from an Oracle db and insert it into a sql db. The problem that I am running into is that I am fetching almost 50 columns of data all of different datatypes. I then proceed to insert these individual column values into a SQL statement which then inserts the data into the sql db. So the algo looks something like this:
Fetch row data{
create a variable for each individual column value ( int value = reader.getInt32(0); )
add a sqlparameter for it (command.Parameters.Add(new SqlParameter("value", value)); )
once all the 50 or so variables have been created make a sql statement
Insert into asdf values (value,........)
}
Doing it this way for a table with <10 columns seems ok but when it exceeds that length this process seems tedious and extraneous. I was wondering if there was a simpler way of doing this like fetch row data and automatically determine column data type and automatically create a varialbe and automatically insert into sql statement. I would appreciate it if anyone could direct me to the right way of doing this.

The data reader has a neutral GetValue method returning an object and the SqlCommand has an AddWithValue method that does not require to specify a parameter type.
for (int i = 0; i < reader.VisibleFieldCount; i++) {
object value = reader.GetValue(i);
command.Parameters.AddWithValue("#" + i, value);
}
You could also create the SQL command automatically
var columns = new StringBuilder();
var values = new StringBuilder();
for (int i = 0; i < reader.VisibleFieldCount; i++) {
values.Append("#").Append(i).Append(", ");
columns.Append("[").Append(reader.GetName(i)).Append("], ");
}
values.Length -= 2; // Remove last ", "
columns.Length -= 2;
string insert = String.Format("INSERT INTO myTable ({0}) VALUES ({1})",
columns.ToString(), values.ToString());

Get MS Access table creation script in C#?

Is is possible to get Ms Access Table script using C# ?
Although there is a tool that does this.
I was thinking if there is any automatic way to get the script of table .
Till now I am using
using (IDataReader reader = command.ExecuteReader(CommandBehavior.KeyInfo))
{
ret = reader.GetSchemaTable();
}
to get the schema of the table
Can we get creation script of access datatable in C# ?
Thank you All

You can create the script using the schema information, looping through columns getting the properties etc. Below is just adding column and datatype, but should be extended if the table is more intricate.
DataTable dt = reader.GetSchemaTable();
string query;
List<string> list = new List<string>();
foreach (DataRow columns in dt.Rows)
{
foreach (DataColumn properties in dt.Columns)
{
list.Add(properties.ColumnName + " " + properties.DataType);
}
}
query = string.Join(",", list);
Then build your string for the execute query.
Create Table [TableName] (
[append string here]
)