Develop Reference

How to import only non existing row from csv and update existing rows to SQL Database C#

With the help from the sylvan.data.csv package i'm now able to apply the scheme from the sql table to the csv. Yet the following problem arises where i want to check if a row from the csv exists in the sql database. If it does, it needs to be updated and if it does not only the non existing rows need to be imported. But with a bulkcopy this is not possible.
I've got the following code:
static void LoadTableCsv(SqlConnection conn, string tableName, string csvFile)
{
// read the column schema of the target table
var cmd = conn.CreateCommand();
conn.Open();
cmd.CommandText = $"select top 0 * from {tableName}"; // beware of sql injection
var reader = cmd.ExecuteReader();
var colSchema = reader.GetColumnSchema();
reader.Close();
// apply the column schema to the csv reader.
var csvSchema = new CsvSchema(colSchema);
var csvOpts = new CsvDataReaderOptions { Schema = csvSchema };
using var csv = CsvDataReader.Create(csvFile, csvOpts);
// Initialize SqlCommand for checking if the record already exists.
using var checkCommand = new SqlCommand("SELECT COUNT(*) FROM {tablename} WHERE TicketID = #value", conn);
checkCommand.Parameters.Add("#value", SqlDbType.Int, 10, "TicketID");
using var bulkCopy = new SqlBulkCopy(conn);
bulkCopy.DestinationTableName = tableName;
bulkCopy.EnableStreaming = true;
// Iterate through the records in the CSV file.
while (csv.Read())
{
// Set the value of the "#value" parameter
checkCommand.Parameters["#value"].Value = csv["TicketID"].ToString();
// Execute the check command to see if the record already exists.
var checkResult = (int)checkCommand.ExecuteScalar();
if (checkResult == 0)
{
// The record does not exist, write it to the SQL database using SqlBulkCopy.
bulkCopy.WriteToServer(new[] { csv });
}
else
{
// The record already exists, update it using an UPDATE statement.
using var updateCommand = new SqlCommand("UPDATE {tablename} SET Column1 = #col1, Column2 = #col2, Column3 = #col3, Column4 = #col4, Column5 = #col5, Column6 = #col6 WHERE TicketID = #value", conn);
// Add parameters for each column you want to update, using the names and types of the columns in the target table.
updateCommand.Parameters.Add("#col1", SqlDbType.Int, 10, "TicketID");
updateCommand.Parameters.Add("#col2", SqlDbType.NVarChar, 50, "TicketTitle");
updateCommand.Parameters.Add("#col3", SqlDbType.NVarChar, 50, "TicketStatus");
updateCommand.Parameters.Add("#col4", SqlDbType.NVarChar, 50, "CustomerName");
updateCommand.Parameters.Add("#col5", SqlDbType.NVarChar, 50, "TechnicianFullName");
updateCommand.Parameters.Add("#col6", SqlDbType.DateTime, 50, "TicketResolvedDate");
updateCommand.Parameters.Add("#value", SqlDbType.Int, 10, "TicketID");
// Set the values of the parameters to the values in the current row of the CSV file.
updateCommand.Parameters["#col1"].Value = int.Parse(csv["TicketID"].ToString());
updateCommand.Parameters["#col2"].Value = csv["TicketTitle"].ToString();
updateCommand.Parameters["#col3"].Value = csv["TicketStatus"].ToString();
updateCommand.Parameters["#col4"].Value = csv["CustomerName"].ToString();
updateCommand.Parameters["#col5"].Value = csv["TechnicianFullName"].ToString();
updateCommand.Parameters["#col6"].Value = DateTime.Parse(csv["TicketResolvedDate"].ToString());
updateCommand.Parameters["#value"].Value = int.Parse(csv["TicketID"].ToString());
// Execute the update command.
updateCommand.ExecuteNonQuery();
}
}
conn.Close();
}
But this gives me an error cause the bulkcopy can't read only one datarow.

Siggermannen's comment was the correct suggestion. You should bulk load all of the data into a temp table, then use SQL command(s) to merge the data from the temp table into the destination table. Ideally, you'd do this with a T-Sql Merge Statement. You could also use separate update and insert statements. This requires knowledge of the table columns to create the commands to merge the data. You can do this dynamically, by reading querying the INFORMATION_SCHEMA for the tables to determine the columns and keys, and using that to dynamically construct the merge statements. Or, if you know the schema at compile time, you can hard-code the statements which will be significantly easier to develop and test.
using Sylvan.Data.Csv;
using System.Data.SqlClient;
static void LoadTableCsv(SqlConnection conn, string tableName, string csvFile)
{
// read the column schema of the target table
var cmd = conn.CreateCommand();
cmd.CommandText = $"select top 0 * from {tableName}"; // beware of sql injection
var reader = cmd.ExecuteReader();
var colSchema = reader.GetColumnSchema();
reader.Close();
// create a temp table to load the data into
// using the destination table as a template
cmd.CommandText = $"select top 0 * into #load from {tableName}";
cmd.ExecuteNonQuery();
// apply the column schema to the csv reader.
var csvSchema = new CsvSchema(colSchema);
var csvOpts = new CsvDataReaderOptions { Schema = csvSchema };
using var csv = CsvDataReader.Create(csvFile, csvOpts);
// push *all* data into the temp table
using var bulkCopy = new SqlBulkCopy(conn);
bulkCopy.DestinationTableName = "#load";
bulkCopy.EnableStreaming = true;
bulkCopy.WriteToServer(csv);
// use sql commands to "MERGE" the data into the destination table
cmd.CommandText = $"""
insert into {tableName}
select * from #load l
where not exists (select * from {tableName} d where d.Id = l.Id)
""";
cmd.ExecuteNonQuery();
}
Doing insert/update statements in a loop is what you're trying to avoid. This produces "chatty" communication with the database where performance becomes dominated by overhead of each operation. Instead, you want to keep your operations as "chunky" as possble, which SqlBulkCopy and MERGE will provide.

Make Data columns from a sql adapter

Is there a way, where I can use a sql adapter and convert the sql query results into a Data Columns? I'm kinda new a datatables. I need to build this dynamically since my column names are stored into a sql table. I keep running into datarows not columns. What I have built so far:
string feedcolumns = #"select FeedColumnName from myTable where FeedProcessID = #feedprocessid";
SqlCommand columnscommand = new SqlCommand(feedcolumns, connUpd);
DataTable dt = new DataTable("datafeed");
foreach(DataColumn dc in dt.Columns)
{
dc = new
dt.Columns.Add(dc);
}

You can fill a DataTable directly from an SqlReader, no need to go column by column.
One thing I did notice was that your SQL statement had a parameter in it that was never assigned to the command object, so I added it in
string feedcolumns = "select FeedColumnName from myTable where FeedProcessID = #feedprocessid";
DataTable dt = new DataTable("datafeed");
using (SqlConnection connUpd = new SqlConnection("MyConnection")) {
using (SqlCommand columnscommand = new SqlCommand(feedcolumns, connUpd)) {
columnscommand.Parameters.AddWithValue("#feedprocessid", feedprocessid);
connUpd.Open();
var dataReader = columnscommand.ExecuteReader();
dt.Load(dataReader);
}
}

How to sort and rank and select a unique value?

I need to get the rank, from a group of players by sorting using sql query.
i have the query.
query = #"SET #rank=0;
SELECT player_ID,player_name,HP,#rank:=#rank+1 As Rank
FROM player_profile ORDER BY HP DESC;"
the problem is i just need the specified player's rank from this query.
Since rank is generated using SQL, i can't use WHERE clause. That will one bring one player from DB, resulting one rank.
I tried to get all of them into a datatable and then filter the single value out of it.
con.Open();
MySqlDataAdapter dt = new MySqlDataAdapter();
DataTable tt = new DataTable();
string query = #"SET #rank=0;
SELECT player_ID,player_name,HP,#rank:=#rank+1 As Rank
FROM player_profile ORDER BY HP DESC;";
MySqlCommand cm1 = new MySqlCommand(query, con);
dt.SelectCommand = cm1;
dt.Fill(tt);
con.Close();
DataRow[] foundRows = tt.Select("player_name=" + Label2.Text); // Error:Cannot find column ["Column name"]
foreach (DataRow dr in foundRows)
{
Label32.Text = dr["Rank"].ToString();
}
this is how my table looks
http://pastebin.com/7KWJ9bn3
any help is appreciated.

I'm a big proponent of not putting business logic in your SQL which it seems like your doing here. The below code is my updated version of your code which should be logically equivalent without having to do any calculation on the SQL side of things.
I'm not entirely sure what you are trying to achieve (is this example code?) so if you can provide more info I can refine this some more, but again this will do the exact same thing in a "better" fashion.
con.Open();
MySqlDataAdapter dt = new MySqlDataAdapter();
DataTable tt = new DataTable();
string query = #"SET #rank=0;
SELECT player_ID,player_name,HP
FROM player_profile ORDER BY HP DESC;";
MySqlCommand cm1 = new MySqlCommand(query, con);
dt.SelectCommand = cm1;
dt.Fill(tt);
con.Close();
DataRow[] foundRows = tt.Select("player_name=" + Label2.Text); // Error:Cannot find column ["Column name"]
int count = 1;
foreach (DataRow dr in foundRows)
{
Label32.Text = count;//dr["Rank"].ToString();
count++;
}

What about selecting from the resulting table?
string query = #"SET #rank=0;
Select a.Rank from (SELECT player_ID,#rank:=#rank+1 As Rank
FROM player_profile ORDER BY HP DESC) a where player_ID=#1";

How do I get sqldatareader to show the data in the fields?

I have an sql server table located on my website (remote). The table is called table1 and contains a bunch of fields. My goal here is to read all the fields of table1 into an array to iterate.
Here is my attempt:
private static void ShowFields()
{
using (SqlConnection connection = new SqlConnection(connectionstring))
{
connection.Open();
SqlCommand command = new SqlCommand("SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME='table1'", connection);
SqlDataReader reader = command.ExecuteReader();
//connection.Close();
int colCount = reader.FieldCount;
while (reader.Read())
{
for (int i = 0; i < colCount; i++)
{
Console.WriteLine(reader[i]);
}
}
}
}
this nearly works, but it shows all the properties of the table, rather than the the data in the fields---e.g., varchar, 50 dao, table etc.
http://i.imgur.com/2bsgMBC.png

If I understand correctly you want the actual data IN the table, however you're querying INFORMATION_SCHEMA which gives you data ABOUT the tables/columns/whatever...
So just query the table like so:
SELECT * FROM table1
I don't know the column names in your table but if you want just some of the columns to show you can replace the * with a column list:
SELECT col1, col2, col3 FROM table1
where col1, col2 and col3 are just the names of the columns.
Is this what you were going for, or am I way off the mark here?

How to get column names of table at runtime in C#?

Lets say I have a table in SQLServer named MyTable
ID FirstName LastName
1 Harry Dan
2 Maria Vicente
3 Joe Martin
Now if I have to insert any data in table, I will simply fire Insert Query like this
INSERT INTO MyTable (ID, FirstName, LastName) VALUES (4, Smith, Dan);
But what if I don't know the column names beforehand, I only know table name. Then is there a way to get the column name of table at runtime?

You can use sql-
SELECT name FROM sys.columns WHERE object_id = OBJECT_ID('TABLE_NAME')

Or you can query for SELECT TOP 0 * FROM TableName. Then, you can get the columns:
using(var reader = command.ExecuteReader())
{
reader.Read();
var table = reader.GetSchemaTable();
foreach (DataColumn column in table.Columns)
{
Console.WriteLine(column.ColumnName);
}
}

Another option using pure C# / .NET code:
First a helper method, that here returns a simple list of column names. Using a DataTable to hold table schema information, means that other information can also be retreived for each column, fx. if it is an AutoIncreament column etc.
private IEnumerable<string> GetColumnNames(string conStr, string tableName)
{
var result = new List<string>();
using (var sqlCon = new SqlConnection(conStr))
{
sqlCon.Open();
var sqlCmd = sqlCon.CreateCommand();
sqlCmd.CommandText = "select * from " + tableName + " where 1=0"; // No data wanted, only schema
sqlCmd.CommandType = CommandType.Text;
var sqlDR = sqlCmd.ExecuteReader();
var dataTable = sqlDR.GetSchemaTable();
foreach (DataRow row in dataTable.Rows) result.Add(row.Field<string>("ColumnName"));
}
return result;
}
The method can be called as:
var sortedNames = GetColumnNames("Data Source=localhost;Initial Catalog=OF2E;Integrated Security=SSPI", "Articles").OrderBy(x => x);
foreach (var columnName in sortedNames) Console.WriteLine(columnName);

Simply by this Query :
SELECT * FROM sys.columns
WHERE object_id = OBJECT_ID('dbo.[table_name]')
OR This Query :
SELECT COLUMN_NAME
FROM information_schema.columns
WHERE TABLE_NAME = [table_name]
ORDER BY COLUMN_NAME

var ColName = "";
var model = new AttributeMappingSource().GetModel(typeof(DataClassesDataContext));
foreach (var mt in model.GetTables())
{
if (mt.TableName == "dbo.Table_Name")
{
foreach (var dm in mt.RowType.DataMembers)
{
ColName = dm.MappedName + ", ";
Response.Write(ColName);
}
}
}

This question was answered before in various threads
check:
How can I get column names from a table in SQL Server?
How can I get column names from a table in Oracle?
How do I list all the columns in a table?