Develop Reference

Bulk Update in SQL Server from C#

I have to update multiple records in SQL Server table from C#. Below are the steps I have to follow and below is the code.
The code is working but the process is taking much longer than expected.
I need a quick way to update 10000 records, not sure whether Bulk Copy would work for Update.
I have seen the other answers which has Bulk insert to temp and then update..But that update has a single statement and here I need to update the records in DB based on Excel data and for this I have to loop each excel record.So how can I achieve faster update.
1) Read the Excel Data and copied the data into a data table
string strDirectory = string. Empty;
strDirectory = System.IO.Directory.GetCurrentDirectory() + "\\" + "Filename.xlsx";
string Connection String = "Provider=Microsoft.ACE.OLEDB.12.0; Data Source = " + strDirectory + "; Extended Properties = \"Excel 12.0;HDR=YES;IMEX=1\"";
using (OleDbConnection conn = new OleDbConnection(Connection String))
{
conn.Open();
DataTable schemaTable = conn.GetOleDbSchemaTableOleDbSchemaGuid.Tables, new object[] { null, null, null, "TABLE" });
DataRow schemaRow = schemaTable. Rows[0];
string sheet = schemaRow["TABLE_NAME"].ToString();
string query = "SELECT * FROM [" + sheet + "]";
OleDbDataAdapter daexcel = new OleDbDataAdapter(query, conn);
daexcel.Fill(dt);
conn.Close();
}
2) Doing some manipulations on datatable data before updating into table.
string strsqlst = string. Empty;
using (SqlConnection sqlConn = new SqlConnection(Connectionstring))
{
sqlConn.Open();
SqlCommand cmd;
StringBuilder sb = new StringBuilder();
sb.AppendLine("DataTable content:");
foreach (DataRow row in dt.Rows)
{
if (row.ItemArray[0].ToString() == "")
break;
strsqlst = "Update table Set col1= " + row.ItemArray[4].ToString() + " ,col2= " + row.ItemArray[5].ToString() + " where <Condition>'";
cmd = new SqlCommand(strsqlst, sqlConn);
cmd.CommandType = CommandType.Text;
cmd.ExecuteNonQuery();
}
sqlConn.Close();
}

The SqlCommand can be a whole SQL batch and is not limited to a single statement. So you can create a single large batch with 10,000 UPDATE statements, or divide it into for example 20 batches of 500 each.
In other words, you can create a single command with CommandText like this:
UPDATE [T] SET Col1='Value1', Col2='Value2' WHERE [Id] = 1;
...
UPDATE [T] SET Col1='Value999', Col2='Value1000' WHERE [Id] = 500;
That said, you should use parameters for all data values (to ensure SQL injection is not possible).
If you want to handle any errors (updates failing due to invalid data) you will need something a bit more sophisticated.

How to insert data from excel to datatable which has headers?

I am reading excel having like million of records first i query my table (no records) and get Datable. i query my table to get columns name as define in my excel sheet using alias.
var dal = new clsConn();
var sqlQuery = "SELECT FETAPE_THEIR_TRANDATE \"Date\" ,ISSUER Issuer, ISSU_BRAN Branch , STAN_NUMB STAN, TERMID TermID, ACQUIRER Acquirer,DEBIT_AMOUNT Debit,CREDIT_AMOUNT Credit,CARD_NUMB \"Card Number\" , DESCRIPTION Description FROM ALLTRANSACTIONS";
var returntable = dal.ReadData(sqlQuery);
DataRow ds = returntable.NewRow();
var dtExcelData = returntable;
So my datatable looks like this,
Then i read records from excel sheet
OleDbConnection con = null;
if (ext == ".xls")
{
con = new OleDbConnection(#"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + filepath + ";Extended Properties=Excel 8.0;");
}
else if (ext == ".xlsx")
{
con = new OleDbConnection(#"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + filepath + ";Extended Properties=\"Excel 12.0;IMEX=2;HDR=NO;TypeGuessRows=0;ImportMixedTypes=Text\"");
}
con.Open();
dt = con.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, null);
string getExcelSheetName = dt.Rows[0]["Table_Name"].ToString();
//OleDbCommand ExcelCommand = new OleDbCommand(#"SELECT * FROM [" + getExcelSheetName + #"]", con);
OleDbCommand ExcelCommand = new OleDbCommand("SELECT F1, F2, F3, F4, F5,F6,F7,F8,F9,F10 FROM [Sheet1$]", con);
OleDbDataAdapter ExcelAdapter = new OleDbDataAdapter(ExcelCommand);
try
{
ExcelAdapter.Fill(dtExcelData); //Here I give the datatable which i made previously
}
catch (Exception ex)
{
//lblAlert2.CssClass = "message-error";
//lblAlert2.Text = ex.Message;
}
It reads successfully and fill data in datatable but creating its own column in data table like F1 to F10 how can i move this data to exactly match with my defined columns in datatable
How Will i manage this to not create other columns (f1,f2..f10)
any workaround will be appreciable or Please explain what i am doing wrong and how can i achieve this.
UPDATE :
My Excel file looks like this

The Microsoft.ACE.OLEDB.12.0 driver will handle both types of excel spreadsheets and using the same Extended Properties. i.e. "Excel 12.0" will open both .xls and .xlsx.
Leave the HDR=NO as OLEDB expects them in the first row and they are actually in row 11.
Sadly "TypeGuessRows=0;ImportMixedTypes=Text" is completely ignored by Microsoft.ACE.OLEDB.12.0, you've got to play around with the registry (yuk). Change your IMEX=2 to IMEX=1 to ensure that mixed data types as handled as text.
Change back to using "Select * From [Sheet1$]" and then I'm afriad that you are going to have to handle the source data manually.
OleDbConnection con = new OleDbConnection(#"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + filepath + ";Extended Properties=\"Excel 12.0;IMEX=1;HDR=NO\"");
con.Open();
DataTable dt = con.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, null);
string getExcelSheetName = (string)dt.Rows[0]["Table_Name"];
DataTable xlWorksheet = new DataTable();
xlWorksheet.Load(new OleDbCommand("Select * From [" + getExcelSheetName + "]", con).ExecuteReader());
//More than 11 rows implies 11 header rows and at least 1 data row
if (xlWorksheet.Rows.Count > 11 & xlWorksheet.Columns.Count >= 10)
{
for (int nRow = 11; nRow < xlWorksheet.Rows.Count; nRow++)
{
DataRow returnRow = returntable.NewRow();
for (int nColumn = 0; nColumn < 10; nColumn++)
{
//Note you will probably get conversion problems here that you will have to handle
returnRow[nColumn] = xlWorksheet.Rows[nRow].ItemArray[nColumn];
}
returntable.Rows.Add(returnRow);
}
}
I'm guessing you simply want to add the excel data into your ALLTRANSACTION table? You don't specify but it seems the likely outcome of this. If so this is a terrible way to do it. You don't need to read the whole table into memory append data and then update the database. All you need to do is read the excel file and insert the data to the Oracle table.
Some thoughts, your returntable will contain data so if you just want the structure of the table then add a "Where RowNum=0" to the Select statement. To add the data to your Oracle Database you could 1) Convert to using the Oracle Data Provider (ODP) and then use using OracleBulkCopy Class or 2) simply modify the above to insert row by row as you read the data. As long as you don't have a LOT of data in your Excel spreadsheet it will work just fine. Having said that a Million rows is a LOT so perhaps not the best option. You will need to validate the input as Excel is not the best data source really.

Query string selections for XLS spreadsheet C#

I am trying to grab cells in XLS spreadsheets, assign them to string arrays, then manipulate the data and export to multiple CVS files.
The trouble is the XLS spreadsheet contains information that is not relevant, useable data doesn't start till row 17 and columns have no headings with just the default Sheet1.
I have looked at related questions and tried figuring it out myself with no success. The following code to read the XLS kinda works but is messy to work with as the row lengths vary from one XLS file to another and it is automatically pulling empty columns and rows.
CODE
public static void xlsReader()
{
string fileName = string.Format("{0}\\LoadsAvailable.xls", Directory.GetCurrentDirectory());
string connectionString = #"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + fileName + ";" + #"Extended Properties='Excel 8.0;HDR=Yes;'";
string queryString = "SELECT * FROM [Sheet1$]";
using (OleDbConnection connection = new OleDbConnection(connectionString))
{
OleDbCommand command = new OleDbCommand(queryString, connection);
connection.Open();
OleDbDataReader reader = command.ExecuteReader();
int counter = 0;
while (reader.Read())
{
Console.WriteLine("Line " + counter + ":" + reader[28].ToString()); // Just for testing
counter++;
}
}
}
I could do a bunch of trickery with loops to get the data that is required but there has to be a query string that could get the data from row 17 with only 8 columns(not 26 columns with 18 empty)?
I have tried many query string examples and can not seam to get any to work with a starting row index or filter out the empty data.

Here is a handy method that converts an excel file to a flat file.
You may want to change the connection string properties to suit your needs. I needed headers for my case.
Note you will need the Access database engine installed on your machine. I needed the 32 bit version since the app i dev'd was 32 bit. I bet you will also need it.
I parameterized the delimiter for the flat file, because I had cases where I didn't need a comma but a pipe symbol.
How to call method ex: ConvertExcelToFlatFile(openFileName, savePath, '|'); // pipe delimited
// Converts Excel To Flat file
private void ConvertExcelToFlatFile(string excelFilePath, string csvOutputFile, char delimeter, int worksheetNumber = 1)
{
if (!File.Exists(excelFilePath)) throw new FileNotFoundException(excelFilePath);
if (File.Exists(csvOutputFile)) throw new ArgumentException("File exists: " + csvOutputFile);
// connection string
var cnnStr = String.Format("Provider=Microsoft.ACE.OLEDB.12.0;Data Source={0};Extended Properties=\"Excel 12.0 Xml; IMEX=1; HDR=NO\"", excelFilePath);
var cnn = new OleDbConnection(cnnStr);
// get schema, then data
var dt = new DataTable();
try
{
cnn.Open();
var schemaTable = cnn.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, null);
if (schemaTable.Rows.Count < worksheetNumber) throw new ArgumentException("The worksheet number provided cannot be found in the spreadsheet");
string worksheet = schemaTable.Rows[worksheetNumber - 1]["table_name"].ToString().Replace("'", "");
string sql = String.Format("select * from [{0}]", worksheet);
var da = new OleDbDataAdapter(sql, cnn);
da.Fill(dt);
}
catch (Exception e)
{
throw e;
}
finally
{
// free resources
cnn.Close();
}
// write out CSV data
using (var wtr = new StreamWriter(csvOutputFile)) // disposes file handle when done
{
foreach (DataRow row in dt.Rows)
{
//MessageBox.Show(row.ItemArray.ToString());
bool firstLine = true;
foreach (DataColumn col in dt.Columns)
{
// skip the first line the initial
if (!firstLine)
{
wtr.Write(delimeter);
}
else
{
firstLine = false;
}
var data = row[col.ColumnName].ToString();//.Replace("\"", "\"\""); // replace " with ""
wtr.Write(String.Format("{0}", data));
}
wtr.WriteLine();
}
}
}

DataSet using first value as column header

I have a piece of code where I want to extract values from the A column of an Excel sheet. Right now, here is the code I'm using and having an issue with:
m_connString = "Provider = Microsoft.ACE.OLEDB.12.0; Data Source = " + m_source + "; Extended properties = 'Excel 12.0; HDR = NO; IMEX = 1;';";
using (OleDbConnection conn = new OleDbConnection(m_connString))
{
conn.Open();
DataTable dt = conn.GetOleDbSchemaGuid(Tables, null);
DataSet ds = new DataSet();
string defaultSheet = ExcelSheets.Rows[0]["TABLE_NAME"].ToString();
OleDbCommand comm = new OleDbCommand("SELECT * FROM [" + defaultSheet + "]", conn);
OleDbDataAdapter adapter = new OleDbDataAdapter(comm);
// Bug appears here
adapter.fill(ds)
// Fill a List<string> with the data found
for (int r = 0; r < ds.Tables[0].Row.Count; r++)
{
m_list.Add(ds.Tables[0].Rows[r][0].ToString();
}
}
What is happening is that, if I have an Excel file with the following content in the A column:
Row1
Row2
Row3
...
RowX
...What I end up getting is all values except for the first value (Row1). It turns out that Row1 is being used as a column name(?) for the DataSet's table. However, I don't want there to be any column names or headers, and I specifically state this in the connection string.
How can I prevent this behavior so I can have all my data placed in the List? Or, failing that, how can I work around this issue and extract Row1 from that DataSet?

Check Microsoft Reference to understand how Extended Properties of the connection HDR=NO works:
Column headings: By default, it is assumed that the first row of your
Excel data source contains columns headings that can be used as field
names. If this is not the case, you must turn this setting off, or
your first row of data "disappears" to be used as field names. This is
done by adding the optional HDR= setting to the Extended Properties of
the connection string. The default, which does not need to be
specified, is HDR=Yes. If you do not have column headings, you need to
specify HDR=No; the provider names your fields F1, F2, etc.
Here is example:
Excel File Data (test.xlsx):
Code:
string m_source = "test.xlsx";
string m_connString = #"Provider = Microsoft.ACE.OLEDB.12.0;
Data Source = " + m_source + #";
Extended properties = 'Excel 12.0;
HDR= NO;
IMEX = 1;';";
using (OleDbConnection conn = new OleDbConnection(m_connString))
{
conn.Open();
string squery = "SELECT f1, f2, f3 FROM [Sheet1$]";
OleDbCommand comm = new OleDbCommand(squery, conn);
OleDbDataAdapter adapter = new OleDbDataAdapter(comm);
DataSet ds = new DataSet();
adapter.Fill(ds);
}
DataSet Visualizer:

Try this way:
DataTable table = ds.Tables[0];
foreach (DataColumn column in table.Columns)
{
string cName = table.Rows[0][column.ColumnName].ToString();
if (!table.Columns.Contains(cName) && cName != "")
{
column.ColumnName = cName;
}
}

change the sql to something along the lines of this
"SELECT * FROM [" + defaultSheet + "] Except Select Top(1)"

Why My SqlBulkCopy Is Not Working

I am using SqlBulkCopy object to write a datatable into an sql server table. However, everytime I recheck my database it remains intact with no changes. A
I have tried to do Google search to determine my problem but i am unable to resolve it.
The datatable came from an .xls file.
public static DataTable dt = new DataTable();
private void ExportToGrid(String path, String filen)
{
int idx = filen.IndexOf(".");
string tf = filen.Remove(idx, 4);
OleDbConnection MyConnection = null;
DataSet DtSet = null;
OleDbDataAdapter MyCommand = null;
MyConnection = new OleDbConnection("provider=Microsoft.Jet.OLEDB.4.0; Data Source='" + path + "';Extended Properties=Excel 8.0;");
ArrayList TblName = new ArrayList();
MyConnection.Open();
DataTable schemaTable = MyConnection.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, new object[] { null, null, null, "TABLE" });
foreach (DataRow row in schemaTable.Rows)
{
TblName.Add(row["TABLE_NAME"]);
}
MyCommand = new System.Data.OleDb.OleDbDataAdapter("select * from [" + TblName[0].ToString() + "]", MyConnection);
DtSet = new System.Data.DataSet();
MyCommand.Fill(DtSet);
MyCommand.FillSchema(DtSet, SchemaType.Source);
DataTable dt = new DataTable();
dt = DtSet.Tables[0];
Session["dt"] = dt;
int x = dt.Rows.Count;
MyConnection.Close();
if (dt.Rows.Count > 0)
{
theGridView.DataSource = dt;
theGridView.DataBind();
}
if (System.IO.File.Exists(path))
{
System.IO.File.Delete(path);
}
}
This is my writer function
private void StartImport()
{
string servername = server;
string database = database;
string tbl = "dbo.LinkDb";
Stopwatch sw = new Stopwatch();
sw.Start();
SqlBulkCopy bulkCopy = new SqlBulkCopy("Data Source=" + servername + ";Initial Catalog=" + database + ";Integrated Security=SSPI", SqlBulkCopyOptions.TableLock);
bulkCopy.DestinationTableName = tbl;
bulkCopy.WriteToServer(dt);
sw.Stop();
lblResult.Visible = true;
lblResult.Text = (sw.ElapsedMilliseconds / 1000.00).ToString();
}
Below are the screenshot of the tables stored in my sql server. I assure you that I have been complying to Case Sensitive rules.
There was no exception thrown and average time elapsed is 0.018 - 0.020 secs
Appreciate any helps.
Thank you

Based on the code you have posted, you are writing an empty datatable to the database. Your "ExportToGrid" method fills dt, a DataTable declared locally, which loses scope outside of the method. Your write function is calling the static DataTable dt which is a new datatable.
Does dt need to be static? it seems as though this could be declared as
private DataTable dt;
then inside "ExportToGrid" instead of declaring another DataTable just instantiate the already declared dt rather than declaring a new one
dt = new DataTable();
Alternatively you could extract the DataTable straight from the GridView during the write Method:
DataTable dt = (DataTable)theGridView.DataSource;
bulkCopy.WriteToServer(dt);
This removes the need for variables outside of the scope of the method.
Lastly since you are storing your datatable within the session (I am not generally an advocate of storing large amounts of data in session variables but without knowing the specifics of your site I cannot really pass judgement), you could use the following:
DataTable dt = (DataTable)Session["dt"];
bulkCopy.WriteToServer(dt);

I dont see anything obvious compared to my use except for the fact that I explicitly map columns from the data table to the database table.
Using cn As New SqlConnection(DataAccessResource.CONNECTIONSTRING)
cn.Open()
Using copy As New SqlBulkCopy(cn)
copy.BulkCopyTimeout = 300
copy.ColumnMappings.Add(0, 0)
copy.ColumnMappings.Add(1, 1)
copy.ColumnMappings.Add(2, 2)
copy.ColumnMappings.Add(3, 3)
copy.DestinationTableName = "Tablename"
copy.WriteToServer(dataset.datatable)
End Using
End Using
Connection string (sql server 2000!) looks like
"data source=DBSERVERNAME;initial catalog=DBNAME;persist security info=True;user id=USERNAME;password=PASSWORD;packet size=4096"
I doubt the connection string is a problem assuming youve used it elsewhere.
Finally have you checked the data types for the columns in the dataset datatable match the ones in the database. In my experience oledb load from excel does not always produce output you might expect, date fields and columns with mixed text and numbers being perticular problems.