I am using SqlBulkCopy object to write a datatable into an sql server table. However, everytime I recheck my database it remains intact with no changes. A
I have tried to do Google search to determine my problem but i am unable to resolve it.
The datatable came from an .xls file.
public static DataTable dt = new DataTable();
private void ExportToGrid(String path, String filen)
{
int idx = filen.IndexOf(".");
string tf = filen.Remove(idx, 4);
OleDbConnection MyConnection = null;
DataSet DtSet = null;
OleDbDataAdapter MyCommand = null;
MyConnection = new OleDbConnection("provider=Microsoft.Jet.OLEDB.4.0; Data Source='" + path + "';Extended Properties=Excel 8.0;");
ArrayList TblName = new ArrayList();
MyConnection.Open();
DataTable schemaTable = MyConnection.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, new object[] { null, null, null, "TABLE" });
foreach (DataRow row in schemaTable.Rows)
{
TblName.Add(row["TABLE_NAME"]);
}
MyCommand = new System.Data.OleDb.OleDbDataAdapter("select * from [" + TblName[0].ToString() + "]", MyConnection);
DtSet = new System.Data.DataSet();
MyCommand.Fill(DtSet);
MyCommand.FillSchema(DtSet, SchemaType.Source);
DataTable dt = new DataTable();
dt = DtSet.Tables[0];
Session["dt"] = dt;
int x = dt.Rows.Count;
MyConnection.Close();
if (dt.Rows.Count > 0)
{
theGridView.DataSource = dt;
theGridView.DataBind();
}
if (System.IO.File.Exists(path))
{
System.IO.File.Delete(path);
}
}
This is my writer function
private void StartImport()
{
string servername = server;
string database = database;
string tbl = "dbo.LinkDb";
Stopwatch sw = new Stopwatch();
sw.Start();
SqlBulkCopy bulkCopy = new SqlBulkCopy("Data Source=" + servername + ";Initial Catalog=" + database + ";Integrated Security=SSPI", SqlBulkCopyOptions.TableLock);
bulkCopy.DestinationTableName = tbl;
bulkCopy.WriteToServer(dt);
sw.Stop();
lblResult.Visible = true;
lblResult.Text = (sw.ElapsedMilliseconds / 1000.00).ToString();
}
Below are the screenshot of the tables stored in my sql server. I assure you that I have been complying to Case Sensitive rules.
There was no exception thrown and average time elapsed is 0.018 - 0.020 secs
Appreciate any helps.
Thank you
Based on the code you have posted, you are writing an empty datatable to the database. Your "ExportToGrid" method fills dt, a DataTable declared locally, which loses scope outside of the method. Your write function is calling the static DataTable dt which is a new datatable.
Does dt need to be static? it seems as though this could be declared as
private DataTable dt;
then inside "ExportToGrid" instead of declaring another DataTable just instantiate the already declared dt rather than declaring a new one
dt = new DataTable();
Alternatively you could extract the DataTable straight from the GridView during the write Method:
DataTable dt = (DataTable)theGridView.DataSource;
bulkCopy.WriteToServer(dt);
This removes the need for variables outside of the scope of the method.
Lastly since you are storing your datatable within the session (I am not generally an advocate of storing large amounts of data in session variables but without knowing the specifics of your site I cannot really pass judgement), you could use the following:
DataTable dt = (DataTable)Session["dt"];
bulkCopy.WriteToServer(dt);
I dont see anything obvious compared to my use except for the fact that I explicitly map columns from the data table to the database table.
Using cn As New SqlConnection(DataAccessResource.CONNECTIONSTRING)
cn.Open()
Using copy As New SqlBulkCopy(cn)
copy.BulkCopyTimeout = 300
copy.ColumnMappings.Add(0, 0)
copy.ColumnMappings.Add(1, 1)
copy.ColumnMappings.Add(2, 2)
copy.ColumnMappings.Add(3, 3)
copy.DestinationTableName = "Tablename"
copy.WriteToServer(dataset.datatable)
End Using
End Using
Connection string (sql server 2000!) looks like
"data source=DBSERVERNAME;initial catalog=DBNAME;persist security info=True;user id=USERNAME;password=PASSWORD;packet size=4096"
I doubt the connection string is a problem assuming youve used it elsewhere.
Finally have you checked the data types for the columns in the dataset datatable match the ones in the database. In my experience oledb load from excel does not always produce output you might expect, date fields and columns with mixed text and numbers being perticular problems.
Related
Can you help me with the following?
I have two databases: one in SQL Server making the connection with ADO.NET; the other is a DBF database which makes a class for the connection of the same. I want to make a comparison of data from one table to the other, since they have the same structure, and I want to see what data is missing from both the SQL Server and the DBF.
namespace log
{
static class Program
{
private static ECEntities1 dc = new ECEntities1();
public static void Main()
{
// Query in BDD SQL Server
var query =
from HInvoice in dc.HInvoice
where HInvoice.DOB == '2020-03-01'
select HInvoice.DOB
// Print data
foreach (var item in query)
{
Console.WriteLine(item);
}
// Query in BDD BDF
string path = AppDomain.CurrentDomain.BaseDirectory + #"db";
DataTable dt = DBF.ObtenerDatos(path, "GNDITEM.Dbf");
// Print Data
foreach (DataRow dtRow in dt.Rows)
{
// On all tables' columns
foreach (DataColumn dc in dt.Columns)
{
var field1 = dtRow[dc].ToString();
Console.WriteLine(field1);
}
}
}
}
}
I will answer myself, I am new so the question was how could I be focused, but that was not the way so I changed the focus what I did was the following:
I created a new project where I sent the DBF table to my SQL Server database through a stored procedure and from there make the comparison
Create the connection to the Dbf database.
OleDbConnection cnn = new OleDbConnection();
OleDbDataAdapter da = new OleDbDataAdapter();
OleDbCommand cmd;
string PathArchivo = (#"\\ruta_archivo");
string NombreArchivo = "GNDITEM.Dbf";
cnn.ConnectionString = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + #PathArchivo + ";Extended Properties=dBASE IV;User ID=;Password=";
cmd = new OleDbCommand("select * from " + NombreArchivo, cnn);
da.SelectCommand = cmd;
cnn.Close();
var dt = new DataTable();
da.Fill(dt);
Pass the parameters to a DataTable
var dtParametro = new DataTable();
dtParametro.Columns.Add("parametro1", typeof(string));
dtParametro.Columns.Add("parametro2", typeof(string));
dtParametro.Columns.Add("parametro3", typeof(string));
dtParametro.Columns.Add("parametro4", typeof(string));
dtParametro.Rows.Add(drAdd);
// On all tables' columns
Then with a For we fill the DataTable with the information from the DBF table
foreach (DataRow dr in dt.Rows)
{
drAdd["TYPE"] = dr["parametro1"].ToString();
drAdd["EMPLOYEE"] = dr["parametro2"].ToString();
drAdd["CHECK"] = dr["parametro3"].ToString();
drAdd["ITEM"] = dr["parametro4"].ToString();
dtParametro.Rows.Add(drAdd);
}
Finally I connect to the SQL Server database and call the stored procedure
string cadena1 = #"conexionabdd...";
var cmda = new SqlCommand("Insertar_tbl_GNDITEM1", new SqlConnection(cadena));
cmda.CommandType = CommandType.StoredProcedure;
cmda.Parameters.Add("#param", SqlDbType.Structured).Value = dtParametro;
cmda.Connection.Open(); // abrimos la conexion
cmda.ExecuteNonQuery(); // lo ejecutamso
cmda.Connection.Close(); // cerramos la conexion
We review the database and see that the data is already in the same
PS: In addition to this, you must create the database in the SQL Server, also create a datatype for that table and the stored procedure, here is the video that will guide me, I hope it will be helpful for someone else Video
I am caching data to decrease SQL Server activity.
When I fetch data from SQL Server the first time it is requested, I then use WriteXml to store it on disk.
Then second time it is requested (i.e. a file called cacheName exists), I fetch the data with ReadXml.
The data in SQL Server has only one row with utcDT value of '2012-03-25 02:01' and when I write the DataTable at Code point 1 Test1() shows there is only one row. When I manually inspect Xml file on disk I also see only 1 row.
However, after reading with ReadXml at code point 2 Test1() shows 2 such rows!
How is this happening?
public static DataTable FetchCache()
{
var cacheName = #"C:\MyCache.xml";
if (File.Exists(cacheName))
{
var ds = new DataSet();
var fs = new FileStream(cacheName, FileMode.Open, FileAccess.Read, FileShare.Read);
using (var sr = new StreamReader(fs))
{
ds.ReadXml(sr, XmlReadMode.ReadSchema);
}
table = ds.Tables[0];
Test1(table); //Code point 2
}
else
{
table = FetchDataTable(connectionString, sqlCommand, nullOnError: nullOnError);
Test1(table); //Code point 1
table.WriteXml(cacheName, XmlWriteMode.WriteSchema);
}
return table;
}
public static void Test1(DataTable table)
{
var rows = table.AsEnumerable()
.Where(x => x.Field<DateTime>("utcDT").Equals(new DateTime(2012, 03, 25, 02, 01, 00)))
.ToArray();
}
public static DataTable FetchDataTable(string connectionString, string sqlCommand, bool nullOnError= false)
{
DataSet ds = new DataSet();
DataTable dt = new DataTable();
try
{
using (SqlConnection conn = new SqlConnection(connectionString))
{
conn.Open();
SqlDataAdapter da = new SqlDataAdapter(sqlCommand, conn);
da.SelectCommand.CommandTimeout = 600;
ds.Reset();
da.Fill(ds);
dt = ds.Tables[0];
}
}
catch (Exception err)
{
if (nullOnError)
return dt;
throw new Exception("[Utils.MsSQLS.FetchDataTable] Could not get data: " + err.Message);
}
return dt;
}
Just in case anyone else has same issue. I found the answer.
My SQL table DateTime columns were originally specified as UnspecifiedLocal, but the data they contain is actually UTC.
In order to save to cache as Xml, I have to create a new DataTable clone from table I fetch from SQL, and change DateTimeMode to UTC , then I copy from the old table to the new (row by row using ItemArray copy).
This is cumbersome but gets rid of the issue.
I am reading excel having like million of records first i query my table (no records) and get Datable. i query my table to get columns name as define in my excel sheet using alias.
var dal = new clsConn();
var sqlQuery = "SELECT FETAPE_THEIR_TRANDATE \"Date\" ,ISSUER Issuer, ISSU_BRAN Branch , STAN_NUMB STAN, TERMID TermID, ACQUIRER Acquirer,DEBIT_AMOUNT Debit,CREDIT_AMOUNT Credit,CARD_NUMB \"Card Number\" , DESCRIPTION Description FROM ALLTRANSACTIONS";
var returntable = dal.ReadData(sqlQuery);
DataRow ds = returntable.NewRow();
var dtExcelData = returntable;
So my datatable looks like this,
Then i read records from excel sheet
OleDbConnection con = null;
if (ext == ".xls")
{
con = new OleDbConnection(#"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + filepath + ";Extended Properties=Excel 8.0;");
}
else if (ext == ".xlsx")
{
con = new OleDbConnection(#"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + filepath + ";Extended Properties=\"Excel 12.0;IMEX=2;HDR=NO;TypeGuessRows=0;ImportMixedTypes=Text\"");
}
con.Open();
dt = con.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, null);
string getExcelSheetName = dt.Rows[0]["Table_Name"].ToString();
//OleDbCommand ExcelCommand = new OleDbCommand(#"SELECT * FROM [" + getExcelSheetName + #"]", con);
OleDbCommand ExcelCommand = new OleDbCommand("SELECT F1, F2, F3, F4, F5,F6,F7,F8,F9,F10 FROM [Sheet1$]", con);
OleDbDataAdapter ExcelAdapter = new OleDbDataAdapter(ExcelCommand);
try
{
ExcelAdapter.Fill(dtExcelData); //Here I give the datatable which i made previously
}
catch (Exception ex)
{
//lblAlert2.CssClass = "message-error";
//lblAlert2.Text = ex.Message;
}
It reads successfully and fill data in datatable but creating its own column in data table like F1 to F10 how can i move this data to exactly match with my defined columns in datatable
How Will i manage this to not create other columns (f1,f2..f10)
any workaround will be appreciable or Please explain what i am doing wrong and how can i achieve this.
UPDATE :
My Excel file looks like this
The Microsoft.ACE.OLEDB.12.0 driver will handle both types of excel spreadsheets and using the same Extended Properties. i.e. "Excel 12.0" will open both .xls and .xlsx.
Leave the HDR=NO as OLEDB expects them in the first row and they are actually in row 11.
Sadly "TypeGuessRows=0;ImportMixedTypes=Text" is completely ignored by Microsoft.ACE.OLEDB.12.0, you've got to play around with the registry (yuk). Change your IMEX=2 to IMEX=1 to ensure that mixed data types as handled as text.
Change back to using "Select * From [Sheet1$]" and then I'm afriad that you are going to have to handle the source data manually.
OleDbConnection con = new OleDbConnection(#"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + filepath + ";Extended Properties=\"Excel 12.0;IMEX=1;HDR=NO\"");
con.Open();
DataTable dt = con.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, null);
string getExcelSheetName = (string)dt.Rows[0]["Table_Name"];
DataTable xlWorksheet = new DataTable();
xlWorksheet.Load(new OleDbCommand("Select * From [" + getExcelSheetName + "]", con).ExecuteReader());
//More than 11 rows implies 11 header rows and at least 1 data row
if (xlWorksheet.Rows.Count > 11 & xlWorksheet.Columns.Count >= 10)
{
for (int nRow = 11; nRow < xlWorksheet.Rows.Count; nRow++)
{
DataRow returnRow = returntable.NewRow();
for (int nColumn = 0; nColumn < 10; nColumn++)
{
//Note you will probably get conversion problems here that you will have to handle
returnRow[nColumn] = xlWorksheet.Rows[nRow].ItemArray[nColumn];
}
returntable.Rows.Add(returnRow);
}
}
I'm guessing you simply want to add the excel data into your ALLTRANSACTION table? You don't specify but it seems the likely outcome of this. If so this is a terrible way to do it. You don't need to read the whole table into memory append data and then update the database. All you need to do is read the excel file and insert the data to the Oracle table.
Some thoughts, your returntable will contain data so if you just want the structure of the table then add a "Where RowNum=0" to the Select statement. To add the data to your Oracle Database you could 1) Convert to using the Oracle Data Provider (ODP) and then use using OracleBulkCopy Class or 2) simply modify the above to insert row by row as you read the data. As long as you don't have a LOT of data in your Excel spreadsheet it will work just fine. Having said that a Million rows is a LOT so perhaps not the best option. You will need to validate the input as Excel is not the best data source really.
I have a piece of code where I want to extract values from the A column of an Excel sheet. Right now, here is the code I'm using and having an issue with:
m_connString = "Provider = Microsoft.ACE.OLEDB.12.0; Data Source = " + m_source + "; Extended properties = 'Excel 12.0; HDR = NO; IMEX = 1;';";
using (OleDbConnection conn = new OleDbConnection(m_connString))
{
conn.Open();
DataTable dt = conn.GetOleDbSchemaGuid(Tables, null);
DataSet ds = new DataSet();
string defaultSheet = ExcelSheets.Rows[0]["TABLE_NAME"].ToString();
OleDbCommand comm = new OleDbCommand("SELECT * FROM [" + defaultSheet + "]", conn);
OleDbDataAdapter adapter = new OleDbDataAdapter(comm);
// Bug appears here
adapter.fill(ds)
// Fill a List<string> with the data found
for (int r = 0; r < ds.Tables[0].Row.Count; r++)
{
m_list.Add(ds.Tables[0].Rows[r][0].ToString();
}
}
What is happening is that, if I have an Excel file with the following content in the A column:
Row1
Row2
Row3
...
RowX
...What I end up getting is all values except for the first value (Row1). It turns out that Row1 is being used as a column name(?) for the DataSet's table. However, I don't want there to be any column names or headers, and I specifically state this in the connection string.
How can I prevent this behavior so I can have all my data placed in the List? Or, failing that, how can I work around this issue and extract Row1 from that DataSet?
Check Microsoft Reference to understand how Extended Properties of the connection HDR=NO works:
Column headings: By default, it is assumed that the first row of your
Excel data source contains columns headings that can be used as field
names. If this is not the case, you must turn this setting off, or
your first row of data "disappears" to be used as field names. This is
done by adding the optional HDR= setting to the Extended Properties of
the connection string. The default, which does not need to be
specified, is HDR=Yes. If you do not have column headings, you need to
specify HDR=No; the provider names your fields F1, F2, etc.
Here is example:
Excel File Data (test.xlsx):
Code:
string m_source = "test.xlsx";
string m_connString = #"Provider = Microsoft.ACE.OLEDB.12.0;
Data Source = " + m_source + #";
Extended properties = 'Excel 12.0;
HDR= NO;
IMEX = 1;';";
using (OleDbConnection conn = new OleDbConnection(m_connString))
{
conn.Open();
string squery = "SELECT f1, f2, f3 FROM [Sheet1$]";
OleDbCommand comm = new OleDbCommand(squery, conn);
OleDbDataAdapter adapter = new OleDbDataAdapter(comm);
DataSet ds = new DataSet();
adapter.Fill(ds);
}
DataSet Visualizer:
Try this way:
DataTable table = ds.Tables[0];
foreach (DataColumn column in table.Columns)
{
string cName = table.Rows[0][column.ColumnName].ToString();
if (!table.Columns.Contains(cName) && cName != "")
{
column.ColumnName = cName;
}
}
change the sql to something along the lines of this
"SELECT * FROM [" + defaultSheet + "] Except Select Top(1)"
I am using following code to update an Excel file into SQL Server. Code is working but it is not able to insert first row into table.
OleDbConnection OleDb = new OleDbConnection(ConnectionString);
OleDbCommand OleDbCmm = new OleDbCommand(Query,OleDb);
OleDbDataReader OleDbdr;
OleDb.Open();
if (OleDb.State == ConnectionState.Open)
{
OleDbdr = OleDbCmm.ExecuteReader();
SqlBulkCopy BulkCopy = new SqlBulkCopy(ConfigurationManager.ConnectionStrings["connstring"].ToString());
BulkCopy.DestinationTableName = "TempTable";
if (OleDbdr.Read())
{
BulkCopy.WriteToServer(OleDbdr);
}
}
OleDb.Close();
Even I was facing same problem, this is because I was using Read() Method like below.
while (dr.Read()) {
bulkcopy.WriteToServer(dr);
}
Solution to above problem is remove dr.Read() method and while loop
use
bulkcopy.WriteToServer(dr)
without any condition and Read() Method.
One possible reason for this may be that you've indicated in your connection string that first row contains column names (HDR=YES) thus that row is not treated as containing data.
EDIT
Another possible reason for this would be the call to OleDbDataReader.Read() method before passing the reader to SqlBulkCopy object. MSDN states:
The copy operation starts at the next available row in the reader. Most of the time, the reader was just returned by ExecuteReader or a similar call, so the next available row is the first row.
Thus, in your case you should not call OleDbdr.Read() because this advances the reader to the first row; you should let BulkCopy call Read() and it will start reading from the first row.
Your code should be:
OleDbdr = OleDbCmm.ExecuteReader();
SqlBulkCopy BulkCopy = new SqlBulkCopy(ConfigurationManager.ConnectionStrings["connstring"].ToString());
BulkCopy.DestinationTableName = "TempTable";
BulkCopy.WriteToServer(OleDbdr);
You need to set header for the Excel to sql as I have done in my
string conn = "Provider=Microsoft.Jet.OLEDB.4.0;" +
"Data Source=" + Your D S+ ";" +
"Extended Properties=Excel 8.0;";
OleDbConnection sSourceConnection = new OleDbConnection(conn);
using (sSourceConnection)
{
DataTable dtExcelData = new DataTable();
string[] SheetNames = GetExcelSheetNames(strFileName);
string[] preColumnHeader = new string[]{ "CarrierId", "StateId", "TerrCd", "ProgramId", "ClassId",
"PremTypeID","Limit50_100", "Limit100_100", "Limit100_200", "Limit300_300", "Limit300_600",
"Limit500_500","Limit500_1mil", "Limit1mil_1mil", "Limit1mil_2mil", "OtherParameter" };
sSourceConnection.Open();
string strQuery = string.Empty;
strQuery = "SELECT * FROM [" + SheetNames[0] + "]";
OleDbDataAdapter oleDA = new OleDbDataAdapter(strQuery, sSourceConnection);
oleDA.Fill(dtExcelData);
sSourceConnection.Close();
string[] colName = new string[dtExcelData.Columns.Count];
int i = 0;
foreach (DataColumn dc in dtExcelData.Columns)
{
colName[i] = dc.ColumnName;
i++;
}
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connStr))
{
bulkCopy.DestinationTableName = "tbl_test";
bulkCopy.WriteToServer(dtExcelData);
}
}