at work I've used closedXML in the past to go from a datatable or dataset to excel extremely quickly without looping. Now I need to go the other way but the only documentation I can find on closed XMl or anything else to go from excel to datatable is to loop. I can't imagine with the current demand for speed, the large amounts of data that can go into excel and the widespread use of Office that nobody has figured out a faster way than looping.
Is there a way in closed XML or another reasonably sized, safe library that quickly moves excel to datables or other system.data objects such as datasets without looping?
You can use a plain OleDb connection to read data from a worksheet into a DataTable:
string strExcelConn = #"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=c:\filename.xlsx;Extended Properties='Excel 12.0 Xml;HDR=YES;'";
using (OleDbConnection connExcel = new OleDbConnection(strExcelConn))
{
string selectString = "SELECT * FROM [CA$A1:D500]";
using (OleDbCommand cmdExcel = new OleDbCommand(selectString,connExcel))
{
cmdExcel.Connection = connExcel;
connExcel.Open();
DataTable dt=new DataTable();
OleDbDataAdapter adp = new OleDbDataAdapter();
adp.SelectCommand = cmdExcel;
adp.FillSchema(dt, SchemaType.Source);
adp.Fill(dt);
int range=dt.Columns.Count;
int row = dt.Rows.Count;
}
}
Related
Sorry for the vague title but I'm not sure how best to word it.
Basically, the issue that I'm having is that I initially was using an oledbconnection in C# to initiate a connection to a spreadsheet, query it, and load the results into a .net datatable. Unfortunately, it maxes out at 255 fields which I have around 600.
So, what I did instead was I created a dataset and tried to load 4 separate data tables with separate queries. Now for some reason, what's crazy to me is, if I load lets say, the first data table with 190 fields, and then I go on to query the spreadsheet again, if I go over that 250 mark (which I'd have 60 left), I get the following error:
An exception of type 'System.Data.OleDb.OleDbException' occurred in System.Data.dll but was not handled in user code
Additional information: No value given for one or more required parameters.
If I reduce the amount of fields in the second table to equal less than 250 though, it works. Anyway, is there a way that I can somehow clear the cache of the oledbconnection to somehow drop the first 190 fields or whatever is holding the result of the excel query so I can move on to the next? I tried doing a data adapter dispose but that didn't work, same issue. If I do a connection.dispose, I have to re-initialize the connection anyway. Is there a way I can still do this without having to drop the connection? Code below:
OleDbConnection cnn = new OleDbConnection(Settings.ExcelCN);
OleDbCommand fillT1 = new OleDbCommand(Settings.Excel_Proj_Select_1, cnn);
OleDbCommand fillT2 = new OleDbCommand(Settings.Excel_Proj_Select_2, cnn);
OleDbCommand fillT3 = new OleDbCommand(Settings.Excel_Proj_Select_3, cnn);
OleDbCommand fillT4 = new OleDbCommand(Settings.Excel_Proj_Select_4, cnn);
OleDbCommand updateCnt = new OleDbCommand(Settings.Excel_Update_Count_Select, cnn);
cnn.Open();
OleDbDataAdapter adp1 = new OleDbDataAdapter(fillT1);
OleDbDataAdapter adp2 = new OleDbDataAdapter(fillT2);
OleDbDataAdapter adp3 = new OleDbDataAdapter(fillT3);
OleDbDataAdapter adp4 = new OleDbDataAdapter(fillT4);
DataTable dt1 = new DataTable();
DataTable dt2 = new DataTable();
DataTable dt3 = new DataTable();
DataTable dt4 = new DataTable();
DataSet ds1 = new DataSet();
adp1.Fill(dt1);
ds1.Tables.Add(dt1);
adp2.Fill(dt2);
ds1.Tables.Add(dt2);
adp3.Fill(dt3);
ds1.Tables.Add(dt3);
adp4.Fill(dt4);
ds1.Tables.Add(dt4);
int rowcount = updateCnt.ExecuteNonQuery();
cnn.Close();
i just was just wondering, how do i import large excel files into mysql with c#? My coding experience isn't great and i was hoping if there's anyone out there who could give me some rough idea to start on it. So far, i was able to load excel files into datagridview with the following codes:
string PathConn = " Provider=Microsoft.JET.OLEDB.4.0;Data Source=" + pathTextBox.Text + ";Extended Properties =\"Excel 8.0;HDR=Yes;\";";
OleDbConnection conn = new OleDbConnection(PathConn);
conn.Open();
OleDbDataAdapter myDataAdapter = new OleDbDataAdapter("Select * from [" + loadTextBox.Text + "$]", conn);
table = new DataTable();
myDataAdapter.Fill(table);
but after that, i don't know how i could extract the information and save it into mysql database. Assuming i have a empty scheme created before, how do i work on uploading excel files into mysql? thanks.
I think you would then need to loop over the items in the datatable and do something with them (maybe an insert statement to your DB)
like so
foreach(DataRow dr in table.Rows)
{
string s = dr[0].ToString() // this will be the first column in the datatabl as they are zero indexed
}
this is what i do in data migration scenarios from one SQL Server to another or DataFiles to SQL:
Create the new Table on the destination SQL Server (Column names, Primary Key etc.)
Load existing Data into a DataTable (Thats what you did already)
Now Query the new Table with the DataAdapter into another DataTable (Same as you did with the excel file except you now query the SQL Table.)
Load OldData from 'table' into 'newTable' using DataTable Method "Load()"
string PathConn = (MYSQL Connection String goes here)
OleDbConnection conn = new OleDbConnection(PathConn);
conn.Open();
OleDbDataAdapter myDataAdapter = new OleDbDataAdapter("Select * from [" + loadTextBox.Text + "$]", conn);
newTable = new DataTable();
myDataAdapter.Fill(newTable);
Now use the Load() Method on the new table:
newTable.Load(table.CreateDataReader(), <Specify LoadOption here>)
Matching columns will be imported into the new DataTable. (You can ensure the mapping through using Aliases in the select statements)
After Loading the existing Data into the new Table you will be able to use an DataAdapter to write the changes back to database.
Example for writing data back: ConnString - connection String for DB,
SelectStmt (can use the same as you did on the empty Table before) and provide the newTable as dtToWrite
public static void writeDataTableToServer(string ConnString, string selectStmt, DataTable dtToWrite)
{
using (OdbcConnection odbcConn = new OdbcConnection(ConnString))
{
odbcConn.Open();
using (OdbcTransaction trans = odbcConn.BeginTransaction())
{
using (OdbcDataAdapter daTmp = new OdbcDataAdapter(selectStmt, ConnString))
{
using (OdbcCommandBuilder cb = new OdbcCommandBuilder(daTmp))
{
try
{
cb.ConflictOption = ConflictOption.OverwriteChanges;
daTmp.UpdateBatchSize = 5000;
daTmp.SelectCommand.Transaction = trans;
daTmp.SelectCommand.CommandTimeout = 120;
daTmp.InsertCommand = cb.GetInsertCommand();
daTmp.InsertCommand.Transaction = trans;
daTmp.InsertCommand.CommandTimeout = 120;
daTmp.UpdateCommand = cb.GetUpdateCommand();
daTmp.UpdateCommand.Transaction = trans;
daTmp.UpdateCommand.CommandTimeout = 120;
daTmp.DeleteCommand = cb.GetDeleteCommand();
daTmp.DeleteCommand.Transaction = trans;
daTmp.DeleteCommand.CommandTimeout = 120;
daTmp.Update(dtToWrite);
trans.Commit();
}
catch (OdbcException ex)
{
trans.Rollback();
throw ex;
}
}
}
}
odbcConn.Close();
}
}
Hope this helps.
Primary Key on the newTable is necessary, otherwise you might get a CommandBuilder exception.
BR
Therak
Your halfway there, You have obtained the information from the Excel spreadsheet and have it stored in a DataTable.
The first thing you need to do before you look to import a significant amount of data into SQL is validate what you have read in from the spreadsheets.
You have a few options, one of which is do something very similar to how you read in your data and that is use a SQLAdapter to perform am INSERT into the SQL Database. All your really needing to do in this case is create a new connection and write the INSERT command.
There are many example of doing this on here.
Another option which i would use, is LINQ to CSV (http://linqtocsv.codeplex.com/).
With this you can load all of your data into class objects which makes it easier to validate each object before you perform your INSERT into SQL.
If you have limited experience then use the SQLAdapter to connect to you DB.
Good Luck
I have been able to read a normal excel file via oledb, but the task file for my project contains various format issues, like multiple columns merged, a lot of cells under formulae among others, and as a result, the data set has been unable to store the data of excel file. If I manage to read contents of sheet- 'Input tab', cell numbers AF6-AF-24, my job is done, but the excel file is not getting parsed in excel at all, before i process the column out. Help please!
OleDbConnection connExcel = new OleDbConnection("Provider= Microsoft.ACE.OLEDB.12.0;Data Source=P:\\DummyTesterF.xlsx; Extended Properties=\"Excel 12.0;HDR=YES;\"");
OleDbCommand cmdExcel = new OleDbCommand();
OleDbDataAdapter Adapter = new OleDbDataAdapter();
cmdExcel.CommandText = String.Format("SELECT * From ['Input Tab$']");
cmdExcel.Connection = connExcel;
Adapter.SelectCommand = cmdExcel;
DataTable Table = new DataTable();
Adapter.Fill(Table);
Better switch to Office interop instead.
I have written a piece of code that reads excel data in data table through ado.net. Now i have seen a strange behavior, That the column data type in ms excel is specified as general, but if the 1st row of the excel of that particular column contains some text, ado.net is unable to read numbers in that particular column and if 1st row contains some numeric information ado.net is unable to read textual data of that particular column. here is the code i am using to read data from excel.
string excelConString = #"Provider=Microsoft.Jet.OLEDB.4.0;
Data Source=D:\A.xls;
Extended Properties=""Excel 8.0;HDR=YES;""";
var oleDbConnection = new OleDbConnection(excelConString);
var cmd = oleDbConnection.CreateCommand();
cmd.CommandText = "select * from [WorkSheet$] where ID>=1500";
OleDbDataAdapter da = new OleDbDataAdapter(cmd);
DataTable dt = new DataTable();
da.Fill(dt);
Any help would be appreciated.
Regards
Ahsan Iqbal
change your connectionstring to
string excelConString = #"Provider=Microsoft.Jet.OLEDB.4.0;
Data Source=D:\A.xls;
Extended Properties=""Excel 8.0;HDR=YES;IMEX=1;TypeGuessRows=0;ImportMixedTypes=Text"""
I'm opening an Excel worksheet into a DataTable using OleDb like this:
string select = string.Format("SELECT * FROM [{0}$]", worksheetName);
using (var con = new OleDbConnection(connectionString))
using (var adapter = new OleDbDataAdapter(select, con))
{
con.Open();
var dt = new DataTable();
adapter.Fill(dt);
con.Close();
return dt;
}
Then I loop through the rows of the DataTable reading various bits of data like this:
decimal charge;
bool isChargeReadable =
decimal.TryParse(row["Charge"].ToString(), out charge);
I discovered just now that my code was choking on cells with dollar amounts such as "$1100.00", which it can't parse to decimal. Not so surprising...except that this is code that was working perfectly before just now.
Further investigation revealed that if I run this code while the workbook is open, it sees one of those cells as "1100". If I run it while the workbook is closed, it sees "$1100.00".
Why is this happening? Obviously I'll have to rework my code to function while the workbook is closed, but why would it make a difference? I would've thought it would just be reading the saved workbook.
The connection string I'm using is this...
"Provider=Microsoft.Jet.OLEDB.4.0;
Data Source={0};
Extended Properties=""Excel 8.0;HDR=NO;IMEX=1"";"
...where {0} is replaced by the Excel file name, of course.
I've found that my oledb operations work better in excel withOUT an IMEX=number set. Perhaps your problem is there?