I'm performing a conversion of DBF + DBT files into the SQL. I use Microsoft.Jet.OLEDB.4.0 connector for accessing the files and SqlConnector for writing data in the MS SQL, to improve performance I use SqlBulkCopy approach. Most of the files are converted fine, but on some the method SqlBulkCopy.WriteToServer throws an exception :
Record is deleted.
The search key was not found in any record.
The copy operation doesn't complete and I'm missing lots of records in the SQL.
Is there a way how to bypass this problem, or am I suppose to give up on SqlBulkCopy and copy row after row?
EDIT:
So I decided to PACK the database, however no luck so far. When I use vfpoledb for reading, it crashes even sooner because of some problem with casting decimals. So I'd like to use PACK first (with vfpoledb) and then the JetOleDb reader.Even though the PACK is executed,and I can see that the dbf and dbt files change, the reader.GetValues() is still throwing the same exception.
try
{
string file = #"f:\Elims\dsm\CPAGEMET.DBF";
string tableName = Path.GetFileNameWithoutExtension(file);
var dbfConnectionString = string.Format(#"Provider=Microsoft.Jet.OLEDB.4.0;Data Source={0};Extended Properties='dBASE III;DELETED=YES;HDR=NO;IMEX=1'", Path.GetDirectoryName(file));
var packConnString = string.Format(#"Provider=vfpoledb;Data Source={0};Collating Sequence=machine;", file);
OleDbConnection packConnector = new OleDbConnection(packConnString);
packConnector.Open();
OleDbCommand command = new OleDbCommand(string.Format("PACK {0}",tableName), packConnector);
var result = command.ExecuteNonQuery();
packConnector.Close();
OleDbConnection oleConnector = new OleDbConnection(dbfConnectionString);
oleConnector.Open();
string cmd = string.Format("SELECT * FROM [{0}]", tableName);
var oleDbCommand = new OleDbCommand(cmd, oleConnector);
OleDbDataReader dataReader = oleDbCommand.ExecuteReader();
object[] values = new object[dataReader.FieldCount];
int iRow = 0;
while (dataReader.Read())
{
iRow++;
Console.WriteLine("Row " + iRow);
dataReader.GetValues(values);
}
oleConnector.Close();
}
catch (Exception e)
{
Console.WriteLine(e.Message + e.StackTrace);
}
Thanks
It looks like some of the records are marked as deleted. Restore them back if you need these records, or delete them permanently (using PACK command) if you don't need these records.
So after some digging, I came out to final resolution, so I will just summarize my findings. The "record is deleted" exception was very misleading, as the problem never actually was a deleted record. There were 3 empty rows, that were actually triggering the exception. Once these got deleted, everything started to work, I didn't even have to pack the database. This applied to a scenario, where I used Microsoft.Jet.OLEDB connector.
I tried to use the vfpoledb connector, but I ran into other problem with decimal numbers. I wrote a fix inspired by this msdn discussion and not only everything started to work (so deleted and empty rows are successfully skipped), but also the import is now 15x faster than with the Jet connector (again using BulkCopy)
Related
I'm not the first to have these issues, and will list some reference posts below, but am still looking for a proper solution.
I need to call a stored procedure (Oracle 10g database) from a C# web service. The web server has an Oracle 9i client installed and I am using Microsofts System.Data.OracleClient.
The procedure takes an XML as a CLOB. When the XML was over 4000 Bytes (which is likely in a normal use case), I stumbled over the following error:
ORA-01460 - unimplemented or unreasonable conversion requested
I've found this, this and this post.
Further I found a promising workaround which doesn't call the stored procedure directly from C# but defines a piece of anonymous PL/SQL code instead. This code is run as an OracleCommand. The XML is embedded as a string literal and the procedure call is done from within that piece of code:
private const string LoadXml =
"DECLARE " +
" MyXML CLOB; " +
" iStatus INTEGER; " +
" sErrMessage VARCHAR2(2000); " +
"BEGIN " +
" MyXML := '{0}'; " +
" iStatus := LoadXML(MyXML, sErrMessage); " +
" DBMS_OUTPUT.ENABLE(buffer_size => NULL); " +
" DBMS_OUTPUT.PUT_LINE(iStatus || ',' || sErrMessage); " +
"END;";
OracleCommand oraCommand = new OracleCommand(
string.Format(LoadXml, xml), oraConnection);
oraCommand.ExecuteNonQuery();
Unfortunately, this approach now fails as soon as the XML is over 32 KBytes or so, which still is very likely in my application. This time the error stems from the PL/SQL compiler which says:
ORA-06550: line1, column 87: PLS-00172: string literal too long
After some research I conclude that it's simply not feasible to solve the problem with my second approach.
Following the above-mentioned posts I have the following two options.
Switch to ODP.NET (because it is supposed to be a bug in Microsoft's deprecated DB client)
Insert the CLOB into a table and make the stored proc read from there
(The first post said some clients are buggy, but mine (9i) does not fall in the mentioned range of 10g/11g versions.)
Can you confirm that these are the only two options left? Or is there another way to help me out?
Just to clarify: the XML won't eventually be saved in any table, but it is processed by the stored procedure which inserts some records in some table based on the XML contents.
My considerations about the two options:
Switching to ODP.NET is difficult because I have to install it on a web server on which I don't have system access so far, and because we might also want to deploy the piece of code on clients, so each client would have to install ODP.NET as part of the deployment.
The detour over a table makes the client code quite a bit more complicated and also takes quite some effort on the database adapting/extending the PL/SQL routines.
I found that there is another way to work around the problem! My fellow employee saved my day pointing me to this blog, which says:
Set the parameter value when
BeginTransaction has already been
called on the DbConnection.
Could it be simpler? The blog relates to Oracle.DataAccess, but it works just as well for System.Data.OracleClient.
In practice this means:
varcmd = new OracleCommand("LoadXML", _oracleConnection);
cmd.CommandType = CommandType.StoredProcedure;
var xmlParam = new OracleParameter("XMLFile", OracleType.Clob);
cmd.Parameters.Add(xmlParam);
// DO NOT assign the parameter value yet in this place
cmd.Transaction = _oracleConnection.BeginTransaction();
try
{
// Assign value here, AFTER starting the TX
xmlParam.Value = xmlWithWayMoreThan4000Characters;
cmd.ExecuteNonQuery();
cmd.Transaction.Commit();
}
catch (OracleException)
{
cmd.Transaction.Rollback();
}
In my case, chiccodoro's solution did not work. I'm using ODP.NET ( Oracle.DataAccess ).
For me the solution is using OracleClob object.
OracleCommand cmd = new OracleCommand("LoadXML", _oracleConnection);
cmd.CommandType = CommandType.StoredProcedure;
OracleParameter xmlParam = new OracleParameter("XMLFile", OracleType.Clob);
cmd.Parameters.Add(xmlParam);
//connection should be open!
OracleClob clob = new OracleClob(_oracleConnection);
// xmlData: a string with way more than 4000 chars
clob.Write(xmlData.ToArray(),0,xmlData.Length);
xmlParam.Value = clob;
try
{
cmd.ExecuteNonQuery();
}
catch (OracleException e)
{
}
I guess I just googled this for you to get cheap points, but there's a great explanation here:
http://www.orafaq.com/forum/t/48485/0/
Basically you cannot use more than 4000 chars in a string literal, and if you need to do more, you must use a stored procedure. Then, you are limited to 32KB at max so you have to "chunk" the inserts. Blech.
-Oisin
chiccodoro is right.
public static int RunProcedure(string storedProcName, IDataParameter[] parameters)
{
using (OracleConnection connection = new OracleConnection(connectionString))
{
int rowsAffected;
OracleCommand command = new OracleCommand(storedProcName, connection);
command.CommandText = storedProcName;
command.CommandType = CommandType.StoredProcedure;
foreach (OracleParameter parameter in parameters)
{
command.Parameters.Add(parameter);
}
connection.Open();
try
{
// start transaction
command.Transaction = connection.BeginTransaction();
rowsAffected = command.ExecuteNonQuery();
command.Transaction.Commit();
}
catch (System.Exception ex)
{
command.Transaction.Rollback();
throw ex;
}
connection.Close();
return rowsAffected;
}
}
I am trying to insert a large number of records into a variety of tables. I cannot fit all of the records into memory at once, so instead, I am using an IDataReader implementation to fit some of the data into memory and then dump it into the database with SqlBulkCopy.
The problem I am experiencing is that the second time I try to write to a table, SqlBulkCopy will succeed but fail to actually insert the records. I thought it was a transaction issue at first, but I disabled all transactions on my connection and am still seeing the same problem. I can also independently confirm the size of the tables before and after both inside the code and outside.
Here is a code snippet:
long before = GetCount(tableName);
DataServerConnection conn = GetConnection();
using (var batch = conn.HasTransaction ?
new SqlBulkCopy((SqlConnection)conn.IDbConnection, SqlBulkCopyOptions.Default, (SqlTransaction)conn.Transaction) :
new SqlBulkCopy((SqlConnection)conn.IDbConnection))
{
batch.DestinationTableName = tableName;
batch.WriteToServer(reader);
}
long after = GetCount(tableName);
if ((reader.Count + before) != after)
{
throw new Exception($"Not all records inserted: Before = {before}, After = {after}, Reader Count = {reader.Count}, Expected = {reader.Count + before}");
}
Any ideas what I am missing? GetCount(tableName) is doing a simple
SELECT COUNT(*) FROM [{tableName}]
reader is a basic IDataReader implementation that I have verified works other places on millions of records. GetConnection() is returning a wrapper for the connection which helps to prevent me from having to manage my connections constantly.
I'm not sure how your reader variable is declared so I'll tell you how I do it sometimes (and this doesn't mean it is the best way to do it).
First I declare a tableAdapter based on the dataSet I have:
DataSetTableAdapter.MyTableAdapter dataTable = new DataSetTableAdapter.MyTableAdapter();
Then, I create the connection.
SqlConnection sql = new SqlConnection(...);
And then the BulkCopy variable:
SqlBulkCopy insertData = new SqlBulkCopy(sql);
Once I have this, I start adding rows to the table Adapter like this:
dataTable.AddMyTableRow(...);
When I am finished, I do the bulk insertion:
sql.Open();
insertData.DestinationTableName = "MyTable";
insertData.WriteToServer(dataTable);
sql.Close();
Let me know if this helps you.
Having problems with Visual Fox Pro ODBC drivers (and OLE DB) the code I'm using to dynamically get all the column names for a DBF table works fine for small datasets. But if I try getting the table schema using the below code for a large table then it can take 60 seconds to get the schema.
OdbcConnection conn = new OdbcConnection(#"Dsn=Boss;sourcedb=u:\32BITBOSS\DATA;sourcetype=DBF;exclusive=No;backgroundfetch=Yes;collate=Machine;null=Yes;deleted=Yes");
OdbcCommand cmd = new OdbcCommand("Select * from " + listBox1.SelectedItem.ToString(), conn);
conn.Open();
try
{
OdbcDataReader reader = cmd.ExecuteReader();
DataTable dt = reader.GetSchemaTable();
listBox2.Items.Clear();
foreach (DataRow row in dt.Rows)
{
listBox2.Items.Add(row[0].ToString());
}
conn.Close();
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
conn.Close();
}
Now I get that this is due to the Select command dragging all 500,000 records or so into the data reader before it can then extract the column names. But visual fox pro is pretty rubbish at limiting the record return.
I ideally want to limit the return to just 1 record, but on the assumption I don't know any of the column names to utilise a WHERE clause this doesn't work.
VFP can utilise the TOP SQL command, but to do that you need an ORDER BY to go with it, and since I don't have the any of the column names to utilise, this doesn't work either.
So am a bit stumped trying to think of a clever way to dynamically get the table schema without the slow down in VFP.
I don't think database conversions on the fly will be any faster. Anyone got any clever ideas (apart from changing the database system, it's inherited in the job and I currently have to work with it :) )?
Change this line:
OdbcCommand cmd = new OdbcCommand("Select * from " + listBox1.SelectedItem.ToString(), conn);
To this line:
OdbcCommand cmd = new OdbcCommand(string.Format("Select * from {0} where 1 = 0", listBox1.SelectedItem.ToString()), conn);
This will bring back zero records, but fill the schema. Now that you have the schema you can build a filter further.
I'm not the first to have these issues, and will list some reference posts below, but am still looking for a proper solution.
I need to call a stored procedure (Oracle 10g database) from a C# web service. The web server has an Oracle 9i client installed and I am using Microsofts System.Data.OracleClient.
The procedure takes an XML as a CLOB. When the XML was over 4000 Bytes (which is likely in a normal use case), I stumbled over the following error:
ORA-01460 - unimplemented or unreasonable conversion requested
I've found this, this and this post.
Further I found a promising workaround which doesn't call the stored procedure directly from C# but defines a piece of anonymous PL/SQL code instead. This code is run as an OracleCommand. The XML is embedded as a string literal and the procedure call is done from within that piece of code:
private const string LoadXml =
"DECLARE " +
" MyXML CLOB; " +
" iStatus INTEGER; " +
" sErrMessage VARCHAR2(2000); " +
"BEGIN " +
" MyXML := '{0}'; " +
" iStatus := LoadXML(MyXML, sErrMessage); " +
" DBMS_OUTPUT.ENABLE(buffer_size => NULL); " +
" DBMS_OUTPUT.PUT_LINE(iStatus || ',' || sErrMessage); " +
"END;";
OracleCommand oraCommand = new OracleCommand(
string.Format(LoadXml, xml), oraConnection);
oraCommand.ExecuteNonQuery();
Unfortunately, this approach now fails as soon as the XML is over 32 KBytes or so, which still is very likely in my application. This time the error stems from the PL/SQL compiler which says:
ORA-06550: line1, column 87: PLS-00172: string literal too long
After some research I conclude that it's simply not feasible to solve the problem with my second approach.
Following the above-mentioned posts I have the following two options.
Switch to ODP.NET (because it is supposed to be a bug in Microsoft's deprecated DB client)
Insert the CLOB into a table and make the stored proc read from there
(The first post said some clients are buggy, but mine (9i) does not fall in the mentioned range of 10g/11g versions.)
Can you confirm that these are the only two options left? Or is there another way to help me out?
Just to clarify: the XML won't eventually be saved in any table, but it is processed by the stored procedure which inserts some records in some table based on the XML contents.
My considerations about the two options:
Switching to ODP.NET is difficult because I have to install it on a web server on which I don't have system access so far, and because we might also want to deploy the piece of code on clients, so each client would have to install ODP.NET as part of the deployment.
The detour over a table makes the client code quite a bit more complicated and also takes quite some effort on the database adapting/extending the PL/SQL routines.
I found that there is another way to work around the problem! My fellow employee saved my day pointing me to this blog, which says:
Set the parameter value when
BeginTransaction has already been
called on the DbConnection.
Could it be simpler? The blog relates to Oracle.DataAccess, but it works just as well for System.Data.OracleClient.
In practice this means:
varcmd = new OracleCommand("LoadXML", _oracleConnection);
cmd.CommandType = CommandType.StoredProcedure;
var xmlParam = new OracleParameter("XMLFile", OracleType.Clob);
cmd.Parameters.Add(xmlParam);
// DO NOT assign the parameter value yet in this place
cmd.Transaction = _oracleConnection.BeginTransaction();
try
{
// Assign value here, AFTER starting the TX
xmlParam.Value = xmlWithWayMoreThan4000Characters;
cmd.ExecuteNonQuery();
cmd.Transaction.Commit();
}
catch (OracleException)
{
cmd.Transaction.Rollback();
}
In my case, chiccodoro's solution did not work. I'm using ODP.NET ( Oracle.DataAccess ).
For me the solution is using OracleClob object.
OracleCommand cmd = new OracleCommand("LoadXML", _oracleConnection);
cmd.CommandType = CommandType.StoredProcedure;
OracleParameter xmlParam = new OracleParameter("XMLFile", OracleType.Clob);
cmd.Parameters.Add(xmlParam);
//connection should be open!
OracleClob clob = new OracleClob(_oracleConnection);
// xmlData: a string with way more than 4000 chars
clob.Write(xmlData.ToArray(),0,xmlData.Length);
xmlParam.Value = clob;
try
{
cmd.ExecuteNonQuery();
}
catch (OracleException e)
{
}
I guess I just googled this for you to get cheap points, but there's a great explanation here:
http://www.orafaq.com/forum/t/48485/0/
Basically you cannot use more than 4000 chars in a string literal, and if you need to do more, you must use a stored procedure. Then, you are limited to 32KB at max so you have to "chunk" the inserts. Blech.
-Oisin
chiccodoro is right.
public static int RunProcedure(string storedProcName, IDataParameter[] parameters)
{
using (OracleConnection connection = new OracleConnection(connectionString))
{
int rowsAffected;
OracleCommand command = new OracleCommand(storedProcName, connection);
command.CommandText = storedProcName;
command.CommandType = CommandType.StoredProcedure;
foreach (OracleParameter parameter in parameters)
{
command.Parameters.Add(parameter);
}
connection.Open();
try
{
// start transaction
command.Transaction = connection.BeginTransaction();
rowsAffected = command.ExecuteNonQuery();
command.Transaction.Commit();
}
catch (System.Exception ex)
{
command.Transaction.Rollback();
throw ex;
}
connection.Close();
return rowsAffected;
}
}
I'm trying to reindex a table in a simple database that I created using SQLite.NET and VS2008. I need to reindex tables after every DELETE command and here is the code snippet I have written (it does not work):
SQLiteCommand currentCommand;
String tempString = "REINDEX tf_questions";
//String tempString = "REINDEX [main].tf_questions";
//String tempString = "REINDEX main.tf_questions";
currentCommand = new SQLiteCommand(myConnection);
currentCommand.CommandText = tempString;
currentCommand.ExecuteNonQuery()
When run within my program, the code produces no errors, but it also doesn't reindex the "tf_questions" table. In the above example, you will also see the other query strings I've tried that also don't work.
Please help,
Thanks
I figured out a workaround for my problem. Consider the following code:
SQLiteCommand currentCommand;
String tempString;
String currentTableName = "tf_questions";
DataTable currentDataTable;
SQLiteDataAdapter myDataAdapter;
currentDataTable = new DataTable();
myDataAdapter = new SQLiteDataAdapter("SELECT * FROM " + currentTableName, myConnection);
myDataAdapter.Fill(currentDataTable);
//"tf_questions" is the name of the table
//"question_id" is the name of the primary key column in "tf_questions" (set to auto inc.)
//"myConnection" is and already open SQLiteConnection pointing to the db file
for (int i = currentDataTable.Rows.Count-1; i >=0 ; i--)
{
currentCommand = new SQLiteCommand(myConnection);
tempString = "UPDATE "+ currentTableName +"\nSET question_id=\'"+(i+1)+"\'\nWHERE (question_id=\'" +
currentDataTable.Rows[i][currentDataTable.Columns.IndexOf("question_id")]+"\')";
currentCommand.CommandText = tempString;
if( currentCommand.ExecuteNonQuery() < 1 )
{
throw new Exception("There was an error executing the REINDEX protion of the code...");
}
}
This code works, though I would have liked to have used the built-in SQL REINDEX command.
If you are doing this for the sake of performance, consider this answer:
REINDEX does not help. REINDEX is only
needed if you change a collating
sequence.
After a lot of INSERTs and DELETEs,
you can sometimes get slightly better
performance by doing a VACUUM. VACUUM
improves locality of reference
slightly.