Develop Reference

SqlBulkCopy WriteToServer throws an exception Record is deleted

I'm performing a conversion of DBF + DBT files into the SQL. I use Microsoft.Jet.OLEDB.4.0 connector for accessing the files and SqlConnector for writing data in the MS SQL, to improve performance I use SqlBulkCopy approach. Most of the files are converted fine, but on some the method SqlBulkCopy.WriteToServer throws an exception :
Record is deleted.
The search key was not found in any record.
The copy operation doesn't complete and I'm missing lots of records in the SQL.
Is there a way how to bypass this problem, or am I suppose to give up on SqlBulkCopy and copy row after row?
EDIT:
So I decided to PACK the database, however no luck so far. When I use vfpoledb for reading, it crashes even sooner because of some problem with casting decimals. So I'd like to use PACK first (with vfpoledb) and then the JetOleDb reader.Even though the PACK is executed,and I can see that the dbf and dbt files change, the reader.GetValues() is still throwing the same exception.
try
{
string file = #"f:\Elims\dsm\CPAGEMET.DBF";
string tableName = Path.GetFileNameWithoutExtension(file);
var dbfConnectionString = string.Format(#"Provider=Microsoft.Jet.OLEDB.4.0;Data Source={0};Extended Properties='dBASE III;DELETED=YES;HDR=NO;IMEX=1'", Path.GetDirectoryName(file));
var packConnString = string.Format(#"Provider=vfpoledb;Data Source={0};Collating Sequence=machine;", file);
OleDbConnection packConnector = new OleDbConnection(packConnString);
packConnector.Open();
OleDbCommand command = new OleDbCommand(string.Format("PACK {0}",tableName), packConnector);
var result = command.ExecuteNonQuery();
packConnector.Close();
OleDbConnection oleConnector = new OleDbConnection(dbfConnectionString);
oleConnector.Open();
string cmd = string.Format("SELECT * FROM [{0}]", tableName);
var oleDbCommand = new OleDbCommand(cmd, oleConnector);
OleDbDataReader dataReader = oleDbCommand.ExecuteReader();
object[] values = new object[dataReader.FieldCount];
int iRow = 0;
while (dataReader.Read())
{
iRow++;
Console.WriteLine("Row " + iRow);
dataReader.GetValues(values);
}
oleConnector.Close();
}
catch (Exception e)
{
Console.WriteLine(e.Message + e.StackTrace);
}
Thanks

It looks like some of the records are marked as deleted. Restore them back if you need these records, or delete them permanently (using PACK command) if you don't need these records.

So after some digging, I came out to final resolution, so I will just summarize my findings. The "record is deleted" exception was very misleading, as the problem never actually was a deleted record. There were 3 empty rows, that were actually triggering the exception. Once these got deleted, everything started to work, I didn't even have to pack the database. This applied to a scenario, where I used Microsoft.Jet.OLEDB connector.
I tried to use the vfpoledb connector, but I ran into other problem with decimal numbers. I wrote a fix inspired by this msdn discussion and not only everything started to work (so deleted and empty rows are successfully skipped), but also the import is now 15x faster than with the Jet connector (again using BulkCopy)

how to update datagridview to database using c#?

I have a datagridview bound to a datasouce, bound to a datatable.
I want to create a button that when I click it, the database will be updated with the new parameters in the datagridview.
I've read some stuff about TableAdapter but I cant really find some good examples and explanations.
So if someone can give some information about tableadapter, it will help me a lot.
Also, if you think that you have a better solution for me regarding update the database, i'll be glad to know about that.
EDIT:
ok so i'm trying to use the mysqlcommandbuilder.
My code now looks like this:
MySqlDataAdapter da = new MySqlDataAdapter();
da.SelectCommand = new MySqlCommand("SELECT * from setups", sql_Class.myConnection);
MySqlCommandBuilder cb = new MySqlCommandBuilder(da);
cb.GetUpdateCommand();
da.Update(dt);
so now my error is in the da.Update(dt) and it says: "input string was not in the correct format".
I've created a loop that runs on all the parameters in the sql query and all the parameters values are null.
for (int i = 0; i <= cb.GetUpdateCommand().Parameters.Count - 1; i++)
{
Console.WriteLine(cb.GetUpdateCommand().Parameters[i].ParameterName + " " + cb.GetUpdateCommand().Parameters[i].Value);
}
I have values in the datatable, i've double checked it, but someone the parameters are null.
Any ideas why?

Managed to update my database in some "old fashion" method.
I'm just running on every single row in the datagridview and updates it using executenonquery.
I get all the parameters manually.
Example:
int setup_id = Convert.ToInt16(stam.Cells["setup_id"].Value);
string setup_name = stam.Cells["setup_name"].Value.ToString();
string mySqlQuery = "UPDATE setups SET setup_name='"+setup_name+"' WHERE setup_id="+setup_id;
MySqlCommand cm = new MySqlCommand(mySqlQuery, sql_Class.myConnection);
cm.ExecuteNonQuery();

fast/efficient way to run a query in Access based on datatable rows?

I have a datatable that may have 1000 or so rows in it. I need to go thru the datatable row by row, get the value of a column, run a query (Access 2007 DB) and update the datatable with the result. Here's what I have so far, which works:
String FilePath = "c:\\MyDB.accdb";
string QueryString = "SELECT MDDB.NDC, MDDB.NDC_DESC "
+ "FROM MDDB_MASTER AS MDDB WHERE MDDB.NDC = #NDC";
OleDbConnection strAccessConn = new OleDbConnection(string.Format("Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + FilePath));
strAccessConn.Open();
OleDbDataReader reader = null;
int rowcount = InputTable.Rows.Count; //InputTable is the datatable
int count = 0;
while (count < rowcount)
{
string NDC = InputTable.Rows[count]["NDC"].ToString();
//NDC is a column in InputTable
OleDbCommand cmd = new OleDbCommand(QueryString, strAccessConn);
cmd.Parameters.Add("#NDC", OleDbType.VarChar).Value = NDC;
reader = cmd.ExecuteReader();
while (reader.Read())
{
//update the NDCDESC column with the query result
//the query should only return 1 line
dataSet1.Tables["InputTable"].Rows[count]["NDCDESC"] = reader.GetValue(1).ToString();
}
dataGridView1.Refresh();
count++;
}
strAccessConn.Close();
However this seems very inefficient since the query needs to run one time for each row in the datatable. Is there a better way?

You're thinking of an update query. You don't actually have to go over every row one by one. SQL is a set based language, so you only have to write a single statement that it should do for all rows.
Do this:
1) Create > Query Design
2) Close the dialog that selects tables
3) Make sure you're in sql mode (top left corner)
4) Paste this:
UPDATE INPUTTABLE
INNER JOIN MDDB_MASTER ON INPUTTABLE.NDC = MDDB_MASTER.NDC
SET INPUTTABLE.NDCDESC = [MDDB_MASTER].[NDC_DESC];
5) Switch to design mode to see what happens. You may have to correct Input table, I couldn't find it's name. I'm assuming they;re both in the same database.
You'll see the query type is now an update query.
You can run this text through cmd.ExecuteNonQuery(sql) and the whole thing should run very quickly. If it doesn't you'll need an index on one of the tables;
THis works by joining the two table on NDC and then copying the NDC_DESC over from MDDB_MASTER to the inputtable.

I missed the part about InputTable coming from Excel.
For better speed, instead of executing the query in Access over and over, you can get all rows from MDDB_MASTER into a datatable in one select statement:
SELECT MDDB.NDC, MDDB.NDC_DESC FROM MDDB_MASTER
And then use the DataTable.Select method to filter the right row.
mddb_master.Select("NDC = '" + NDC +'")
This will be done in memory and should be much faster than all the round trips you have now. Especially over the network these round trips are expensive. 225k rows should be only a few MB (roughly a JPEG image) so that shouldn't be an issue.

You could use the "IN" clause to build a bigger query such as:
string QueryString = "SELECT MDDB.NDC, MDDB.NDC_DESC "
+ "FROM MDDB_MASTER AS MDDB WHERE MDDB.NDC IN (";
int rowcount = InputTable.Rows.Count; //InputTable is the datatable
int count = 0;
while (count < rowcount)
{
string NDC = InputTable.Rows[count]["NDC"].ToString();
QueryString += (count == 0 ? "" : ",") + "'" + NDC + "'";
}
QueryString += ")";
You can optimize that with StringBuilders since that could be a lot of strings but that's a job for you. :)
Then in a single query, you would get all the NDC descriptions you need and avoid performing 1000 queries. You would then roll through the reader, find values in the InputTable, and update them. Of course, in this case, you're looping through the InputTable multiple times but it might be a better option. Especially if yor InputTable could hold duplicate NDC values.
Also, note that you have a OleDbDataReader leak in your code. You keep reassigning the reader reference to a new instance of a reader before disposing of the old reader. Same with commands. You keep instantiating a new command but are not disposing of it properly.

SQL Server CE WHERE statement behaving wrong? very confused

I am developing a windows mobile app. Right now I am just testing that it can correctly query the local SQL Server CE database. It works fine until I put a WHERE statement in.
Here is my code:
private void buttonStart_Click(object sender, EventArgs e)
{
System.Data.SqlServerCe.SqlCeConnection conn = new System.Data.SqlServerCe.SqlCeConnection(
("Data Source=" + (System.IO.Path.Combine(System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase), "ElectricReading.sdf") + ";Max Database Size=2047")));
try
{
// Connect to the local database
conn.Open();
System.Data.SqlServerCe.SqlCeCommand cmd = conn.CreateCommand();
SqlCeParameter param = new SqlCeParameter();
param.ParameterName = "#Barcode";
param.Value = "%" + textBarcode.Text.Trim() + "%";
// Insert a row
cmd.CommandText = "SELECT * FROM Main2 WHERE Reading LIKE #Barcode";
cmd.Parameters.Add(param);
cmd.ExecuteNonQuery();
DataTable data = new DataTable();
using (SqlCeDataReader reader = cmd.ExecuteReader())
{
if (reader.Read())
{
data.Load(reader);
}
}
if (data != null)
{
this.dataGrid1.DataSource = data;
}
}
finally
{
conn.Close();
}
The database contains this data:
Okay so you can see I changed the WHERE statement to use the Reading column just for testing purposes. When I enter "111" into the textbox and run --> it returns only the row where reading ="1111" and not the row that contains "111".
If I enter "1111" it does not return any data.
If I enter "1" it will return both the "1111" row and the "111" row which is the correct behavior.
However if I enter "11" it once again only returns the "1111" row.
Any other data entry of 2's or 9's attempting to return those rows does not work.
I'm not sure what is going on? This does not make any sense. It is not behaving like I would expect in any way shape or form. I know this must be a little confusing to read. I hope it makes enough sense to get some answers. Please help!
NOTE: I added the "%" before and after the text in an attempt to get better results. This is not desired.
EDIT <<<-----------------------I did have Reading = #Barcode, I just accidently typed Location for this question, that is not the problem.

Firstly, some things to note:
1) As other commentators have noted, use the Reading column, not the Location column. I know you have mentioned you are testing, but swapping around column names and then changing code isn't the easiest way to troubleshoot these things. Try to only change one thing at a time.
2) If Reading is numeric, you are going to have to convert the column value first.
So your query becomes:
"SELECT * FROM Main2 WHERE CONVERT(varchar, Reading) LIKE #Barcode";
Also see How to use parameter with LIKE in Sql Server Compact Edition for more help with working with parameters in SqlServerCE.
3) Set a parameter type on your SqlCEParameter. I've linked to the appropriate page in the code example below.
4) You are using ExecuteNonQuery for no reason. Just get rid of it in this context. It's for when you want to make a change to the database (like an insert, update, delete) or execute something (like a stored proc that can also insert, update, delete etc) that returns no rows. You've probably cut and paste this code from another place in your app :-)
5) Use using on disposable objects (see example below). This will make managing your connection lifecycle much simpler. It's also more readable (IMO) and will take care of issues when exceptions occur.
6) Use the using statement to import the BCL (Base Class Libraries) into your current namespace:
Add the following using statements to the top of your class (.cs). This will make using all of the .Net classes a lot simpler (and is much easier to read and less wear on your keyboard ;-)
using System.Data.SqlServerCe;
using System.IO;
using System.Reflection;
A more complete example would look like the following
private void buttonStart_Click(object sender, EventArgs e)
{
using(SqlCeConnection conn = new SqlCeConnection(
("Data Source=" + (Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().GetName().CodeBase), "ElectricReading.sdf") + ";Max Database Size=2047"))))
{
// Connect to the local database
conn.Open();
using(SqlCeCommand cmd = conn.CreateCommand())
{
SqlCeParameter param = new SqlCeParameter();
param.ParameterName = "#Barcode";
param.DBType = DBType.String; //Intellisense is your friend here but See http://msdn.microsoft.com/en-US/library/system.data.sqlserverce.sqlceparameter.dbtype(v=VS.80).aspx for supported types
param.Value = "%" + textBarcode.Text.Trim() + "%";
// SELECT rows
cmd.CommandText = "SELECT * FROM Main2 WHERE CONVERT(varchar, Reading) LIKE #Barcode";
cmd.Parameters.Add(param);
//cmd.ExecuteNonQuery(); //You don't need this line
DataTable data = new DataTable();
using (SqlCeDataReader reader = cmd.ExecuteReader())
{
data.Load(reader); //SqlCeDataReader does not support the HasRows property.
if(data.Rows.Count > 0)
{
this.dataGrid1.DataSource = data;
}
}
}
}
}
Intellisense should be able to clean up any errors with the above but feel free to ask for more help.
Finally, you also might be able to set the data source of the grid directly to a datareader, try it!
using (SqlCeDataReader reader = cmd.ExecuteReader())
{
dataGrid1.DataSource = reader;
}
You can then get rid of the DataTable.

Change the following line:
cmd.CommandText = "SELECT * FROM Main2 WHERE Location LIKE #Barcode";
to
cmd.CommandText = "SELECT * FROM Main2 WHERE Reading LIKE #Barcode";
You are comparing the wrong columns.

How to do a batch update?

I am wondering is there a way to do batch updating? I am using ms sql server 2005.
I saw away with the sqlDataAdaptor but it seems like you have to first the select statement with it, then fill some dataset and make changes to dataset.
Now I am using linq to sql to do the select so I want to try to keep it that way. However it is too slow to do massive updates. So is there away that I can keep my linq to sql(for the select part) but using something different to do the mass update?
Thanks
Edit
I am interested in this staging table way but I am not sure how to do it and still not clear how it will be faster since I don't understand how the update part works.
So can anyone show me how this would work and how to deal with concurrent connections?
Edit2
This was my latest attempt at trying to do a mass update using xml however it uses to much resources and my shared hosting does not allow it to go through. So I need a different way so thats why I am not looking into a staging table.
using (TestDataContext db = new TestDataContext())
{
UserTable[] testRecords = new UserTable[2];
for (int count = 0; count < 2; count++)
{
UserTable testRecord = new UserTable();
if (count == 1)
{
testRecord.CreateDate = new DateTime(2050, 5, 10);
testRecord.AnotherField = true;
}
else
{
testRecord.CreateDate = new DateTime(2015, 5, 10);
testRecord.AnotherField = false;
}
testRecords[count] = testRecord;
}
StringBuilder sBuilder = new StringBuilder();
System.IO.StringWriter sWriter = new System.IO.StringWriter(sBuilder);
XmlSerializer serializer = new XmlSerializer(typeof(UserTable[]));
serializer.Serialize(sWriter, testRecords);
using (SqlConnection con = new SqlConnection(connectionString))
{
string sprocName = "spTEST_UpdateTEST_TEST";
using (SqlCommand cmd = new SqlCommand(sprocName, con))
{
cmd.CommandType = CommandType.StoredProcedure;
cmd.CommandType = System.Data.CommandType.StoredProcedure;
SqlParameter param1 = new SqlParameter("#UpdatedProdData", SqlDbType.VarChar, int.MaxValue);
param1.Value = sBuilder.Remove(0, 41).ToString();
cmd.Parameters.Add(param1);
con.Open();
int result = cmd.ExecuteNonQuery();
con.Close();
}
}
}
# Fredrik Johansson I am not sure what your saying will work. Like it seems to me you want me to make a update statement for each record. I can't do that since I will have need update 1 to 50,000+ records and I will not know till that point.
Edit 3
So this is my SP now. I think it should be able to do concurrent connections but I wanted to make sure.
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE [dbo].[sp_MassUpdate]
#BatchNumber uniqueidentifier
AS
BEGIN
update Product
set ProductQty = 50
from Product prod
join StagingTbl stage on prod.ProductId = stage.ProductId
where stage.BatchNumber = #BatchNumber
DELETE FROM StagingTbl
WHERE BatchNumber = #BatchNumber
END

You can use the sqlDataAdapter to do a batch update. It dosen’t matter how you fill your dataset. L2SQL or whatever, you can use different methods to do the update. Just define the query to run using the data in your datatable.
The key here is the UpdateBatchSize. The dataadapter will send the updates in batches of whatever size you define. You need to expirement with this value to see what number works best, but typicaly numbers of 500-1000 do best. SQL can then optimize the update and execute a little faster. Note that when doing batchupdates, you cannot update the row source of the datatable.
I use this method to do updates of 10-100K and it usualy runs in under 2 minutes. It will depend on what you are updating though.
Sorry, this is in VB….
Using da As New SqlDataAdapter
da.UpdateCommand = conn.CreateCommand
da.UpdateCommand.CommandTimeout = 300
da.AcceptChangesDuringUpdate = False
da.ContinueUpdateOnError = False
da.UpdateBatchSize = 1000 ‘Expirement for best preformance
da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None 'Needed if UpdateBatchSize > 1
sql = "UPDATE YourTable"
sql += " SET YourField = #YourField"
sql += " WHERE ID = #ID"
da.UpdateCommand.CommandText = sql
da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None
da.UpdateCommand.Parameters.Clear()
da.UpdateCommand.Parameters.Add("#YourField", SqlDbType.SmallDateTime).SourceColumn = "YourField"
da.UpdateCommand.Parameters.Add("#ID", SqlDbType.SmallDateTime).SourceColumn = "ID"
da.Update(ds.Tables("YourTable”)
End Using
Another option is to bulkcopy to a temp table, and then run a query to update the main table from it. This may be faster.

As allonym said, Use SqlBulkCopy, which is very fast(I found speed improvements of over 200x - from 1500 secs to 6s). However you can use the DataTable and DataRows classes to provide data to SQlBulkCopy (which seems easier). Using SqlBulkCopy this way has the added advantage of bein .NET 3.0 compliant as well (Linq was added only in 3.5).
Checkout http://msdn.microsoft.com/en-us/library/ex21zs8x%28v=VS.100%29.aspx for some sample code.

Use SqlBulkCopy, which is lightning-fast. You'll need a custom IDataReader implementation which enumerates over your linq query results. Look at http://code.msdn.microsoft.com/LinqEntityDataReader for more info and some potentially suitable IDataReader code.

You have to work with the expression trees directly, but it's doable. In fact, it's already been done for you, you just have to download the source:
Batch Updates and Deletes with LINQ to SQL
The alternative is to just use stored procedures or ad-hoc SQL queries using the ExecuteMethodCall and ExecuteCommand methods of the DataContext.

You can use SqlDataAdapter to do a batch-update even if a datatable is filled manually/programmatically (from linq of any other source).
Just remember to manually set the RowState for the rows in the datatable. Use dataRow.SetModified() for this.