I have a service which continuously writes data in a separate thread into SQL database.Now from the same service if i am trying to read from the same table, since i already am writing into it,I get this exception : There is already an open DataReader associated with this Command which must be closed first.
So can anyone help me how to do this simultaneously?
Here s my code for reading data:
public Collection ReadData(string query)
{
{
_result = new Collection<string[]>();
string[] tempResult;
SqlDataReader _readerRead;
using (_command = new SqlCommand(query, _readConnection))
{
_readerRead = _command.ExecuteReader();
while (_readerRead.Read())
{
tempResult = new string[4];
tempResult[0] = _reader[0].ToString();
tempResult[1] = _reader[1].ToString();
tempResult[2] = _reader[2].ToString();
tempResult[3] = _reader[3].ToString();
_result.Add(tempResult);
//Console.WriteLine("Name : {0} Type : {1} Value : {2} timestamp : {3}", _reader[0], _reader[1], _reader[2], _reader[3]);
}
if (_readerRead != null)
{
_readerRead.Close();
}
_readConnection.Close();
return _result;
}
}
}
and here it is for writing to it :
public void WriteData(Collection<TagInfo> tagInfoList)
{
int i = 0;
for (i = 0; i < tagInfoList.Count; i++)
{
using( _command = new SqlCommand(insert statement here)
{
_command.Parameters.AddWithValue("Name", tagInfoList[i].Name);
_command.Parameters.AddWithValue("Type", tagInfoList[i].TagType);
_command.Parameters.AddWithValue("Value", tagInfoList[i].Value);
_reader = _command.ExecuteReader();
if (_reader != null)
{
_reader.Close();
}
}
}
}
You need a different SQLConnection to the database for your writer. You cannot use the same db connection for both.
Although its possible to do, using a separate connection I would question why you need to do this.
If you are reading and writing data to one table in the same service you will be placing unnecessary load on one SQL table, and depending on the number of queries you intend to make this could cause you problems. If you already have this data (in a different thread) why not Marshall the data from the background thread to where you need it as you write it into the database, and you don't need to read the data anymore.
However.... it is difficult to give an fair answer without seeing the code/what you are looking to achieve.
Related
So I have this code that checks if new data is added to online database by comparing the rows of online and local database. If new data is found it inserts the new data to local database.
public class Reservation
{
public string res_no { get; set; }
public string mem_fname { get; set; }
}
My Code :
private async void updateDineList()
{
DBconnector.OpenConnection();
//Gets data from online database
HttpClient client = new HttpClient();
var response = await client.GetStringAsync("http://example.com/Reservation/view_pending_reservation");
var persons = JsonConvert.DeserializeObject<List<Reservation>>(response);
//Gets data from Local database
string string_reservation = "SELECT res_no,mem_fname FROM res_no WHERE res_status='pending';";
DataTable reservation_table = new DataTable();
MySqlDataAdapter adapter_reservartion = new MySqlDataAdapter(string_reservation, DBconnector.Connection);
adapter_reservartion.Fill(reservation_table);
//Gets the row of each table
int local = reservation_table.Rows.Count;
int online = persons.Count;
//Compares rows of online and local database
if (local < online)
{
//if the rows of online database is greater than local database
//inserts the new data from local database
string Command_membership = "INSERT INTO reservation_details (res_no,mem_fname) VALUES (#res_no, #mem_fname);";
for (int i = local; i < online; i++)
{
//inserts new data from online to local database
using (MySqlCommand myCmd = new MySqlCommand(Command_membership, DBconnector.Connection))
{
myCmd.CommandType = CommandType.Text;
myCmd.Parameters.AddWithValue("#res_no", persons[i].res_no);
myCmd.Parameters.AddWithValue("#mem_fname", persons[i].mem_fname);
myCmd.ExecuteNonQuery();
}
}
MessageBox.Show("New Records Found");
}
else
{
MessageBox.Show("No new Records");
}
DBconnector.Connection.Close();
}
So my question is there any problem could occur with this code, it works fine but is there any way to improve this. I know MySQL replication is better but I am only using free Web Hosting with few MySQL privileges.
The clear improvement is not to create a new command for every row. You should either create the command and parameters once and then set the parameters and call for each row, or better still package the set of updates into a single structure, like and xml string, and then pass the whole lot to the database via a stored procedure call.
Other probably problematic issue is that you are checking purely based on row counts. Don't know if that is valid in your scenario but it sounds dangerous. What if rows are deleted? or is that not possible in your scenario. Some other way of checking last updates would probably be preferable.
Without more context that's about all I can see.
My setup is MySql.Data.MySqlClient v6.9.8.0 and Microsoft.Practices.EnterpriseLibrary.Data v6.0.0.
The program is a long running program that runs continuously listening for tasks and then performs the job with some form of database action (depending on what the request was.) Sometimes the requests will be one after the other, sometimes there will be several hours between them.
I've tried using Pooling=true in the connection string but it causes me a lot of problems (not all the time - these are intermittent problems.)
Here is an example:
[MySqlException (0x80004005): Authentication to host 'localhost' for user 'root' using method 'mysql_native_password' failed with message: Reading from the stream has failed.]
Turning off pooling fixes the problem but at the same time it makes the queries slower because we can't reuse connections. I've searched online and a lot of people have this same issue and the only fix/workaround I've found is Pooling=false which I'd rather avoid if possible.
Here is an example of my query code:
Database db = this.GetDatabase(databaseName);
List<dynamic> results = new List<dynamic>();
// Run the sql query
using (DbCommand dbCommand = db.GetSqlStringCommand(query))
{
foreach (var parameter in inParameters)
{
db.AddInParameter(dbCommand, parameter.Key, parameter.Value.Item1, parameter.Value.Item2);
}
foreach (var parameter in outParameters)
{
db.AddOutParameter(dbCommand, parameter.Key, parameter.Value.Item1, parameter.Value.Item2);
}
using (IDataReader dataReader = db.ExecuteReader(dbCommand))
{
IDictionary<string, object> instance;
do
{
// Read each row
while (dataReader.Read())
{
instance = new ExpandoObject() as IDictionary<string, object>;
// Populate the object on the fly with the data
for (int i = 0; i < dataReader.FieldCount; i++)
{
instance.Add(dataReader.GetName(i), dataReader[i]);
}
// Add the object to the results list
results.Add(instance);
}
} while (dataReader.NextResult());
}
return results;
}
Any ideas?
Can you try this? I know, I know. using "using" should mean I don't have to call the dataReader.Close() method...but I still do it. I also slightly altered the dr.Read block.
This guy talks about it.
http://www.joseguay.com/uncategorized/ensure-proper-closure-disposal-of-a-datareader
I know, I know. You shouldn't have to. Even when using Ent library, I do an extra .Close step to try and make sure.
Database db = this.GetDatabase(databaseName);
List<dynamic> results = new List<dynamic>();
// Run the sql query
using (DbCommand dbCommand = db.GetSqlStringCommand(query))
{
foreach (var parameter in inParameters)
{
db.AddInParameter(dbCommand, parameter.Key, parameter.Value.Item1, parameter.Value.Item2);
}
foreach (var parameter in outParameters)
{
db.AddOutParameter(dbCommand, parameter.Key, parameter.Value.Item1, parameter.Value.Item2);
}
using (IDataReader dataReader = db.ExecuteReader(dbCommand))
{
IDictionary<string, object> instance;
while (dataReader.Read())
{
instance = new ExpandoObject() as IDictionary<string, object>;
// Populate the object on the fly with the data
for (int i = 0; i < dataReader.FieldCount; i++)
{
instance.Add(dataReader.GetName(i), dataReader[i]);
}
// Add the object to the results list
results.Add(instance);
}
if (dataReader != null)
{
try
{
dataReader.Close();
}
catch { }
}
}
return results;
}
I have a method that queries a table for the count of its records. QA has discovered an "edge case" where if a particular operation is canceled in a particular order and speed (as fast as possible), the GUI "forgets" about the rest of the records in that table (the contents of the tables are uploaded to a server; when each one finishes, the corresponding table is deleted).
To be clear, this table that is having records deleted from it and then queried for count ("workTables") is a table of table names, that are deleted after they are processed.
What I have determined (I'm pretty sure) is that this anomaly occurs when a record from the "workTables" table is in the process of being deleted when the workTables table is queried for the count of its records. This causes an exception, which causes the method to return -1, which in our case indicates we should cuase the GUI to not display those records.
Is there a way to check if a table is in the process of having a record deleted from it, and wait until after that operation has completed, before proceeding with the query, so that it won't throw an exception?
For those interested in the specifics, this method is the one that, under those peculiar circumstances, throws an exception:
public int isValidTable(string tableName)
{
int validTable = -1;
string tblQuery = "SELECT COUNT(*) FROM ";
tblQuery += tableName;
openConnectionIfPossibleAndNecessary();
try
{
SqlCeCommand cmd = objCon.CreateCommand();
cmd.CommandText = tblQuery;
object objcnt = cmd.ExecuteScalar();
validTable = Int32.Parse(objcnt.ToString());
}
catch (Exception ex)
{
validTable = -1;
}
return validTable;
}
...and this is the method that deletes a record from the "workTables" table after the corresponding table has had its contents uploaded:
private void DropTablesAndDeleteFromTables(string recordType, string fileName)
{
try
{
WorkFiles wrkFile = new WorkFiles();
int tableOK = 0;
DataSet workfiles;
tableOK = wrkFile.isValidWorkTable(); // -1 == "has no records"
if (tableOK > 0) //Table has at least one record
{
workfiles = wrkFile.getAllRecords();
//Go thru dataset and find filename to clean up after
foreach (DataRow row in workfiles.Tables[0].Rows)
{
. . .
dynSQL = string.Format("DELETE FROM workTables WHERE filetype = '{0}' and Name = '{1}'", tmpType, tmpStr);
dbconn = DBConnection.GetInstance();
dbconn.DBCommand(dynSQL, false);
populateListBoxWithWorkTableData();
return;
} // foreach (DataRow row in workfiles.Tables[0].Rows)
}
}
catch (Exception ex)
{
SSCS.ExceptionHandler(ex, "frmCentral.DropTablesAndDeleteFromTables");
}
}
// method called by DropTablesAndDeleteFromTables() above
public int isValidWorkTable() //reverted to old way to accommodate old version of DBConnection
{
// Pass the buck
return dbconn.isValidTable("workTables");
}
I know this code is very funky and klunky and kludgy; refactoring it to make more sense and be more easily understood is a long and ongoing process.
UPDATE
I'm not able to test this code:
lock (this)
{
// drop the table
}
...yet, because the handheld is no longer allowing me to copy files to it (I get, "Cannot copy [filename.[dll,exe] The device has either stopped responding or has been disconnected" (it is connected, as shown by ActiveStync))
If that doesn't work, I might have to try this:
// global var
bool InDropTablesMethod;
// before querying that database from elsewhere:
while (InDropTablesMethod)
{
Pause(500);
}
UPDATE 2
I've finally been able to test my lock code (copies of binaries were present in memory, not allowing me to overwrite them; the StartUp folder had a *.lnk to the .exe, so every time I started the handheld, it tried to run the buggy versions of the .exe), but it doesn't work - I still get the same conflict/contention.
UPDATE 3
What seems to work, as kludgy as it may be, is:
public class CCRUtils
{
public static bool InDropTablesMethod;
. . .
if (CCRUtils.InDropTablesMethod) return;
CCRUtils.InDropTablesMethod = true;
. . . // do it all; can you believe somebody from El Cerrito has never heard of CCR?
CCRUtils.InDropTableMethod = false;
UPDATE 4
Wrote too soon - the bug is back. I added this MessageBox.Show(), and do indeed see the text "proof of code re-entrancy" at run-time.
while (HHSUtils.InDropTablesMethod)
{
MessageBox.Show("proof of code re-entrancy");
i++;
if (i > 1000000) return;
}
try
{
HHSUtils.InDropTablesMethod = true;
. . .
}
HHSUtils.InDropTablesMethod = false;
...so my guess that code re-entrancy may be a problem is correct...
In short I am writing a class handler to handle to database integration of some software I am writing for myself, however as there is not always a connection to the remote database I thought I would use SQLCE to create a local database buffer so when a connection is made the changes can be synchronized.
So far it is going well except for the parameters. The function I am looking to call is shown below however this function is complaining about invalid arguments.
public Object run(string query, List<Object> dbparams = null)
{
if (MyDB.isConnected())
{
return MyDB.run(query, dbparams);
}
else
{
SqlCeCommand sql = _OfflineConnection.CreateCommand();
sql.CommandText = query;
if (dbparams.Count > 0)
{
sql.Parameters.AddRange(dbparams.ToArray());
}
return sql;
}
}
MyDB.run is the exact same code as in the else statement except for mysql, the line that it is moaning about is the return mydb.run as the mydb class is expecting the dbparams list to be of mysqlparameters.
Does anyone know how I can achieve this? I attempted to use LINQ to do a convert but that failed miserably.
EDIT
At present I have the following working but I am sure there is a cleaner solution
public Object run(string query, List<Object> dbparams = null)
{
if (MyDB.isConnected())
{
List<MySqlParameter> mydbparams = null;
for (int i = 0; i < dbparams.Count; i++)
{
mydbparams.Add((MySqlParameter)dbparams[i]);
}
return MyDB.run(query, mydbparams);
}
else
{
SqlCeCommand sql = _OfflineConnection.CreateCommand();
sql.CommandText = query;
if (dbparams.Count > 0)
{
sql.Parameters.AddRange(dbparams.ToArray());
}
return sql;
}
}
A bit cleaner solution would be
mydbparams = dbparams.Cast<MySqlParameters>().ToList();
Also, you should check for and handle the null condition of dbparams.
I'm building a console application that have to process a bunch of document.
To stay simple, the process is :
for each year between X and Y, query the DB to get a list of document reference to process
for each of this reference, process a local file
The process method is, I think, independent and should be parallelized as soon as input args are different :
private static bool ProcessDocument(
DocumentsDataset.DocumentsRow d,
string langCode
)
{
try
{
var htmFileName = d.UniqueDocRef.Trim() + langCode + ".htm";
var htmFullPath = Path.Combine("x:\path", htmFileName;
missingHtmlFile = !File.Exists(htmFullPath);
if (!missingHtmlFile)
{
var html = File.ReadAllText(htmFullPath);
// ProcessHtml is quite long : it use a regex search for a list of reference
// which are other documents, then sends the result to a custom WS
ProcessHtml(ref html);
File.WriteAllText(htmFullPath, html);
}
return true;
}
catch (Exception exc)
{
Trace.TraceError("{0,8}Fail processing {1} : {2}","[FATAL]", d.UniqueDocRef, exc.ToString());
return false;
}
}
In order to enumerate my document, I have this method :
private static IEnumerable<DocumentsDataset.DocumentsRow> EnumerateDocuments()
{
return Enumerable.Range(1990, 2020 - 1990).AsParallel().SelectMany(year => {
return Document.FindAll((short)year).Documents;
});
}
Document is a business class that wrap the retrieval of documents. The output of this method is a typed dataset (I'm returning the Documents table). The method is waiting for a year and I'm sure a document can't be returned by more than one year (year is part of the key actually).
Note the use of AsParallel() here, but I never got issue with this one.
Now, my main method is :
var documents = EnumerateDocuments();
var result = documents.Select(d => {
bool success = true;
foreach (var langCode in new string[] { "-e","-f" })
{
success &= ProcessDocument(d, langCode);
}
return new {
d.UniqueDocRef,
success
};
});
using (var sw = File.CreateText("summary.csv"))
{
sw.WriteLine("Level;UniqueDocRef");
foreach (var item in result)
{
string level;
if (!item.success) level = "[ERROR]";
else level = "[OK]";
sw.WriteLine(
"{0};{1}",
level,
item.UniqueDocRef
);
//sw.WriteLine(item);
}
}
This method works as expected under this form. However, if I replace
var documents = EnumerateDocuments();
by
var documents = EnumerateDocuments().AsParrallel();
It stops to work, and I don't understand why.
The error appears exactly here (in my process method):
File.WriteAllText(htmFullPath, html);
It tells me that the file is already opened by another program.
I don't understand what can cause my program not to works as expected. As my documents variable is an IEnumerable returning unique values, why my process method is breaking ?
thx for advises
[Edit] Code for retrieving document :
/// <summary>
/// Get all documents in data store
/// </summary>
public static DocumentsDS FindAll(short? year)
{
Database db = DatabaseFactory.CreateDatabase(connStringName); // MS Entlib
DbCommand cm = db.GetStoredProcCommand("Document_Select");
if (year.HasValue) db.AddInParameter(cm, "Year", DbType.Int16, year.Value);
string[] tableNames = { "Documents", "Years" };
DocumentsDS ds = new DocumentsDS();
db.LoadDataSet(cm, ds, tableNames);
return ds;
}
[Edit2] Possible source of my issue, thanks to mquander. If I wrote :
var test = EnumerateDocuments().AsParallel().Select(d => d.UniqueDocRef);
var testGr = test.GroupBy(d => d).Select(d => new { d.Key, Count = d.Count() }).Where(c=>c.Count>1);
var testLst = testGr.ToList();
Console.WriteLine(testLst.Where(x => x.Count == 1).Count());
Console.WriteLine(testLst.Where(x => x.Count > 1).Count());
I get this result :
0
1758
Removing the AsParallel returns the same output.
Conclusion : my EnumerateDocuments have something wrong and returns twice each documents.
Have to dive here I think
This is probably my source enumeration in cause
I suggest you to have each task put the file data into a global queue and have a parallel thread take writing requests from the queue and do the actual writing.
Anyway, the performance of writing in parallel on a single disk is much worse than writing sequentially, because the disk needs to spin to seek the next writing location, so you are just bouncing the disk around between seeks. It's better to do the writes sequentially.
Is Document.FindAll((short)year).Documents threadsafe? Because the difference between the first and the second version is that in the second (broken) version, this call is running multiple times concurrently. That could plausibly be the cause of the issue.
Sounds like you're trying to write to the same file. Only one thread/program can write to a file at a given time, so you can't use Parallel.
If you're reading from the same file, then you need to open the file with only read permissions as not to put a write lock on it.
The simplest way to fix the issue is to place a lock around your File.WriteAllText, assuming the writing is fast and it's worth parallelizing the rest of the code.