I am using c# sqlite version 3. My application requires saving up to 20 columns of data at 4000Hz. To accomplish this I am using threads that take commands from a queue and periodically write/read to the database. Currently, I am just working on the writing portion and have found that ExecuteNonQueryAsync() and ExecuteNonQuery() have relatively the same execution time which makes no sense to me. My function is as follows:
public async Task<int> dumpStorage(string cmdStr)
{
int r;
using (SQLiteCommand cmd = new SQLiteCommand(sqliteConnection))
{
string times = "";
Stopwatch tTime = new Stopwatch();
tTime.Start();
cmd.CommandText = cmdStr;
times += "tTime1 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
tTime.Restart();
r = await cmd.ExecuteNonQueryAsync();
//cmd.ExecuteNonQuery();
//cmd.ExecuteScalar();
//cmd.ExecuteScalarAsync();
times += "tTime2 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
tTime.Restart();
transaction.Commit();
times += "tTime2 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
tTime.Restart();
transaction = sqliteConnection.BeginTransaction();
tTime.Stop();
times += "tTime3 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
timeStr = times;
storeToDBWatchDog.Restart();
}
return r;
}
at the moment I am saving 40,000 rows with 20 (plus a primary key) columns which represents 10 seconds of data. I am unable to get the async to have any time advantage over the non-async function. in the future I will need to introduce a scheme that will manage if there was any failures to insert.
I would expect that ExecuteNonQueryAsync() should make tTime2 essentially 0 and it would work in the background however that is not what I am seeing as it appear to wait for the ...Async() function to complete.
the execute non query async and non-async require roughly 2.1 seconds. The end goal is to save the data at least 20 times faster than it is coming in. The only time that this becomes very important is on initial load. By this I mean i am reading an existing file (not in a database) and then saving it into the database. so, at the moment I only have approximately 4 times the performance. by this i mean it would take 5 minutes to save a file that consists of data associated with 20 minutes. this seems unexceptionably slow.
typical time output regardless of which function call is used (both scalar functions take ~50% longer):
Edit:
I recently came across this unfortunate post, which essentially states that async doesn't do anything so you should use the WAL setting:
https://learn.microsoft.com/en-us/dotnet/standard/data/sqlite/async
i have since changed my code to include the WAL journal-mode:
connectionString = "Data Source=" + filename + "; Version=3" + "; PRAGMA journal_mode=WAL;";
sqliteConnection = new SQLiteConnection(connectionString);
I did not see any performance increases setting the journal_mode to WAL.
Related
I keep getting this exception over and over. I've tried separating my query into two separate queries, that didn't work. I've checked to make sure the db connection is closed elsewhere before it's opened during this method, it's definitely closed before the function is called and opened before any queries.
Below iss the code for the function. I've set breakpoints and the query itself is fine. The code is the exact same that I used previously for updating a PIN function, with just the query string changed, so I don't know why it's causing issues:
Code:
public void transferMoney(string senderIban, decimal senderBalance, string receiverIban, decimal transferAmount,string messageOptional)
{
//myBankAccount.AccountPin = updatedPin;
DataTable dtUser = new DataTable();
sqlconnConnection.Open();
string strQuery2 = #"UPDATE Accounts SET Balance = Balance + " + Convert.ToDecimal(transferAmount) + " WHERE GUID = '" + receiverIban + "';"
+ "UPDATE Accounts SET Balance = Balance - " + Convert.ToDecimal(transferAmount) + " WHERE GUID = '" + senderIban + "';";
// example of a Paramaterised SQL statement.
SQLiteCommand sqlcomCommand2 = new SQLiteCommand(strQuery2, sqlconnConnection);
SQLiteDataAdapter sqldatadptAdapter = new SQLiteDataAdapter(sqlcomCommand2); // local SQL data Adaptor
try
{
// sqldatadptAdapter.Fill(dtUser);
sqlcomCommand2.ExecuteNonQuery();
}
catch (Exception ex)
{
// Exception will the "thrown" when there was a problem
throw new Exception($"UPDATE WAS unsuccessful:\n{ex.Message}");
}
finally
{
sqlconnConnection.Close();
}
Maybe you have a DB browser opened? Or you have accessed the DB some other way. This error only occurs when DB is modified or used elsewhere. If you can't find it, I'd suggest restarting PC just in case there something hidden :)
P.S. Posting this as answer as I cannot comment under the question for technical reasons :)
I am using c# sqlite version 3. My application requires saving up to 20 columns of data at 4000Hz. To accomplish this I am using threads that take commands from a queue and periodically write/read to the database. Currently, I am just working on the writing portion and have found that ExecuteNonQueryAsync() and ExecuteNonQuery() have relatively the same execution time which makes no sense to me. My function is as follows:
public async Task<int> dumpStorage(string cmdStr)
{
int r;
using (SQLiteCommand cmd = new SQLiteCommand(sqliteConnection))
{
string times = "";
Stopwatch tTime = new Stopwatch();
tTime.Start();
cmd.CommandText = cmdStr;
times += "tTime1 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
tTime.Restart();
r = await cmd.ExecuteNonQueryAsync();
//cmd.ExecuteNonQuery();
//cmd.ExecuteScalar();
//cmd.ExecuteScalarAsync();
times += "tTime2 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
tTime.Restart();
transaction.Commit();
times += "tTime2 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
tTime.Restart();
transaction = sqliteConnection.BeginTransaction();
tTime.Stop();
times += "tTime3 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
timeStr = times;
storeToDBWatchDog.Restart();
}
return r;
}
at the moment I am saving 40,000 rows with 20 (plus a primary key) columns which represents 10 seconds of data. I am unable to get the async to have any time advantage over the non-async function. in the future I will need to introduce a scheme that will manage if there was any failures to insert.
I would expect that ExecuteNonQueryAsync() should make tTime2 essentially 0 and it would work in the background however that is not what I am seeing as it appear to wait for the ...Async() function to complete.
the execute non query async and non-async require roughly 2.1 seconds. The end goal is to save the data at least 20 times faster than it is coming in. The only time that this becomes very important is on initial load. By this I mean i am reading an existing file (not in a database) and then saving it into the database. so, at the moment I only have approximately 4 times the performance. by this i mean it would take 5 minutes to save a file that consists of data associated with 20 minutes. this seems unexceptionably slow.
typical time output regardless of which function call is used (both scalar functions take ~50% longer):
Edit:
I recently came across this unfortunate post, which essentially states that async doesn't do anything so you should use the WAL setting:
https://learn.microsoft.com/en-us/dotnet/standard/data/sqlite/async
i have since changed my code to include the WAL journal-mode:
connectionString = "Data Source=" + filename + "; Version=3" + "; PRAGMA journal_mode=WAL;";
sqliteConnection = new SQLiteConnection(connectionString);
I did not see any performance increases setting the journal_mode to WAL.
Hi all and thanks in advance. I am trying to fix a method that inserts information to a database table. Currently its experiencing timeouts because its running in a while loop that is taking too long to process all the contents. While I know I could just increase the command timeout I don't think that solves the problem because I think its the code. But I'm not certain what the correct fix is. I have access to Dapper and I wonder if it would be more efficient to make a method that passes the necessary variables and executes just a quick simple statement for that group then goes to get the next one? Or is that just perpetuating what's below just in a different way? Should I move this out of the code and onto the server for better performance?
UPDATE Full error message:
Exception of type 'System.Web.HttpUnhandledException' was thrown.
File: c:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\6caa4c91\19b853c6\App_Web_o3102kpb.9.cs
Method: ProcessRequest Line Number : 0
Inner Exception: {Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
File: z:\inetpub\wwwroot\SessionTransfer.aspx.cs
Method: AddSessionToDatabase Line Number : 94
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
File: z:\inetpub\wwwroot\SessionTransfer.aspx.cs Method: Page_Load Line Number : 33 }
Here is the original code:
SqlConnection con = new SqlConnection(connectionString);
SqlCommand cmd = new SqlCommand();
con.Open();
cmd.Connection = con;
int i = 0;
string strSql, guid = GetGuid();
string temp = "";
while (i < Session.Contents.Count)
{
if (Session.Contents[i] == null)
temp = "";
else {
if ((Session.Contents[i].ToString().Trim().Length) > 0)
temp = Session.Contents[i].ToString().Replace("'", "''");
else
temp = "";
}
strSql = "INSERT INTO SessionTable (GUID, SessionKey, SessionValue) " +
"VALUES ('" + guid + "', '" + Session.Contents.Keys[i].ToString() + "', '" + temp + "')";
cmd.CommandText = strSql;
cmd.ExecuteNonQuery();
i++;
}
con.Close();
cmd.Dispose();
con.Dispose();
return guid;
UPDATE - FINAL SOLUTION:
var SessionList = new List<Session>();
while (i < Session.Contents.Count)
{
string temp = "";
if (Session.Contents[i] == null)
temp = "";
else
{
temp = (Session.Contents[i].ToString().Trim().Length) > 0 ? Session.Contents[i].ToString().Replace("'", "''") : "";
}
var s = new Session
{
TempGuid = guidTemp,
Contents = Session.Contents[i] != null ? Session.Contents[i].ToString() : null,
Temp = temp
};
SessionList.Add(s);
i++;
}
mySession = SerializationUtilities.SerializeObjectToXML(SessionList);
using (var con = new SqlConnection())
{
con.ExecuteHGW("Transfer", new { mySession }, commandType: CommandType.StoredProcedure);
}
Then on the SQL side I just put the XML in a table and did one single insert statement against the table, time is significantly improved.
I would like to suggest you 2 improvements:
in the loop you are executing insert statements one by one against db. It takes much time to open connection to db then send the query, execute it and return result. It is much better to batch them. So gather like 10,000 of such insert statements, build the whole instuction with StringBuilder and execute it against db in one go. This will really increase the speed of your app. Amount of Insert instructions in one batch you should choose yourself basing on system tests.
If after applying 1) hint the problem with timeout still occurs I would suggest 2 possible solutions:
a). do not send all elements to be processed against db to web service at once but instead, as previoulsy, batch them (apply second batching). So for instance send to web service 50,000 elements to be insterted, then wait for confirmation from web service and then proceed with next batch. The big advantage is that you can show to user easily progress bar showing him current operation state.
b). send all items to be processed against db at once but do not wait for result. In your app just show that items are processed and each 10 s send to web service request to ask if the job is finished. When it is finished signal it to user.
I have an ASP.NET c# web application published to a server of ours. It consists of three pages, each with a form on them, sort of like a 3-page login. After the third page's form is validated, it sends the user to a different site. Here's a little diagram so I can explain it better:
NamePage ---> DateOfBirthPage ---> IDNumberPage ---> OtherSite
This has been working great in all of our development tests and stress tests. However, after we put it into production, occasionally when the "Next" button on the IDNumberPage is clicked, the user sees "This page cannot be displayed" with a "diagnose connection problems" button. When this occurs for one user, the same problem occurs for all users (meaning once it happens, nobody can fully authenticate). NamePage and DateOfBirthPage always work, and when the crash occurs, the IDNumberPage link doesn't change, suggesting that the crash is occurring on this side of the application, not after it redirects to OtherSite. I have friendly HTTP errors turned off but it's not showing any errors on the page. If we go into the server and restart the application, it works again.
The frustrating part is that we can't replicate this error to see how/why it's occurring.
Some things that are noteworthy:
Each page uses one query on a MS SQL server database
Each page passes up to 4 Session variables (only small Strings containing what was entered into the textbox form on the previous page(s))
The session is abandoned when the final "next" button is clicked.
All resultsets/connections/commands are closed before redirect.
Redirects use the overloaded version using Response.Redirect(siteName, false)
Sorry if all of this is very vague, but the problem itself has done an oddly good job of hiding from us. We have tried hammering the server with test requests (many at once, many over a period of time, etc) and different combinations of logging in/trying to break the page in general, to no avail. Can anyone suggest some things to try to diagnose/fix/replicate this problem?
Edit: The click function on IDNumberPage's code-behind that is causing the problem:
{ SqlConnection dbconn = new SqlConnection(Application["dbconn"].ToString());
SqlCommand sqlValidate = dbconn.CreateCommand();
dbconn.Open();
sqlValidate.CommandText = "SELECT lastName, csn FROM Demographics WHERE lastName = '" + Session["lastName"].ToString() + "' " +
"AND dob = '" + Session["dobCheck"].ToString() + "' AND mrn = " + strMRN;
SqlDataReader results = sqlValidate.ExecuteReader();
if (results.HasRows)
{
string csn = "";
while (results.Read())
{
if (!String.IsNullOrEmpty(results["csn"].ToString()))
{
csn = results["csn"].ToString();
break;
}
}
string url = Application["surveyUrlString"] + "&lastname=" + Session["lastName"].ToString() + "&mrn=" + strMRN + "&dobday=" + Session["dobday"].ToString()
+ "&dobmonth=" + Session["dobmonth"].ToString() + "&dobyear=" + Session["dobyear"].ToString() + "&csn=" + csn;
results.Close();
dbconn.Close();
Response.Redirect(url, false);
}
The problem is due to leaking sql connections.
You aren't properly disposing of your resources. Over time these are going to stack up in the connection pool until you reach a point where the pool overflows and your app dies. Resetting will obviously fix the issue.
Also this issue might not show up in "stress" tests depending on how, exactly you are testing the application.
The solution is to reformat that code to handle your database call better.
{
string url = string.empty;
using (SqlConnection dbconn = new SqlConnection(Application["dbconn"].ToString())) {
using (SqlCommand sqlValidate = dbconn.CreateCommand()) {
dbconn.Open();
sqlValidate.CommandText = "SELECT lastName, csn FROM Demographics WHERE lastName = '" + Session["lastName"].ToString() + "' " +
"AND dob = '" + Session["dobCheck"].ToString() + "' AND mrn = " + strMRN;
using (SqlDataReader results = sqlValidate.ExecuteReader()) {
if (results.HasRows) {
string csn = "";
while (results.Read())
{
if (!String.IsNullOrEmpty(results["csn"].ToString()))
{
csn = results["csn"].ToString();
break;
}
}
url = Application["surveyUrlString"] + "&lastname=" + Session["lastName"].ToString() + "&mrn=" + strMRN + "&dobday=" + Session["dobday"].ToString()
+ "&dobmonth=" + Session["dobmonth"].ToString() + "&dobyear=" + Session["dobyear"].ToString() + "&csn=" + csn;
}
} // sqldatareader
} // using sqlcommand
} // using sqlconnection
if (!String.IsNullOrEmpty(url)) {
Response.Redirect(url, false);
}
}
notice that you aren't redirecting until after everything is cleaned up.
SqlConnection, SqlCommand and SqlDataReader all implement IDisposable. You have to properly clean up after use otherwise the resources will be left hanging. The "best" way of doing this is to wrap them in a using clause. This ensures they are properly removed once the code block is exited as they aren't garbage collected like other objects.
Also note that the above code has a good side benefit. Namely, in case of error it STILL cleans up after you. Whereas the original code that was posted would clearly leak in the event the DB server didn't respond or threw some type of error when running the query.
The query could error out depending on the values contained in dboCheck, lastname and mrn parameters. For example, what if "BOB" was passed in for the dobCheck field, or Nothing was passed in for mrn... If dob is a datetime field in your database then the query will throw an error that will result in a leaked connection. Do that enough times and your site is down.
Upon further review, I'm guessing that's probably what is happening: people are putting in garbage data that your app is allowing to get to this point and the query is failing. Most likely this isn't something that you've handled in your test cases.
Side note: Please don't create your sql statements by using concatentation. That is a complete security no no. At the very least, parameterize those queries.
Nice answer Chris, one question aren't the .Close() statements missing in the Using statements?. Both for the connection and the datareader:
results.Close();
} // using sqldatareader
} // using sqlcommand
dbconn.Close();
} // using sqlconnection
We are trying to parse a file and store it in a MySQL database. The commands will be importing a large trace file the could be several gigabytes in size, so it it may of interest for the user to track the progress of the command. We are using the following command:
String commandText = "SET AUTOCOMMIT = 0; "
+ "START TRANSACTION; "
+ "LOAD DATA LOCAL INFILE \'" + filePath + "\' "
+ "INTO TABLE testdatabase.metadata "
+ #"FIELDS TERMINATED BY '\t' "
+ #"LINES TERMINATED BY '\n' "
+ "(Position,"
+ "Timespace,"
+ "Duration,"
+ "Disk,"
+ "Request,"
+ "Sector,"
+ "Length); "
+ "COMMIT;";
Is there a way to track the progress while the command is being executed in order to implement a progress bar?
You can make a progress bar that changes progress based on the stage a query is in. It's definitely possible to put a GUID in a comment somewhere in the query, then use SHOW FULL PROCESSLIST to figure which stage a query is in. But there's no exact way of gauging actual progress. With InnoDB you can try using SHOW INNODB STATUS but even this isn't precise.