c# - Optimize SQLite Select statement - c#

I am using System.Data.SQLite for my database, and my select statements are very slow. It takes around 3-5 minutes to query around 5000 rows of data. Here is the code I am using:
string connectionString;
connectionString = string.Format(#"Data Source={0}", documentsFolder + ";Version=3;New=False;Compress=True;");
//Open a new SQLite Connection
SQLiteConnection conn = new SQLiteConnection(connectionString);
conn.Open();
SQLiteCommand cmd = new SQLiteCommand();
cmd.Connection = conn;
cmd.CommandText = "Select * From urls";
//Assign the data from urls to dr
SQLiteDataReader dr = cmd.ExecuteReader();
SQLiteCommand com = new SQLiteCommand();
com.CommandText = "Select * From visits";
SQLiteDataReader visit = com.ExecuteReader();
List<int> dbID2 = new List<int>();
while (visit.Read())
{
dbID2.Add(int.Parse(visit[1].ToString()));
}
//Read from dr
while (dr.Read())
{
string url = dr[1].ToString();
string title = dr[2].ToString();
long visitlong = Int64.Parse(dr[5].ToString());
string browser = "Chrome";
int dbID = int.Parse(dr[0].ToString());
bool exists = dbID2.Any(item => item == dbID);
int frequency = int.Parse(dr["visit_count"].ToString());
bool containsBoth = url.Contains("file:///");
if (exists)
{
if (containsBoth == false)
{
var form = Form.ActiveForm as TestURLGUI2.Form1;
URLs.Add(new URL(url, title, browser, visited, frequency));
Console.WriteLine(String.Format("{0} {1}", title, browser));
}
}
}
//Close the connection
conn.Close();
And here is another example that takes long:
IEnumerable<URL> ExtractUserHistory(string folder, bool display)
{
// Get User history info
DataTable historyDT = ExtractFromTable("moz_places", folder);
// Get visit Time/Data info
DataTable visitsDT = ExtractFromTable("moz_historyvisits",
folder);
// Loop each history entry
foreach (DataRow row in historyDT.Rows)
{
// Select entry Date from visits
var entryDate = (from dates in visitsDT.AsEnumerable()
where dates["place_id"].ToString() == row["id"].ToString()
select dates).LastOrDefault();
// If history entry has date
if (entryDate != null)
{
// Obtain URL and Title strings
string url = row["Url"].ToString();
string title = row["title"].ToString();
int frequency = int.Parse(row["visit_count"].ToString());
string visit_type;
//Add a URL to list URLs
URLs.Add(new URL(url, title, browser, visited, frequency));
// Add entry to list
// URLs.Add(u);
if (title != "")
{
Console.WriteLine(String.Format("{0} {1}", title, browser));
}
}
}
return URLs;
}
DataTable ExtractFromTable(string table, string folder)
{
SQLiteConnection sql_con;
SQLiteCommand sql_cmd;
SQLiteDataAdapter DB;
DataTable DT = new DataTable();
// FireFox database file
string dbPath = folder + "\\places.sqlite";
// If file exists
if (File.Exists(dbPath))
{
// Data connection
sql_con = new SQLiteConnection("Data Source=" + dbPath +
";Version=3;New=False;Compress=True;");
// Open the Connection
sql_con.Open();
sql_cmd = sql_con.CreateCommand();
// Select Query
string CommandText = "select * from " + table;
// Populate Data Table
DB = new SQLiteDataAdapter(CommandText, sql_con);
DB.Fill(DT);
// Clean up
sql_con.Close();
}
return DT;
}
Now, how can I optimize these so that they are faster?

In addition to moving more of the data aggregation to SQL as joins, you might also consider getting your SQLiteDataReader to provide the data types instead of always parsing the values.
For example, you have the line:
long visitlong = Int64.Parse(dr[5].ToString());
dr[5] is a Sqlite value which you are first converting to a string, then parsing it to a long. These parse operations take time. Why not instead do:
long visitlong = dr.GetInt64(5);
Or:
long visitlong = dr.GetInt64(dr.GetOrdinal("columnName"));
Check out the various methods that SqliteDataReader offers and utilize them whenever possible instead of parsing values.
Edit:
Note that this requires the data be stored as the correct type. If everything in the database is stored as a string, some parsing will be unavoidable.

Make sure that you've recently run the SQL command "ANALYZE {db|table|index};".
I recently ran into a situation where queries were running fast (<1 sec) in my ER software (Navicat), ie: not debugging, but they were very slow (>1 min) debugging in Visual Studio. It turned out that because I did my database design in Navicat (SQLite v3.7), the statistics were not the same as those used by System.Data.SQLite in Visual Studio (v3.8). Running "ANALYZE;" on the entire database file from Visual Studio updated the [sqlite_statX] tables used by v3.8. Both places were the same speed after that.

Related

How to run multiple data readers on single my sql connection

MySqlConnection con = GetConnection();
string sql = "SELECT COUNT(ID) FROM booking WHERE DATE(Booking_Date) = CURDATE()";
string sql2 = "SELECT SUM(Fare) FROM booking WHERE DATE(Booking_Date) = CURDATE()";
MySqlCommand cmd = new MySqlCommand(sql, con);
cmd.CommandType = System.Data.CommandType.Text;
MySqlDataReader BookingToday = cmd.ExecuteReader();
while (BookingToday.Read())
{
label6.Text = BookingToday.GetValue(0).ToString();
}
This is my C# code and I want to run both the given queries to get result at once. But I don't know how to run multiple data readers on single connection or run 2 connections at once. Anyone please help me in this regard
You can use one query:
string sql = "SELECT COUNT(ID), SUM(Fare) FROM booking WHERE DATE(Booking_Date) = CURDATE()";
In this specific case: you don't need to - you can use the code shown in slaakso's answer to perform both aggregates in one query.
In the more general case:
using (var reader = cmd1.ExecuteReader())
{
// while .Read(), etc
}
using (var reader = cmd2.ExecuteReader())
{
// while .Read(), etc
}
i.e. sequentially and not overlapping (unless your provider supports the equivalent of "MARS").
You can also often issue multiple queries (select) in a single command; you use NextResult() to move between them:
using (var reader = cmd.ExecuteReader())
{
// while .Read(), etc, first grid
if (reader.NextResult())
{
// while .Read(), etc, second grid
}
}
You can't open multiple data readers simultaneously from a single connection.
If you want a single result, you can use ExecuteScalar.
var result1 = (DateTime)new MySqlCommand("select now()", con).ExecuteScalar();
Console.WriteLine(result1);
var result2 = (DateTime)new MySqlCommand("select date_add(now(), interval 10 day)", con).ExecuteScalar();
Console.WriteLine(result2);
For multi-row results, you can store them in a DataTable.
var dt1 = new DataTable();
dt1.Load(new MySqlCommand("select id from table1", con).ExecuteReader());
var dt2 = new DataTable();
dt2.Load(new MySqlCommand("select name from table1", con).ExecuteReader());
foreach(var row in dt1.AsEnumerable())
{
Console.WriteLine($"id:{row["id"]}");
}
foreach (var row in dt2.AsEnumerable())
{
Console.WriteLine($"name:{row["name"]}");
}

Can't Figure Out Missing Parameter Error When Running an Oracle Query

I am very new to database queries and even more so, Oracle. I am also new to development work and, believe it or not, am creating this an for work purely out of frustration with the current process. Anyway, I am attempting to collect input from a multi-line text box and run a query. Each line corresponds to a single string that needs to be passed into the WHERE statement and the results will be dumped into a data table. Unfortunately, Oracle has still not released its developer tools for VS2019 so I am having to do this the harder way.
UPDATE # 2:
I have completely rebuilt the query since it was not running even when using known working code from another query. Below is what I have pieced together from various places on the interwebs. While debugging, it appears to parse and format the text correctly and pass it into the OracleParameter without issue. I am getting a Missing Expression error but I don't know what I am missing.
var connString =
ConfigurationManager.ConnectionStrings["dB"].ConnectionString;
string query = "SELECT col1, col2, col3, col4 FROM table WHERE col5 IN (";
using (OracleConnection conn = new OracleConnection(connString))
try
{
var input = "";
input = uniLookup.UniList;
var uniList = string.Join(",", Regex.Split(input, #"(?:\r\n|\n|\r)"));
string allParams = uniList;
string formattedParams = allParams.Replace(" ", string.Empty);
string[] splitParams = formattedParams.Split(',');
List<OracleParameter> parameters = new List<OracleParameter>();
using (OracleCommand cmd = new OracleCommand(query, conn))
{
for (int i = 0; i < splitParams.Length; i++)
{
query += #":Uni" + i + ",";
parameters.Add(new OracleParameter(":Uni" + i, splitParams[i]));
{
query = query.Substring(0, (query.Length - 1));
query += ')';
conn.Open();
using (OracleDataReader reader = cmd.ExecuteReader()) <==ERROR
{
if (!reader.HasRows)
{
while (reader.Read())
{
reader.Read();
{
MessageBox.Show(reader.GetString(1));
}
}
}
You can use IN in your where clause in this way to get rows from multiple values as:
string query = "SELECT dummyCol FROM dummytable WHERE altCol IN ("+text+");";
where you just have to change your text as text="'value1','value2','value3'"; this will not produce any syntax error.
You can convert your multi line text into same comma separated values using this :
foreach (String s in textBox1.Text.Split('\n'))
{
text +="'"+ s+"',";
}
text = text.TrimEnd(',');
this will help you achieve what you need. you can ask If there is any confusion.
Your final code will become :
public void GetData()
{
if (string.IsNullOrWhiteSpace(textbox1.Text) || textbox1.Text == "")
{
MessageBox.Show("Please Enter at least 1 Value and Try Again!");
}
else
{
System.Data.DataTable dt = new System.Data.DataTable();
// string[] lines = textbox1.Text.Split('\n');
string text = "";
foreach (String s in textBox1.Text.Split('\n'))
{
text += "'" + s + "',";
}
text = text.TrimEnd(',');
//Connection Credentials
string credentials = "Credentials";
string query = "SELECT dummyCol FROM dummytable WHERE altCol IN ("+text+");";
OracleConnection conn = new OracleConnection(credentials);
try
{
//Open The Connection
conn.Open();
using (OracleCommand cmd = new OracleCommand(query, conn))
{
//Call the Oracle Reader
using (OracleDataReader reader = cmd.ExecuteReader())
{
if (!reader.HasRows)
{
MessageBox.Show("Unable to Retrieve Data");
return;
}
else if (reader.HasRows)
{
reader.Read();
DataRow row = dt.NewRow();
// create variables to accept reader data for each column
// insert data from query into each column here
dt.Rows.Add(row);
}
}
}
}
}
}

Querying an SQL database using unknown number of values in ASP.NET C#

Okay, I've built a SQL Server database which is being accessed and manipulated by an ASP.NET UI (which I have also developed) to allow others at work to search the DB easily. The database holds data on numerous locations where we have network equipment operating.
I have been asked to build into the UI the capability to query the database for multiple IP addresses - e.g. a user will enter into a textbox "192.168.1.0, 18.15.156.4", click enter and be presented with the results in the gridview. Multiple IP addresses will be separated with a ,.
The code below basically strips out space characters, looks for a , (to determine how many ips there are to query) and if found puts them into an array. A for loop then puts each array item into its own session variable, and from there they are ready to be queried:
protected void siteSearchButton_Click(object sender, EventArgs e)
{
//checks IP search textbox is empty
if (ipQueryTextBox.Text != null)
{
searchErrorLabel.Visible = false;
string addresses = ipQueryTextBox.Text;
//checks for any blank spaces in the addresses variable
if (addresses.Contains(" "))
{
addresses = addresses.Replace(" ", "");
}
//sceens for multiple search items by looking for a ','
if (addresses.Contains(","))
{
//declaring int variables to be used in each of the respective loops
int j = 0;
string[] IParray = addresses.Split(',');
//if i is equal to the length of the "addresses" variable, execute the for loop enclosed
foreach (string s in IParray)
{
Session["IP" + j] = IParray[j];
j = j + 1;
}
}
}
}
As the number of ips to be queried against the database is dynamic I have come to the conclusion that I will have to use C# code (which I'm okay with), but as far as what I've got so far below, I'm unsure how to go about querying the db 'x' amount of times using code presumably I'll need to use a while loop, is anyone able to offer some insight?
//****THE SQL COMMAND BELOW NEEDS ADAPTING TO ALLOW MULTIPLE QUERIES FOR EACH OF THE VALUES STORED IN IParray ---> each session variable
if()
{
//opens a new sqlconnection to read and populate edit textboxes from the Inventory database
using (SqlConnection connection = new SqlConnection("Data Source=localhost;Initial Catalog=Inventory;Integrated Security=True"))
{
//declares SQLCommand type named 'command' and assigns it a string value of SQL code
SqlCommand command =
new SqlCommand("select * from LOCATION WHERE IP_ADDRESS=#IP_ADDRESS", connection);
//outlines parameters
command.Parameters.Add("#IP_ADDRESS", System.Data.SqlDbType.VarChar);
command.Parameters["#IP_ADDRESS"].Value = Session["IP"+j];;
connection.Open();
//opens database connection
SqlDataReader read = command.ExecuteReader();
//while loop will convert each record to string value and print entry into textbox. Will continue untill it runs out of lines
while (read.Read())
{
}
read.Close();
}
}
Instead of using multiple queries, just use SQL's IN clause. It does require a little bit more work to set the query parameters though.
string[] ips = new string[] { "192.168.0.1", "192.168.0.2", "192.168.0.3" };
string[] parameters = ips.Select(
(ip, index) => "#ip" + index.ToString()
).ToArray();
string commandFormat = "SELECT * FROM LOCATION WHERE IP_ADDRESS IN ({0})";
string parameterText = string.Join(",", parameters);
string commandText = string.Format(commandFormat, parameterText);
using (SqlCommand command = new SqlCommand(commandText)) {
for(int i = 0; i < parameters.Length; i++) {
command.Parameters.AddWithValue(parameters[i], ips[i]);
}
}
In the above example, the generated command will be SELECT * FROM LOCATION WHERE IP_ADDRESS IN (#ip1,#ip2,#ip3), and the parameter values will be set accordingly.
(The above solution was very much inspired by this answer.)
First thing - why create multiple session objects when you needs just one to store the values?
I'd try to change your code like this:
if (ipQueryTextBox.Text != null)
{
searchErrorLabel.Visible = false;
string addresses = ipQueryTextBox.Text;
addresses = addresses.Replace(" ", "");
addresses = addresses.Replace(",", "','");
Session["addresses"] = addresses;
}
For the SQL part, you can now easily utilize SQL IN operator, documented here for instance: http://www.w3schools.com/sql/sql_in.asp
SqlCommand command = new SqlCommand("select * from LOCATION WHERE IP_ADDRESS IN (#IP_ADDRESSES)", connection);
command.Parameters.AddWithValue("IP_ADDRESSES", Session["addresses"]);
This should probably work, but I haven't tested it, so it may require some tweaking. Hope you get the idea.
why do you need parameters at all.
//get IP address from UI;
string IPAddress = ipQueryTextBox.Text; //e.g. "192.168.0.1,192.168.0.2,192.168.0.3"
string commandFormat = "SELECT * FROM LOCATION WHERE IP_ADDRESS IN ('" + string.Join("','", IPAddress.split(",")) + "')";
now execute the query
Thank you to everyone who has responded, here is the solution I came up with derived from the answers above:
protected void siteSearchButton_Click(object sender, EventArgs e)
{
//checks IP search textbox is empty
if (ipQueryTextBox.Text != null)
{
searchErrorLabel.Visible = false;
string addresses = ipQueryTextBox.Text;
//checks for any blank spaces in the addresses variable
if (addresses.Contains(" "))
{
addresses = addresses.Replace(" ", "");
}
//sceens for multiple search items by looking for a ','
if (addresses.Contains(","))
{
string[] IParray = addresses.Split(',');
string[] Parameters= IParray.Select((IP, index)=>"#ip"+ index.ToString()).ToArray();
string commandformat ="SELECT * FROM LOCATION WHERE IP_ADDRESS IN ({0})";
string parametertxt= string.Join(",",Parameters);
string commandtxt= string.Format(commandformat,parametertxt);
//creates an SQL connection "connection" opens the connection creates the sql command to be executed & binds and refreshes the gridview
using (SqlConnection connection = new SqlConnection("Data Source=localhost;Initial Catalog=Inventory;Integrated Security=True"))
{
SqlDataReader reader = null;
connection.Open();
SqlCommand command = new SqlCommand(commandtxt, connection);
for(int i =0; i<Parameters.Length; i++)
{
command.Parameters.AddWithValue(Parameters[i],IParray[i]);
}
reader = command.ExecuteReader();
browseSiteGridView.DataSource = reader;
browseSiteGridView.DataBind();
reader.Close();
connection.Close();
}
}
else
{
//creates an SQL connection "connection" opens the connection creates the sql command to be executed & binds and refreshes the gridview
string commandtxt="SELECT * FROM LOCATION WHERE IP_ADDRESS ='"+addresses+"'";
using (SqlConnection connection = new SqlConnection("Data Source=localhost;Initial Catalog=Inventory;Integrated Security=True"))
{
SqlDataReader reader = null;
connection.Open();
SqlCommand command = new SqlCommand(commandtxt, connection);
reader = command.ExecuteReader();
browseSiteGridView.DataSource = reader;
browseSiteGridView.DataBind();
reader.Close();
connection.Close();
}
}
}

Error "There is already an open DataReader associated with this Command which must be closed first" when using 2 distinct commands

I have this legacy code :
private void conecta()
{
if (conexao.State == ConnectionState.Closed)
conexao.Open();
}
public List<string[]> get_dados_historico_verificacao_email_WEB(string email)
{
List<string[]> historicos = new List<string[]>();
conecta();
sql =
#"SELECT *
FROM historico_verificacao_email
WHERE nm_email = '" + email + #"'
ORDER BY dt_verificacao_email DESC, hr_verificacao_email DESC";
com = new SqlCommand(sql, conexao);
SqlDataReader dr = com.ExecuteReader();
if (dr.HasRows)
{
while (dr.Read())
{
string[] dados_historico = new string[6];
dados_historico[0] = dr["nm_email"].ToString();
dados_historico[1] = dr["dt_verificacao_email"].ToString();
dados_historico[1] = dados_historico[1].Substring(0, 10);
dados_historico[2] = dr["hr_verificacao_email"].ToString();
dados_historico[3] = dr["ds_tipo_verificacao"].ToString();
sql =
#"SELECT COUNT(e.cd_historico_verificacao_email) QT
FROM emails_lidos e
WHERE e.cd_historico_verificacao_email =
'" + dr["cd_historico_verificacao_email"].ToString() + "'";
tipo_sql = "seleção";
conecta();
com2 = new SqlCommand(sql, conexao);
SqlDataReader dr3 = com2.ExecuteReader();
while (dr3.Read())
{
//quantidade de emails lidos naquela verificação
dados_historico[4] = dr3["QT"].ToString();
}
dr3.Close();
conexao.Close();
//login
dados_historico[5] = dr["cd_login_usuario"].ToString();
historicos.Add(dados_historico);
}
dr.Close();
}
else
{
dr.Close();
}
conexao.Close();
return historicos;
}
I have created two separates commands to correct the issue, but it still continues: "There is already an open DataReader associated with this Command which must be closed first".
An additional info: the same code is working in another app.
Just add the following in your connection string:
MultipleActiveResultSets=True;
The optimal solution could be to try to transform your solution into a form where you don't need to have two readers open at a time. Ideally it could be a single query. I don't have time to do that now.
If your problem is so special that you really need to have more readers open simultaneously, and your requirements allow not older than SQL Server 2005 DB backend, then the magic word is MARS (Multiple Active Result Sets). http://msdn.microsoft.com/en-us/library/ms345109%28v=SQL.90%29.aspx. Bob Vale's linked topic's solution shows how to enable it: specify MultipleActiveResultSets=true in your connection string. I just tell this as an interesting possibility, but you should rather transform your solution.
in order to avoid the mentioned SQL injection possibility, set the parameters to the SQLCommand itself instead of embedding them into the query string. The query string should only contain the references to the parameters what you pass into the SqlCommand.
You can get such a problem when you are two different commands on same connection - especially calling the second command in a loop. That is calling the second command for each record returned from the first command. If there are some 10,000 records returned by the first command, this issue will be more likely.
I used to avoid such a scenario by making it as a single command.. The first command returns all the required data and load it into a DataTable.
Note: MARS may be a solution - but it can be risky and many people dislike it.
Reference
What does "A severe error occurred on the current command. The results, if any, should be discarded." SQL Azure error mean?
Linq-To-Sql and MARS woes - A severe error occurred on the current command. The results, if any, should be discarded
Complex GROUP BY on DataTable
I suggest creating an additional connection for the second command, would solve it. Try to combine both queries in one query. Create a subquery for the count.
while (dr3.Read())
{
dados_historico[4] = dr3["QT"].ToString(); //quantidade de emails lidos naquela verificação
}
Why override the same value again and again?
if (dr3.Read())
{
dados_historico[4] = dr3["QT"].ToString(); //quantidade de emails lidos naquela verificação
}
Would be enough.
I bet the problem is being shown in this line
SqlDataReader dr3 = com2.ExecuteReader();
I suggest that you execute the first reader and do a dr.Close(); and the iterate historicos, with another loop, performing the com2.ExecuteReader().
public List<string[]> get_dados_historico_verificacao_email_WEB(string email)
{
List<string[]> historicos = new List<string[]>();
conecta();
sql = "SELECT * FROM historico_verificacao_email WHERE nm_email = '" + email + "' ORDER BY dt_verificacao_email DESC, hr_verificacao_email DESC";
com = new SqlCommand(sql, conexao);
SqlDataReader dr = com.ExecuteReader();
if (dr.HasRows)
{
while (dr.Read())
{
string[] dados_historico = new string[6];
dados_historico[0] = dr["nm_email"].ToString();
dados_historico[1] = dr["dt_verificacao_email"].ToString();
dados_historico[1] = dados_historico[1].Substring(0, 10);
//System.Windows.Forms.MessageBox.Show(dados_historico[1]);
dados_historico[2] = dr["hr_verificacao_email"].ToString();
dados_historico[3] = dr["ds_tipo_verificacao"].ToString();
dados_historico[5] = dr["cd_login_usuario"].ToString();
historicos.Add(dados_historico);
}
dr.Close();
sql = "SELECT COUNT(e.cd_historico_verificacao_email) QT FROM emails_lidos e WHERE e.cd_historico_verificacao_email = '" + dr["cd_historico_verificacao_email"].ToString() + "'";
tipo_sql = "seleção";
com2 = new SqlCommand(sql, conexao);
for(int i = 0 ; i < historicos.Count() ; i++)
{
SqlDataReader dr3 = com2.ExecuteReader();
while (dr3.Read())
{
historicos[i][4] = dr3["QT"].ToString(); //quantidade de emails lidos naquela verificação
}
dr3.Close();
}
}
return historicos;
Add MultipleActiveResultSets=true to the provider part of your connection string. See the example below:
<add name="DbContext" connectionString="Data Source=(LocalDb)\v11.0;Initial Catalog=dbName;Persist Security Info=True;User ID=userName;Password=password;MultipleActiveResultSets=True" providerName="System.Data.SqlClient" />
Try to combine the query, it will run much faster than executing an additional query per row.
Ik don't like the string[] you're using, i would create a class for holding the information.
public List<string[]> get_dados_historico_verificacao_email_WEB(string email)
{
List<string[]> historicos = new List<string[]>();
using (SqlConnection conexao = new SqlConnection("ConnectionString"))
{
string sql =
#"SELECT *,
( SELECT COUNT(e.cd_historico_verificacao_email)
FROM emails_lidos e
WHERE e.cd_historico_verificacao_email = a.nm_email ) QT
FROM historico_verificacao_email a
WHERE nm_email = #email
ORDER BY dt_verificacao_email DESC,
hr_verificacao_email DESC";
using (SqlCommand com = new SqlCommand(sql, conexao))
{
com.Parameters.Add("email", SqlDbType.VarChar).Value = email;
SqlDataReader dr = com.ExecuteReader();
while (dr.Read())
{
string[] dados_historico = new string[6];
dados_historico[0] = dr["nm_email"].ToString();
dados_historico[1] = dr["dt_verificacao_email"].ToString();
dados_historico[1] = dados_historico[1].Substring(0, 10);
//System.Windows.Forms.MessageBox.Show(dados_historico[1]);
dados_historico[2] = dr["hr_verificacao_email"].ToString();
dados_historico[3] = dr["ds_tipo_verificacao"].ToString();
dados_historico[4] = dr["QT"].ToString();
dados_historico[5] = dr["cd_login_usuario"].ToString();
historicos.Add(dados_historico);
}
}
}
return historicos;
}
Untested, but maybee gives some idea.

SQLite, Copy DataSet / DataTable to DataBase file

I have filled a DataSet with a Table that was created from another database file. The table is NOT in the database file which I want to be able to copy the Table to.
Now I want to save all those records (DataTable) to a newly created SQLite database file...
How can i do that?
Also I really want to avoid loops if this is possible.
The best answer is by me :) so i'll share it.This is loop but writes 100k entries in 2-3secs.
using (DbTransaction dbTrans = kaupykliuduomConn.BeginTransaction())
{
downloadas.Visible = true; //my progressbar
downloadas.Maximum = dataSet1.Tables["duomenys"].Rows.Count;
using (DbCommand cmd = kaupykliuduomConn.CreateCommand())
{
cmd.CommandText = "INSERT INTO duomenys(Barkodas, Preke, kiekis) VALUES(?,?,?)";
DbParameter Field1 = cmd.CreateParameter();
DbParameter Field2 = cmd.CreateParameter();
DbParameter Field3 = cmd.CreateParameter();
cmd.Parameters.Add(Field1);
cmd.Parameters.Add(Field2);
cmd.Parameters.Add(Field3);
while (n != dataSet1.Tables["duomenys"].Rows.Count)
{
Field1.Value = dataSet1.Tables["duomenys"].Rows[n]["Barkodas"].ToString();
Field2.Value = dataSet1.Tables["duomenys"].Rows[n]["Preke"].ToString();
Field3.Value = dataSet1.Tables["duomenys"].Rows[n]["kiekis"].ToString();
downloadas.Value = n;
n++;
cmd.ExecuteNonQuery();
}
}
dbTrans.Commit();
}
In this case dataSet1.Tables["duomenys"] is already filled with all the data i need to transfer to another database. I used loop to fill dataset too.
When you load the DataTable from the source database, set the AcceptChangesDuringFill property of the data adapter to false, so that loaded records are kept in the Added state (assuming that the source database is SQL Server)
var sqlAdapter = new SqlDataAdapter("SELECT * FROM the_table", sqlConnection);
DataTable table = new DataTable();
sqlAdapter.AcceptChangesDuringFill = false;
sqlAdapter.Fill(table);
Create the table in the SQLite database, by executing the CREATE TABLE statement directly with SQLiteCommand.ExecuteNonQuery
Create a new DataAdapter for the SQLite database connection, and use it to Update the db:
var sqliteAdapter = new SQLiteDataAdapter("SELECT * FROM the_table", sqliteConnection);
var cmdBuilder = new SQLiteCommandBuilder(sqliteAdapter);
sqliteAdapter.Update(table);
If the source and target tables have the same column names and compatible types, it should work fine...
The way to import SQL data to SQLite will take long time. When you want to import data in millions, It will take lot of time. So the shortest and easiest way to do that is just fill fetch the data from SQL database in a DataTable and insert all its rows to SQLite database.
public bool ImportDataToSQLiteDatabase(string Proc, string SQLiteDatabase, params object[] obj)
{
DataTable result = null;
SqlConnection conn = null;
SqlCommand cmd = null;
try
{
result = new DataTable();
using (conn = new SqlConnection(ConStr))
{
using (cmd = CreateCommand(Proc, CommandType.StoredProcedure, obj))
{
cmd.Connection = conn;
conn.Open();
result.Load(cmd.ExecuteReader());
}
}
using (SQLiteConnection con = new SQLiteConnection(string.Format("Data Source={0};Version=3;New=False;Compress=True;Max Pool Size=100;", SQLiteDatabase)))
{
con.Open();
using (SQLiteTransaction transaction = con.BeginTransaction())
{
foreach (DataRow row in result.Rows)
{
using (SQLiteCommand sqlitecommand = new SQLiteCommand("insert into table(fh,ch,mt,pn) values ('" + Convert.ToString(row[0]) + "','" + Convert.ToString(row[1]) + "','"
+ Convert.ToString(row[2]) + "','" + Convert.ToString(row[3]) + "')", con))
{
sqlitecommand.ExecuteNonQuery();
}
}
transaction.Commit();
new General().WriteApplicationLog("Data successfully imported.");
return true;
}
}
}
catch (Exception ex)
{
result = null;
return false;
}
finally
{
if (conn.State == ConnectionState.Open)
conn.Close();
}
}
It will take a very few time as compare to upper given answers.

Categories

Resources