Unity sqlite android database query issue ("Table not found") - c#

Im trying to implement a local database (sqlite) in an Android/PC game
I am able to load the database from the editor and the standalone pc build of the games but the android version has an error that does not find the tables im trying to query from
This is the code that loads the database into three lists from the streaming assets folders
using System;
using System.IO;
using System.Data;
using System.Collections;
using System.Collections.Generic;
using Mono.Data.Sqlite;
using UnityEngine;
using UnityEngine.SceneManagement;
public class DBAdministrator : MonoBehaviour {
//
//Android sdk > platform-tools > adb logcat MarsGames.ChippyRemix *:E
public List<LevelDP> levelsList;
public List<LevelDP> defaultLevels;
public List<LevelDP> myLevels;
public List<LevelDP> otherLevels;
GameObject playMenu;
public string myName;
public string defaultName = "chippy";
public string debugString;
void Start () {
//myName = PlayerPrefs.GetString("MyName");
playMenu = GameObject.FindWithTag("UI");
//levelsList = new List<LevelDP>();
defaultLevels = new List<LevelDP>();
myLevels = new List<LevelDP>();
otherLevels = new List<LevelDP>();
//Fill the list
FillLists();
//PrintAllLevels();
if(playMenu != null){
playMenu.SendMessage("CreateDisplayLists");
}
}
void FillLists(){
//Path to database
string conn = "";
//Debug.Log(Application.streamingAssetsPath);
#if UNITY_EDITOR_WIN
//Debug.Log("Using unity editor conn");
conn = "URI=file:" + Application.dataPath + "/StreamingAssets/chickdb.db";
#elif UNITY_ANDROID
debugString = "Using andriod";
conn = Application.persistentDataPath + "/chickdb.db";
if(!File.Exists(conn)){
debugString = "DB file does not exist";
//Open Streaming assets and load the db
WWW loadDb = new WWW("jar:file://" + Application.dataPath + "!/assets/chickdb.db");
while (!loadDb.isDone) { }
File.WriteAllBytes(conn,loadDb.bytes);
}else{
debugString = "File exists";
//debugString = conn;
}
//Once file is loaded, use the appropiate filepath to access db
conn = "URI=file:" + Application.persistentDataPath + "/chickdb.db";
#elif UNITY_STANDALONE
//debugString = "Using standalone PC";
//string conn = "URI=file:" + System.IO.Path.Combine(Application.streamingAssetsPath, "Database/TMDB.s3db");
//conn = "URI=File:" + System.IO.Path.Combine(Application.streamingAssetsPath,"/StreamingAssets/chickdb.db");
conn = "URI=file:" + Application.streamingAssetsPath + "/chickdb.db";
debugString = conn;
/*
#elif UNITY_IOS
debugString = "Using ios";
// this is the path to your StreamingAssets in iOS
var loadDb = Application.dataPath + "/Raw/chickdb.do";
// then save to Application.persistentDataPath
File.Copy(loadDb, filepath);
//conn = "URI=file " + Application.persistentDataPath + "/chickdb.db";
*/
#endif
//Debug.Log("ST Connection DB: " + conn);
#if UNITY_ANDROID
debugString = "Connection Attempted";
#endif
IDbConnection dbconn = (IDbConnection) new SqliteConnection(conn);
//Open connection to the database.
dbconn.Open();
#if UNITY_ANDROID
debugString = "Database Open";
#endif
IDbCommand dbcmd = dbconn.CreateCommand();
string sqlQuery = "SELECT * FROM levels";
dbcmd.CommandText = sqlQuery;
IDataReader reader = dbcmd.ExecuteReader();
#if UNITY_ANDROID
debugString = "Query executed";
#endif
while (reader.Read()){
LevelDP lp = new LevelDP();
lp.SetLevelID(reader.GetInt32(0));
lp.SetLevelName(reader.GetString(1));
lp.SetCreationDate(reader.GetString(2));
lp.SetUserID(reader.GetString(3));
lp.SetBlocks(reader.GetString(4));
lp.SetCompleted(reader.GetInt32(5));
//levelsList.Add(lp);
if(lp.GetUserID().Equals(defaultName)){
defaultLevels.Add(lp);
}else if(lp.GetUserID().Equals(myName)){
myLevels.Add(lp);
}else{
otherLevels.Add(lp);
}
}
//Close the db reader
reader.Close();
reader = null;
//Close the comand executer
dbcmd.Dispose();
dbcmd = null;
//Close the db connection
dbconn.Close();
dbconn = null;
}
void PrintAllLevels(){
Debug.Log("Print all levels");
for(int i = 0; i < defaultLevels.Count; i++){
Debug.Log(defaultLevels[i].ToString());
}
for(int i = 0; i < myLevels.Count; i++){
Debug.Log(myLevels[i].ToString());
}
for(int i = 0; i < otherLevels.Count; i++){
Debug.Log(otherLevels[i].ToString());
}
}
}
From this code i've been able to print ("Database Open") but when the database is supposed to query i get an error (from the adb tool) that states ("Table levels not found") and i dont know where the error is
I do have to add the editor, and pc build work just fine,
Steps i've followed are:
Place the .db file in the Streaming Assets folder Unpack the .apk and check the values present in the .db file via sqlitebrowser
i have a theory that maybe im not loading the db correctly into memory in the #if UNITY_ANDROID conditional statement but almost every other example ive seen works this way


I can't comment, that's why Im writing an answer.
Try to use Debug.Log to see the pathfile when you are testing the game/app.
And then checking if your sq3db file is in the correct folder.
I Have a similar code for streaming the sqlite DB but if I test it on PC the DB should be in Assets folder not in StreamingAssets folder.
Edit 1:
First:
I'll show you a fragment of the code I use when I make games:
if(Application.platform != RuntimePlatform.Android)
_filepath = Application.dataPath + "/" + _DBName;
else
{
_filepath = Application.persistentDataPath + "/" + _DBName;
if (!File.Exists(_filepath))
{
//Debug.Log("Check in");
WWW loadDB = new WWW("jar:file://" + Application.dataPath +
"!/assets/" + _DBName);
while (!loadDB.isDone) { }
File.WriteAllBytes(_filepath, loadDB.bytes);
}
}
_StringConnection = "URI=file:" + _filepath;
Second:
When I test the game on pc (unity editor) I have my Database (.s3db) here:
ProjectName/Assets/Database.s3db
When I run the game on android I have my Database here:
ProjectName/Assets/StreamingAssets/Database.s3db
Something you may notice is that my Database file extension is s3db, I use SQlite3, but it should work anyway.


In Android sqlite db should save in persistent data path. I put my sqlite db in resources with .txt extension and read it as text. then chagne the extension in code and save it in Application.persistentDataPath.

Related

Error when running on device, not in Unity editor

I am trying to upload a new version of my game to my iPhone via Xcode. With this app this has not been a problem before but now I do have a problem.
So I created a new smaller project to test. It works in the editor but not when upload to my iPhone (IOS).
I am running Unity 2021.2.7f1 Apple Silicon.
The problem is related to some missing module so I cannot open the db from IOS.
I think it looks like something with opening the database but I am not sure. The database is copied to the "Application.persistentDataPath" path for sure.
I am running Unity 2021.2.7f1 Apple Silicon.
"dbconn.Open();" is where it fails on IOS but not the Mac. I have not yet tested on Android.
Here is the full code in my test project:
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using TMPro;
using Mono.Data.Sqlite;
using System.Data;
using System.IO;
using System;
public class TestScript : MonoBehaviour
{
public TMP_Text txt_Output;
private string dbPath;
private string myRecord;
private static string fileName;
private static IDbConnection dbconn;
private static string connection, checkDbPath;
private static IDbCommand dbcmd;
private static string sqlQuery;
private static IDataReader reader;
private int recCounter;
void Start()
{
//dbPath = Application.dataPath + "/StreamingAssets/TestDB.db";
dbPath = InitDBLocation();
txt_Output.text = "NU STARTAR VI!";
DoTheDB();
}
void DoTheDB()
{
connection = "URI=file:" + dbPath; //Path to database.
dbconn = (IDbConnection)new SqliteConnection(connection); //creates database connection
dbconn.Open();
//sqlQuery = "SELECT MyTable, Name, answer2, answer3, answer4, lvl from questions WHERE id = '" + _idNr.ToString() + "'";
sqlQuery = "SELECT Name FROM MyTable;";
dbcmd = dbconn.CreateCommand();
dbcmd.CommandText = sqlQuery;
reader = dbcmd.ExecuteReader();
while (reader.Read())
{
myRecord = reader.GetString(0);
recCounter++;
}
reader.Close();
dbconn.Close();
reader = null;
dbcmd.Dispose();
dbcmd = null;
dbconn = null;
txt_Output.text = "# RECORDS: " + recCounter.ToString();
}
public static string InitDBLocation()
{
fileName = "TestDB.db";
#if UNITY_EDITOR
Debug.Log("\n #if UNITY_EDITOR \n");
return Application.dataPath + "/StreamingAssets/TestDB.db";
#elif UNITY_ANDROID
Debug.Log("\n #elif UNITY_ANDROID \n");
Debug.Log("========= GET ANDROID PATH =========");
PlayerPrefs.SetString("DB ERROR", "");
oldCurrentDbPathVersion = PlayerPrefs.GetInt("currentDbPathVersion");
Debug.Log("oldCurrentDbPathVersion: " + oldCurrentDbPathVersion + " | currentDbPathVersion: " + currentDbPathVersion);
if (File.Exists(Path.Combine(Application.persistentDataPath, fileName)) && oldCurrentDbPathVersion == currentDbPathVersion)
{
Debug.Log("return Path: " + Path.Combine(Application.persistentDataPath, fileName));
return Path.Combine(Application.persistentDataPath, fileName);
}
else
{
Debug.Log("Copy Database");
if (File.Exists(Path.Combine(Application.persistentDataPath, fileName)))
{
File.Delete(Path.Combine(Application.persistentDataPath, fileName));
}
PlayerPrefs.SetInt("currentDbPathVersion", currentDbPathVersion);
// °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°° //
try
{
var loadingRequest = UnityWebRequest.Get(Path.Combine(Application.streamingAssetsPath, fileName));
loadingRequest.SendWebRequest();
while (!loadingRequest.isDone)
{
if (loadingRequest.isNetworkError || loadingRequest.isHttpError)
{
break;
}
}
Debug.Log("Write database to persistent");
File.WriteAllBytes(Path.Combine(Application.persistentDataPath, fileName), loadingRequest.downloadHandler.data);
Debug.Log("return Path: " + Path.Combine(Application.persistentDataPath, fileName));
return Path.Combine(Application.persistentDataPath, fileName);
}
catch (Exception e)
{
PlayerPrefs.SetString("DB ERROR", e.ToString());
PlayerPrefs.SetString("dbPath", "");
Debug.LogError("*** FILECOPY ERROR ***\n\n" + e); //" + exception.Message + "\n" + exception.StackTrace);
return null;
}
Debug.Log("========= END ANDROID dbPath=========");
// °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°° //
}
#elif UNITY_IOS
Debug.Log("\n #elif UNITY_IOS \n");
Debug.Log("TEST GENERAL: " + Application.streamingAssetsPath);
// ========= IOS ========= //
Debug.Log("========= IOS DB PATH =========");
if (File.Exists(Path.Combine(Application.persistentDataPath, fileName)))
{
Debug.Log("\n>>>> DB File.Exists <<<<<<\n");
return Path.Combine(Application.persistentDataPath, fileName);
}
else
{
if (File.Exists(Path.Combine(Application.persistentDataPath, fileName)))
{
File.Delete(Path.Combine(Application.persistentDataPath, fileName));
}
// °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°° //
try
{
System.IO.File.Copy((Path.Combine(UnityEngine.Application.streamingAssetsPath, fileName)), Path.Combine(Application.persistentDataPath, fileName), true);
Debug.Log("\n return Path: " + Path.Combine(Application.persistentDataPath, fileName));
Debug.Log("\n To PATH: " + Application.persistentDataPath + "\" + fileName);
return Path.Combine(Application.persistentDataPath, fileName);
}
catch (Exception e)
{
Debug.LogError("*** FILECOPY ERROR ***\n\n" + e); //" + exception.Message + "\n" + exception.StackTrace);
PlayerPrefs.SetString("DB ERROR", e.ToString());
PlayerPrefs.SetString("dbPath", "");
return null;
}
Debug.Log("========= END IOS dbPath=========");
// °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°° //
}
#endif
}
}
Here is the Xcode log:
2022-01-08 22:41:04.039955+0100 NewSQLitetest[2350:900169] Built from '2021.2/staging' branch, Version '2021.2.7f1 (6bd9e232123f)', Build type 'Release', Scripting Backend 'il2cpp'
2022-01-08 22:41:04.040632+0100 NewSQLitetest[2350:900169] MemoryManager: Using 'Default' Allocator.
[UnityMemory] Configuration Parameters - Can be set up in boot.config
"memorysetup-bucket-allocator-granularity=16"
"memorysetup-bucket-allocator-bucket-count=8"
"memorysetup-bucket-allocator-block-size=4194304"
"memorysetup-bucket-allocator-block-count=1"
"memorysetup-main-allocator-block-size=16777216"
"memorysetup-thread-allocator-block-size=16777216"
"memorysetup-gfx-main-allocator-block-size=16777216"
"memorysetup-gfx-thread-allocator-block-size=16777216"
"memorysetup-cache-allocator-block-size=4194304"
"memorysetup-typetree-allocator-block-size=2097152"
"memorysetup-profiler-bucket-allocator-granularity=16"
"memorysetup-profiler-bucket-allocator-bucket-count=8"
"memorysetup-profiler-bucket-allocator-block-size=4194304"
"memorysetup-profiler-bucket-allocator-block-count=1"
"memorysetup-profiler-allocator-block-size=16777216"
"memorysetup-profiler-editor-allocator-block-size=1048576"
"memorysetup-temp-allocator-size-main=4194304"
"memorysetup-job-temp-allocator-block-size=2097152"
"memorysetup-job-temp-allocator-block-size-background=1048576"
"memorysetup-job-temp-allocator-reduction-small-platforms=262144"
"memorysetup-temp-allocator-size-background-worker=32768"
"memorysetup-temp-allocator-size-job-worker=262144"
"memorysetup-temp-allocator-size-preload-manager=262144"
"memorysetup-temp-allocator-size-nav-mesh-worker=65536"
"memorysetup-temp-allocator-size-audio-worker=65536"
"memorysetup-temp-allocator-size-cloud-worker=32768"
"memorysetup-temp-allocator-size-gfx=262144"
-> applicationDidFinishLaunching()
-> applicationDidBecomeActive()
GfxDevice: creating device client; threaded=1; jobified=1
Initializing Metal device caps: Apple A13 GPU
Initialize engine version: 2021.2.7f1 (6bd9e232123f)
2022-01-08 22:41:04.410523+0100 NewSQLitetest[2350:900465] fopen failed for data file: errno = 2 (No such file or directory)
2022-01-08 22:41:04.410600+0100 NewSQLitetest[2350:900465] Errors found! Invalidating cache...
2022-01-08 22:41:04.831038+0100 NewSQLitetest[2350:900465] fopen failed for data file: errno = 2 (No such file or directory)
2022-01-08 22:41:04.831115+0100 NewSQLitetest[2350:900465] Errors found! Invalidating cache...
2022-01-08 22:41:05.052099+0100 NewSQLitetest[2350:900169] Unbalanced calls to begin/end appearance transitions for <UnityDefaultViewController: 0x103429f00>.
UnloadTime: 1.043667 ms
#elif UNITY_IOS
TestScript:InitDBLocation()
TestScript:Start()
TEST GENERAL: /private/var/containers/Bundle/Application/2FE4741C-1580-4698-9BC1-9512D578CFF6/NewSQLitetest.app/Data/Raw
TestScript:InitDBLocation()
TestScript:Start()
========= IOS DB PATH =========
TestScript:InitDBLocation()
TestScript:Start()
return Path: /var/mobile/Containers/Data/Application/F3747560-C7AD-4FBC-A1C6-F95843F09906/Documents/TestDB.db
TestScript:InitDBLocation()
TestScript:Start()
To PATH: /var/mobile/Containers/Data/Application/F3747560-C7AD-4FBC-A1C6-F95843F09906/DocumentsTestDB.db
TestScript:InitDBLocation()
TestScript:Start()
MissingMethodException: System.Runtime.InteropServices.Marshal::SetLastWin32Error(System.Int32)
at System.Runtime.InteropServices.CriticalHandle.Cleanup () [0x00000] in <00000000000000000000000000000000>:0
at Mono.Data.Sqlite.SqliteStatement.Dispose () [0x00000] in <00000000000000000000000000000000>:0
at Mono.Data.Sqlite.SqliteCommand.ClearCommands () [0x00000] in <00000000000000000000000000000000>:0
at Mono.Data.Sqlite.SqliteCommand.set_CommandText (System.String value) [0x00000] in <00000000000000000000000000000000>:0
at Mono.Data.Sqlite.SqliteConnection.Open () [0x00000] in <00000000000000000000000000000000>:0
at TestScript.DoTheDB () [0x00000] in <00000000000000000000000000000000>:0

I finally solved this problem based on the above post, see thread in my comment. Comment from Baydogan:
Finally I have found a workaround until unity fixes this bug.
Don't modify any unity editor file. Just build for iOS as usual.
Search for "// System.Void System.Runtime.InteropServices.CriticalHandle::Cleanup()" on xcode project.
Move a few lines below and comment out the line that starts with : "Marshal_SetLastWin32Error_"
DONE
example:
IL_002b:
{ il2cpp_codegen_runtime_class_init_inline(Marshal_tD976A56A90263C3CE2B780D4B1CADADE2E70B4A7_il2cpp_TypeInfo_var);
// Marshal_SetLastWin32Error_mC871D8DAC36418FAA95AFD23CD2A3ECC94628D1C(G_B6_0, NULL); il2cpp_codegen_runtime_class_init_inline(GC_t920F9CF6EBB7C787E5010A4352E1B587F356DC58_il2cpp_TypeInfo_var);
GC_SuppressFinalize_m3352E2F2119EB46913B51B7AAE2F217C63C35F2A(__this, NULL);
return;
}

How to open multiple Microsoft Access (mdb) files in C#?

Referring to this question, I figured out how open and connect to a single mdb file.
At the moment I am doing:
String accessConnectionString = #"Provider=Microsoft.ACE.OLEDB.12.0;
Data Source=C:\MyMDB\MyMDBFile.mdb;
Persist Security Info = False; ";
using (OleDbConnection accessConnection = new OleDbConnection(accessConnectionString))
{
ReadContent();
}
But I want to open multiple files from the directory, basically I want to:
String[] mdbFiles = Directory.GetFiles(#"C:\MyMDB\", "*.mdb");
And use this in the accessConnectionString
I know it should be something like,
foreach (var filePath in mdbFiles)
{
accessConnectionString = #"Provider=Microsoft.ACE.OLEDB.12.0;"
+ "Data Source=" + filePath + " Persist Security Info = False; ";
}
But is this the only way to access multiple mdb files?

Relative path not working while accessing a sqlite Database through C#

I am creating a Excel Addin through which i want to access a database. code is as follows
[ExcelFunction("My First Excel-DNA Function")]
public static string GreetFunction(string name)
{
GetConnection();
return "Hello" + " " + name;
}
public static void GetConnection()
{
//db = new SQLiteConnection("Data Source="+System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase)+"\\Database\\XLSQLiteDemo.sqlite");
db = new SQLiteConnection("Data Source=Database/XLSQLiteDemo.sqlite");
try
{
db.Open();
cmd = db.CreateCommand();
System.Windows.MessageBox.Show("Connection created");
}
catch (SQLiteException ex)
{
System.Windows.MessageBox.Show(ex.ToString());
}
}
so when i give absolute path like c:/test/firstlibrary.../XLSQLiteDemo.sqlite it works.
but when i use relative path like db = new SQLiteConnection("Data Source=Database/XLSQLiteDemo.sqlite");
it throws an exception: unable to open database file error code 14.
the code which is in comment i.e.
//db = new SQLiteConnection("Data Source="+System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase)+"\\Database\\XLSQLiteDemo.sqlite");
also doesn't work i.e. it calculates the absolute path but when i tried to debug; debugging is automatically terminated after db.Open();
and output in excel sheet is also #Value which indicates some error.

#adrino may be the "file" word in your string is the problem.remove it.
string relativePath = #"Database\XLSQLiteDemo.sqlite";
string currentPath = System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase);
string absolutePath = System.IO.Path.Combine(currentPath, relativePath);
absolutePath=absolutePath.Remove(0, 6);//this code is written to remove file word from absolute path
string connectionString = string.Format("Data Source={0}", absolutePath);
this works on my machine.tell me if its correct.

sqlBulk insert C#

I have a page where I want to upload a CSV file from my computer to database on the server and I have my opentext that looks like the following
using (StreamReader sr = File.OpenText(#"c:\users\workstationUsername\FileName.csv"))
This works fine on my local machine but when I push this to the server it tries to read the server's C Drive and I want it to read the physical file location that is sitting on the desktop of the user's computer not the server, when they click browse and upload..
Thank you
below is the complete code:
if (IsPostBack)
{
// SetDefaultDates();
Boolean fileOK = false;
String dateString = DateTime.Now.ToString("MMddyyyy");
String UserName = User.Identity.Name;
String path = Server.MapPath("~/Uploads/CSVs/");
string stringpath = Environment.GetFolderPath(Environment.SpecialFolder.DesktopDirectory);
String fileName = System.IO.Path.GetFileName(FileUpload1.PostedFile.FileName);
stringpath = stringpath + fileName;
String LocationToSave = path + "\\" + fileName;
if (FileUpload1.HasFile)
{
String fileExtension =
System.IO.Path.GetExtension(FileUpload1.FileName).ToLower();
String[] allowedExtensions = { ".csv" };
for (int i = 0; i < allowedExtensions.Length; i++)
{
if (fileExtension == allowedExtensions[i])
{
fileOK = true;
}
}
}
if (fileOK)
{
try
{
//FileUpload1.PostedFile.SaveAs(LocationToSave + dateString + "-" + FileUpload1.FileName);
FileUpload1.PostedFile.SaveAs(LocationToSave);
Label1.Text = "File " + FileUpload1.FileName + " uploaded!";
DataTable dt = new DataTable();
string line = null;
int i = 0;
using (StreamReader sr = File.OpenText(stringpath))
{
while ((line = sr.ReadLine()) != null)
{
string[] data = line.Split(',');
if (data.Length > 0)
{
if (i == 0)
{
foreach (var item in data)
{
dt.Columns.Add(new DataColumn());
}
i++;
}
DataRow row = dt.NewRow();
row.ItemArray = data;
dt.Rows.Add(row);
}
}
}
using (SqlConnection cn = new SqlConnection(ConfigurationManager.ConnectionStrings["Myconnection"].ConnectionString))
{
cn.Open();
using (SqlBulkCopy copy = new SqlBulkCopy(cn))
{
copy.WriteToServer(dt);
}
}
}
catch (Exception ex)
{
Label1.Text = "File " + FileUpload1.FileName + " could not be uploaded." + ex.Message;
}
}
else
{
Label1.Text = "Cannot accept files of this type. " + FileUpload1.FileName;
}
}
SetDefaultDates();
}

If you have a FileUpload control, then instead of using (StreamReader sr = File.OpenText(#"c:\users\workstationUsername\FileName.csv")) which obvously is pointing to the server's hard drive you can do this:
(StreamReader sr = new StreamReader(fileUploadControl.FileContent))
//Do your stuff

You can't access the client's hard drive. That's a major security concern. You'll need to upload the file to your server, and read it from there.

It doesnt make sense to have a static read to the local machine, rather get user to upload it then update the database, this code is very limiting and has a high security risk. Rather create a steamreader object get the user to upload it then use the steam reader to process the csv.

import from text file to SQL Server Database, is ADO.NET too slow?

My program is now still running to import data from a log file into a remote SQL Server Database. The log file is about 80MB in size and contains about 470000 lines, with about 25000 lines of data. My program can import only 300 rows/second, which is really bad. :(
public static int ImportData(string strPath)
{
//NameValueCollection collection = ConfigurationManager.AppSettings;
using (TextReader sr = new StreamReader(strPath))
{
sr.ReadLine(); //ignore three first lines of log file
sr.ReadLine();
sr.ReadLine();
string strLine;
var cn = new SqlConnection(ConnectionString);
cn.Open();
while ((strLine = sr.ReadLine()) != null)
{
{
if (strLine.Trim() != "") //if not a blank line, then import into database
{
InsertData(strLine, cn);
_count++;
}
}
}
cn.Close();
sr.Close();
return _count;
}
}
InsertData is just a normal insert method using ADO.NET. It uses a parsing method:
public Data(string strLine)
{
string[] list = strLine.Split(new[] {'\t'});
try
{
Senttime = DateTime.Parse(list[0] + " " + list[1]);
}
catch (Exception)
{
}
Clientip = list[2];
Clienthostname = list[3];
Partnername = list[4];
Serverhostname = list[5];
Serverip = list[6];
Recipientaddress = list[7];
Eventid = Convert.ToInt16(list[8]);
Msgid = list[9];
Priority = Convert.ToInt16(list[10]);
Recipientreportstatus = Convert.ToByte(list[11]);
Totalbytes = Convert.ToInt32(list[12]);
Numberrecipient = Convert.ToInt16(list[13]);
DateTime temp;
if (DateTime.TryParse(list[14], out temp))
{
OriginationTime = temp;
}
else
{
OriginationTime = null;
}
Encryption = list[15];
ServiceVersion = list[16];
LinkedMsgid = list[17];
MessageSubject = list[18];
SenderAddress = list[19];
}
InsertData method:
private static void InsertData(string strLine, SqlConnection cn)
{
var dt = new Data(strLine); //parse the log line into proper fields
const string cnnStr =
"INSERT INTO LOGDATA ([SentTime]," + "[client-ip]," +
"[Client-hostname]," + "[Partner-Name]," + "[Server-hostname]," +
"[server-IP]," + "[Recipient-Address]," + "[Event-ID]," + "[MSGID]," +
"[Priority]," + "[Recipient-Report-Status]," + "[total-bytes]," +
"[Number-Recipients]," + "[Origination-Time]," + "[Encryption]," +
"[service-Version]," + "[Linked-MSGID]," + "[Message-Subject]," +
"[Sender-Address]) " + " VALUES ( " + "#Senttime," + "#Clientip," +
"#Clienthostname," + "#Partnername," + "#Serverhostname," + "#Serverip," +
"#Recipientaddress," + "#Eventid," + "#Msgid," + "#Priority," +
"#Recipientreportstatus," + "#Totalbytes," + "#Numberrecipient," +
"#OriginationTime," + "#Encryption," + "#ServiceVersion," +
"#LinkedMsgid," + "#MessageSubject," + "#SenderAddress)";
var cmd = new SqlCommand(cnnStr, cn) {CommandType = CommandType.Text};
cmd.Parameters.AddWithValue("#Senttime", dt.Senttime);
cmd.Parameters.AddWithValue("#Clientip", dt.Clientip);
cmd.Parameters.AddWithValue("#Clienthostname", dt.Clienthostname);
cmd.Parameters.AddWithValue("#Partnername", dt.Partnername);
cmd.Parameters.AddWithValue("#Serverhostname", dt.Serverhostname);
cmd.Parameters.AddWithValue("#Serverip", dt.Serverip);
cmd.Parameters.AddWithValue("#Recipientaddress", dt.Recipientaddress);
cmd.Parameters.AddWithValue("#Eventid", dt.Eventid);
cmd.Parameters.AddWithValue("#Msgid", dt.Msgid);
cmd.Parameters.AddWithValue("#Priority", dt.Priority);
cmd.Parameters.AddWithValue("#Recipientreportstatus", dt.Recipientreportstatus);
cmd.Parameters.AddWithValue("#Totalbytes", dt.Totalbytes);
cmd.Parameters.AddWithValue("#Numberrecipient", dt.Numberrecipient);
if (dt.OriginationTime != null)
cmd.Parameters.AddWithValue("#OriginationTime", dt.OriginationTime);
else
cmd.Parameters.AddWithValue("#OriginationTime", DBNull.Value);
//if OriginationTime was null, then insert with null value to this column
cmd.Parameters.AddWithValue("#Encryption", dt.Encryption);
cmd.Parameters.AddWithValue("#ServiceVersion", dt.ServiceVersion);
cmd.Parameters.AddWithValue("#LinkedMsgid", dt.LinkedMsgid);
cmd.Parameters.AddWithValue("#MessageSubject", dt.MessageSubject);
cmd.Parameters.AddWithValue("#SenderAddress", dt.SenderAddress);
cmd.ExecuteNonQuery();
}
How can my program run faster?
Thank you so much!

Use SqlBulkCopy.
Edit: I created a minimal implementation of IDataReader and created a Batch type so that I could insert arbitrary in-memory data using SqlBulkCopy. Here is the important bit:
IDataReader dr = batch.GetDataReader();
using (SqlTransaction tx = _connection.BeginTransaction())
{
try
{
using (SqlBulkCopy sqlBulkCopy =
new SqlBulkCopy(_connection, SqlBulkCopyOptions.Default, tx))
{
sqlBulkCopy.DestinationTableName = TableName;
SetColumnMappings(sqlBulkCopy.ColumnMappings);
sqlBulkCopy.WriteToServer(dr);
tx.Commit();
}
}
catch
{
tx.Rollback();
throw;
}
}
The rest of the implementation is left as an exercise for the reader :)
Hint: the only bits of IDataReader you need to implement are Read, GetValue and FieldCount.

Hmmm, let's break this down a little bit.
In pseudocode what you did is the ff:
Open the file
Open a connection
For every line that has data:
Parse the string
Save the data in SQL Server
Close the connection
Close the file
Now the fundamental problems in doing it this way are:
You are keeping a SQL connection open while waiting for your line parsing (pretty susceptible to timeouts and stuff)
You might be saving the data line by line, each in its own transaction. We won't know until you show us what the InsertData method is doing
Consequently you are keeping the file open while waiting for SQL to finish inserting
The optimal way of doing this is to parse the file as a whole, and then insert them in bulk. You can do this with SqlBulkCopy (as suggested by Matt Howells), or with SQL Server Integration Services.
If you want to stick with ADO.NET, you can pool together your INSERT statements and then pass them off into one large SQLCommand, instead of doing it this way e.g., setting up one SQLCommand object per insert statement.

You create the SqlCommand object for every row of data. The simplest improvement would therefore to create a
private static SqlCommand cmdInsert
and declare the parameters with the Parameters.Add() method. Then for each data row, set the parameter values using
cmdInsert.Parameters["#paramXXX"].Value = valueXXX;
A second performance improvement might be to skip creation of Data objects for each row, and assign Parameter values directly from the list[] array.

Categories

Resources