We are trying to parse a file and store it in a MySQL database. The commands will be importing a large trace file the could be several gigabytes in size, so it it may of interest for the user to track the progress of the command. We are using the following command:
String commandText = "SET AUTOCOMMIT = 0; "
+ "START TRANSACTION; "
+ "LOAD DATA LOCAL INFILE \'" + filePath + "\' "
+ "INTO TABLE testdatabase.metadata "
+ #"FIELDS TERMINATED BY '\t' "
+ #"LINES TERMINATED BY '\n' "
+ "(Position,"
+ "Timespace,"
+ "Duration,"
+ "Disk,"
+ "Request,"
+ "Sector,"
+ "Length); "
+ "COMMIT;";
Is there a way to track the progress while the command is being executed in order to implement a progress bar?
You can make a progress bar that changes progress based on the stage a query is in. It's definitely possible to put a GUID in a comment somewhere in the query, then use SHOW FULL PROCESSLIST to figure which stage a query is in. But there's no exact way of gauging actual progress. With InnoDB you can try using SHOW INNODB STATUS but even this isn't precise.
Related
I am using c# sqlite version 3. My application requires saving up to 20 columns of data at 4000Hz. To accomplish this I am using threads that take commands from a queue and periodically write/read to the database. Currently, I am just working on the writing portion and have found that ExecuteNonQueryAsync() and ExecuteNonQuery() have relatively the same execution time which makes no sense to me. My function is as follows:
public async Task<int> dumpStorage(string cmdStr)
{
int r;
using (SQLiteCommand cmd = new SQLiteCommand(sqliteConnection))
{
string times = "";
Stopwatch tTime = new Stopwatch();
tTime.Start();
cmd.CommandText = cmdStr;
times += "tTime1 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
tTime.Restart();
r = await cmd.ExecuteNonQueryAsync();
//cmd.ExecuteNonQuery();
//cmd.ExecuteScalar();
//cmd.ExecuteScalarAsync();
times += "tTime2 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
tTime.Restart();
transaction.Commit();
times += "tTime2 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
tTime.Restart();
transaction = sqliteConnection.BeginTransaction();
tTime.Stop();
times += "tTime3 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
timeStr = times;
storeToDBWatchDog.Restart();
}
return r;
}
at the moment I am saving 40,000 rows with 20 (plus a primary key) columns which represents 10 seconds of data. I am unable to get the async to have any time advantage over the non-async function. in the future I will need to introduce a scheme that will manage if there was any failures to insert.
I would expect that ExecuteNonQueryAsync() should make tTime2 essentially 0 and it would work in the background however that is not what I am seeing as it appear to wait for the ...Async() function to complete.
the execute non query async and non-async require roughly 2.1 seconds. The end goal is to save the data at least 20 times faster than it is coming in. The only time that this becomes very important is on initial load. By this I mean i am reading an existing file (not in a database) and then saving it into the database. so, at the moment I only have approximately 4 times the performance. by this i mean it would take 5 minutes to save a file that consists of data associated with 20 minutes. this seems unexceptionably slow.
typical time output regardless of which function call is used (both scalar functions take ~50% longer):
Edit:
I recently came across this unfortunate post, which essentially states that async doesn't do anything so you should use the WAL setting:
https://learn.microsoft.com/en-us/dotnet/standard/data/sqlite/async
i have since changed my code to include the WAL journal-mode:
connectionString = "Data Source=" + filename + "; Version=3" + "; PRAGMA journal_mode=WAL;";
sqliteConnection = new SQLiteConnection(connectionString);
I did not see any performance increases setting the journal_mode to WAL.
I am using c# sqlite version 3. My application requires saving up to 20 columns of data at 4000Hz. To accomplish this I am using threads that take commands from a queue and periodically write/read to the database. Currently, I am just working on the writing portion and have found that ExecuteNonQueryAsync() and ExecuteNonQuery() have relatively the same execution time which makes no sense to me. My function is as follows:
public async Task<int> dumpStorage(string cmdStr)
{
int r;
using (SQLiteCommand cmd = new SQLiteCommand(sqliteConnection))
{
string times = "";
Stopwatch tTime = new Stopwatch();
tTime.Start();
cmd.CommandText = cmdStr;
times += "tTime1 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
tTime.Restart();
r = await cmd.ExecuteNonQueryAsync();
//cmd.ExecuteNonQuery();
//cmd.ExecuteScalar();
//cmd.ExecuteScalarAsync();
times += "tTime2 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
tTime.Restart();
transaction.Commit();
times += "tTime2 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
tTime.Restart();
transaction = sqliteConnection.BeginTransaction();
tTime.Stop();
times += "tTime3 " + tTime.Elapsed.Seconds + ":" + tTime.Elapsed.Milliseconds + "\n";
timeStr = times;
storeToDBWatchDog.Restart();
}
return r;
}
at the moment I am saving 40,000 rows with 20 (plus a primary key) columns which represents 10 seconds of data. I am unable to get the async to have any time advantage over the non-async function. in the future I will need to introduce a scheme that will manage if there was any failures to insert.
I would expect that ExecuteNonQueryAsync() should make tTime2 essentially 0 and it would work in the background however that is not what I am seeing as it appear to wait for the ...Async() function to complete.
the execute non query async and non-async require roughly 2.1 seconds. The end goal is to save the data at least 20 times faster than it is coming in. The only time that this becomes very important is on initial load. By this I mean i am reading an existing file (not in a database) and then saving it into the database. so, at the moment I only have approximately 4 times the performance. by this i mean it would take 5 minutes to save a file that consists of data associated with 20 minutes. this seems unexceptionably slow.
typical time output regardless of which function call is used (both scalar functions take ~50% longer):
Edit:
I recently came across this unfortunate post, which essentially states that async doesn't do anything so you should use the WAL setting:
https://learn.microsoft.com/en-us/dotnet/standard/data/sqlite/async
i have since changed my code to include the WAL journal-mode:
connectionString = "Data Source=" + filename + "; Version=3" + "; PRAGMA journal_mode=WAL;";
sqliteConnection = new SQLiteConnection(connectionString);
I did not see any performance increases setting the journal_mode to WAL.
Just to clear the wind I am aware off that granting users sysadmin in T-SQL in not a good thing but per customer requirements it needs to be done.
I have this :
public void AddUserToServer(List<string> users, List<string> servers)
{
foreach (var server in servers)
{
setConnection(server);
foreach (var user in users)
{
string sql =
"USE [master]" +
" CREATE LOGIN" + " [TP1\\" + user + "] FROM WINDOWS WITH DEFAULT_DATABASE=[master], " +
"DEFAULT_LANGUAGE=[us_english] " +
"EXEC sys.sp_addsrvrolemember #loginame = N'TP1\\" + user + "', #rolename = N'sysadmin'";
_dbContext.Database.ExecuteSqlCommand(sql);
}
}
}
I got the above script by creating the user manually in ssms and then generated the script. However when running it I am getting this current error:
'The procedure 'sys.sp_addsrvrolemember' cannot be executed within a transaction.
So I changed it to :
string sql =
"USE [master]" +
" CREATE LOGIN" + " [TP1\\" + user + "] FROM WINDOWS WITH DEFAULT_DATABASE=[master], " +
"DEFAULT_LANGUAGE=[us_english] " +
"GRANT ADMINISTER BULK OPERATIONS TO[TP1\\" + user + "]";
This creates the user BUT I do not get the sysadmin role.
How do I solve this?
So typical that I spent some time on this and finally after positing I find the answer.
I believe that this is a question more people will ask so I can provide what I found:
string sql =
"USE [master]" +
" CREATE LOGIN" + " [TP1\\" + user + "] FROM WINDOWS WITH DEFAULT_DATABASE=[master], " +
"DEFAULT_LANGUAGE=[us_english] " +
"EXEC sys.sp_addsrvrolemember #loginame = N'TP1\\" + user + "', #rolename = N'sysadmin'";
_dbContext.Database.ExecuteSqlCommand(TransactionalBehavior.DoNotEnsureTransaction, sql);
NOTE:
TransactionalBehavior.DoNotEnsureTransaction
Starting with EF6 Database.ExecuteSqlCommand() by default will wrap
the command in a transaction if one was not already present. There are
overloads of this method that allow you to override this behavior if
you wish. Also in EF6 execution of stored procedures included in the
model through APIs such as ObjectContext.ExecuteFunction() does the
same (except that the default behavior cannot at the moment be
overridden).
Found the answer here
I am attempting to create an ASP.NET 4.0 website (C#) that does the following: after the user clicks a button, take user inputs (from text boxes) and call a Powershell script on the server. The Powershell script output is logged to a textfile on the same machine. The script executes some stored procedures on a SQL server. Once the script is done executing, the page reads the contents of the log file and returns them in the page. The reason I log this as I do is because I need to capture the "verbose" output of these operations.
The problem I am having is that the whole set of commands that are run after the user clicks the button is queued if another user has the site open and is attempting the same operation. I am not clear as to why this happens. When run outside of the website, the Powershell scripts execute in parallel. The log files that the application writes to use a timestamp and user inputted information to ensure that they are different depending on who is inputting the info.
Here is the code that is executed when the button is clicked:
protected void Test_Click(object sender, EventArgs e)
{
// Get the current date/time to use a timestamp for the log file
DateTime curDate = DateTime.Now;
string logFileName = "D:\\test1\\logs\\" + curDate.Year + curDate.Month + curDate.Day +
curDate.Hour + curDate.Minute + curDate.Second + curDate.Millisecond + "_" +
TextBox1.Text.Replace("\\", "_") + "_Test.txt";
// Clean the Result TextBox
ResultBox.Text = string.Empty;
// Initialize PowerShell engine
var shell = PowerShell.Create();
// Add a script to the PowerShell shell object that will execute the instructions with the given input
shell.Commands.AddScript("powershell.exe -file D:\\test1\\ps_script0.ps1 '" + TextBox3.Text + "' '" +
TextBox1.Text + "' '" + TextBox2.Text + "' '" + TextBox4.Text + "' > " + logFileName);
// Execute the script
shell.Invoke();
// Create a StreamReader object to read the results of the script instructions and then store the file's contents in a string
StreamReader sr = File.OpenText(logFileName);
String results = sr.ReadToEnd();
results = results.Replace("VERBOSE: ", "");
sr.Close();
// Set the text of the result box to the string that was read in
ResultBox.Text = results;
}
I am hoping to find out if there is information I can read about allowing multiple users to click this button, and have the backend script run simultaneously for each of them. I looked into asynchronous pages and threads, but I don't think this is necessary in my case since each user's page should have its own thread. Any help here is much appreciated. Thanks!
I'm trying to write simple application in C# which will allow me to backup, zip and send over ftp my SQL Server database.
One problem I have encountered is that I'm not able to create the backup file (.bak) if I try to do it in different location than "C:\Program Files\Microsoft SQL Server\MSSQL.3\MSSQL\Backup" or "C:\Program Files\Microsoft SQL Server\MSSQL.3\MSSQL\Data" folder. I understand that this is a premission problem. Could someone point me to the resources or write here a short snippet how to programmatically add such a permission to any folder on my system.
Regards
Kris
i assume you are running your programm as a scheduled task ... did you give writing permissions to the target folder for the executing user of the task??
edit:
with permissions you can have 2 scenarios:
windows authenification
mixed authentification
if you are using windows authentification, the read and write permissions of the windows user are taken. otherwise the permissions for the sql server service account.
and this behaviour makes sense to me and maybe hits the nail in your scenario!
edit 2:
i don't want to encourage you to do so ... some admins may hate you when you mess up their acl's
but this may do the trick
btw: Magnus Johansson already gave you a "try-this" link
no matter for which method you go - be sure to hand in the correct user (as descriped above!)
(for full history)
...
side-note:
i know this is not the exact answer to your question, but i would recommend you smo to generate backups ...
like
using Microsoft.SqlServer.Management.Smo;
var bdi = new BackupDeviceItem(/* your path inlcuding desired file */);
var backup = new Backup
{
Database = /* name of the database */,
Initialize = true
};
backup.Devices.Add(bdi);
var server = new Server(this.SqlServer);
try
{
backup.SqlBackup(server);
}
catch (Exception ex)
{
// * log or sth
}
you only have to care for the .dll's. take assemblies for the desired server version (some params/properties vary through different server versions)
more info here
Ok Guys, Magnus and dittodhole! Thanks a lot for your help. I have combined Magnus'es link to the article on setting up permisions on the folder together with some more research and finally I've got it :).
So reassuming, I'm using Smo, and to create a folder with proper permissions I have to look for the group instead of win32_Users. Here you go a short snippet if someone finds this post he can find it usefull:
string tempPath = Directory.CreateDirectory("C:\\path_to_your_folder").FullName;
//set permissions
SelectQuery sQuery = new SelectQuery("Win32_Group",
"Domain='" +
System.Environment.UserDomainName.ToString() +
"'");
try
{
DirectoryInfo myDirectoryInfo = new DirectoryInfo("C:\\path_to_your_folder");
DirectorySecurity myDirectorySecurity = myDirectoryInfo.GetAccessControl();
ManagementObjectSearcher mSearcher = new ManagementObjectSearcher(sQuery);
foreach (ManagementObject mObject in mSearcher.Get())
{
string User = System.Environment.UserDomainName + "\\" + mObject["Name"];
if(User.StartsWith("your-machine-name\\SQL"))
{
myDirectorySecurity.
AddAccessRule(new FileSystemAccessRule(User,
FileSystemRights.FullControl,
AccessControlType.Allow));
}
}
myDirectoryInfo.SetAccessControl(myDirectorySecurity);
}
catch (Exception ex)
{
Console.WriteLine(ex.StackTrace);
}
Again thanks everyone for your help! Stackoverflow rocks!
Here is a procedure is use for back up in C#.Hope it helps
public void BackupDatabase (string BackUpLocation, string BackUpFileName, string
DatabaseName, string ServerName )
{
DatabaseName = "[" + DatabaseName + "]";
string fileUNQ = DateTime.Now.Day.ToString() + "_" + DateTime.Now.Month.ToString() + "_" + DateTime.Now.Year.ToString() +"_"+ DateTime.Now.Hour.ToString()+ DateTime.Now .Minute .ToString () + "_" + DateTime .Now .Second .ToString () ;
BackUpFileName = BackUpFileName + fileUNQ + ".bak";
string SQLBackUp = #"BACKUP DATABASE " + DatabaseName + " TO DISK = N'" + BackUpLocation + #"\" + BackUpFileName + #"'";
string svr = "Server=" + ServerName + ";Database=master;Integrated Security=True";
SqlConnection cnBk = new SqlConnection(svr);
SqlCommand cmdBkUp = new SqlCommand(SQLBackUp, cnBk);
try
{
cnBk.Open();
cmdBkUp.ExecuteNonQuery();
Label1.Text = "Done";
Label2.Text = SQLBackUp + " ######## Server name " + ServerName + " Database " + DatabaseName + " successfully backed up to " + BackUpLocation + #"\" + BackUpFileName + "\n Back Up Date : " + DateTime.Now.ToString();
}
catch (Exception ex)
{
Label1.Text = ex.ToString();
Label2.Text = SQLBackUp + " ######## Server name " + ServerName + " Database " + DatabaseName + " successfully backed up to " + BackUpLocation + #"\" + BackUpFileName + "\n Back Up Date : " + DateTime.Now.ToString();
}
finally
{
if (cnBk.State == ConnectionState.Open)
{
cnBk .Close();
}
}
}
Take a look at this article.
Remember to set the permissions for the account that the SQL Server instance is running with.
Although this may not answer your immediate question, I'd advice you to look into SQL Server Integration Services (SSIS). This looks like the exact thing SSIS was created for, and in the 2008 version there's the possibility to use C# code if needed, should the standard components not do what you need (earlier versions used VB.NET).
MSDN SSIS Info Link 1
SSIS 2005 Tutorial Link 2
Take a look at it.