I have many files for procedures, views, functions, etc.
I want to execute these files (creating components) in appropriate database on SQL Server 2005/2008.
Also the point is I want to execute them using C#.
Another point to mention, I want the application to be such that I can execute this files on a remote SQL Server too. Also client machine may not have osql,sqlcmd command tool.
Can someone please guide me on this.
This depends on what sort of files they are. If, for example, they only contain actual T-SQL commands (and aren't batch files that you'd run in, say, SSMS, which would contain a batch separator like GO), then you just need to create a connection, a command, then read the contents of the file and use that to populate the CommandText property of the command.
For example:
void ExecuteFile(string connectionString, string fileName)
{
using(SqlConnection conn = new SqlConnection(connectionString))
{
string data = System.IO.File.ReadAllText(fileName);
conn.Open();
using(SqlCommand cmd = conn.CreateCommand())
{
cmd.CommandText = data;
cmd.ExecuteNonQuery();
}
}
}
If it's a batch file, you'll need to split the file into individual batches and process those individually. The simplest method is just to use string.Split, but bear in mind that it won't respect SQL parsing rules when it splits (for example, if GO appears within a SQL statement, it's going to split the command up into two batches, which will obviously fail).
More generally, you can see what you'd need to do here by modifying the code in this way:
string[] batches = SplitBatches(System.IO.File.ReadAllText(fileName));
conn.Open();
using(SqlCommand cmd = conn.CreateCommand())
{
foreach(string batch in batches)
{
cmd.CommandText = batch;
cmd.ExecuteNonQuery();
}
}
The implementation of a function called SplitBatches is up to you.
Typically, the simplest way is to split the script on the "GO" statement and execute each item separately. This solution will not work if the script contains a GO statement within a comment.
private readonly Regex _sqlScriptSplitRegEx = new Regex( #"^\s*GO\s*$", RegexOptions.IgnoreCase | RegexOptions.Multiline | RegexOptions.Compiled );
public void ExecuteSqlScript( string scriptText )
{
if ( string.IsNullOrEmpty( scriptText ) )
return;
var scripts = _sqlScriptSplitRegEx.Split( scriptText );
using ( var conn = new SqlConnection( "connection string" ) )
{
using ( var ts = new TransactionScope( TransactionScopeOption.Required, new TimeSpan( 0, 10, 0 ) ) )
{
foreach ( var scriptLet in scripts )
{
if ( scriptLet.Trim().Length == 0 )
continue;
using ( var cmd = new SqlCommand( scriptLet, conn ) )
{
cmd.CommandTimeout = this.CommandTimeout;
cmd.CommandType = CommandType.Text;
cmd.ExecuteNonQuery();
}
}
ts.Complete();
}
}
}
Normally all you have to do is just execute them with SqlCommand.ExecuteNonQuery, one batch at a time. That leaves you the task of splitting the scripts into batches, using the currently set batch delimiter (usually GO). The free library dbutilsqlcmd can be used to handle SQL scripts, as it processes not only the delimiter, but SQLCMD extensions as well (:setvar, :connect etc).
Using standard ADO would "work" -- SQL DDL can be sent using SqlCommand.ExecuteNonQuery, for instance.
...but I'd recommend using a Database Project (with a remote staging deploy or VSDBCMD) or one of the tools such as SQLCMD (perhaps in conjunction with a PowerShell script :-) or SQL Management Studio, etc. They are designed for this sort of stuff.
You could read in the commands using a TextReader, then use ExecuteNonQuery with the TextReader results as the command.
Related
I am working on a big solution in C#. This solution uses different oracle databases and is checked in in TFS. Actually I develop a small library which can deploy the SQL Scripts in the solution to the used databases. The deployment of the application is handled with TFS and in the future my tool should be included in the build process. For the Oracle Connection an execution of the statements I use Oracle.ManagedDataAccess.Client.
If there are only several DDL or DML statements in the SQL Files the first tests were successful. But there are a few special special things I have to handle, like SET DEFINE OFF. This is no real SQL Statement and if I run the Script with ORacle SQL*PLus there is no problem. In the beginning I tried to execute the whole script but this doesn't work because multiple statements were put in one line by my application and the only solution I found was the splitting of the file and single execution.
Example:
/*************************************************************************************************************************
##NAME ScriptName.sql
##DESCR ScriptDescription
*************************************************************************************************************************/
Create table tmp_thm_Test
( tmp_thm_Test_id number(8)
, text varchar2(200));
Create table tmp_thm_Test2
( tmp_thm_Test2_id number(8)
, text varchar2(200));
This is an example content for a script. Also there can be Insert (Set Define off is needed while having the & in strings), Update Statements. Also Begin/End Scripts.
If there are only DDL and DML Statements I can split them but Begin/End parts are not splittable. At the beginning I thought I can deploy a script like in the past with SQL*Plus (#Scriptname.sql). Read the whole script an execute it.
Any ideas?
Since I've worked with Oracle in C# before I kind of understand you even with the lack of concrete samples.
SET DEFINE OFF never worked for me in C#, so I simply use query = query.Replace("'","''");. You could also use command parameters like the one below to avoid exceptions.
string query = string.Format(
#"UPDATE CardStatus
SET DateExpired = #dateExpired,
ModifiedBy = #modifiedBy,
ModifiedOn = #modifiedOn
WHERE
IDNo = #idNo");
List<OracleParameter> parameters = new List<OracleParameter>();
OracleParameter pIdNo = new OracleParameter("idNo", OracleDbType.Varchar2);
pIdNo.Value = idNo;
OracleParameter pDateExpired = new OracleParameter("dateExpired", OracleDbType.Date);
pDateExpired.Value = dateExpired;
OracleParameter pModifiedBy = new OracleParameter("modifiedBy", OracleDbType.Varchar2);
pModifiedBy.Value = "SIS";
OracleParameter pModifiedOn = new OracleParameter("modifiedOn", OracleDbType.Date);
pModifiedOn.Value = DateTime.Now;
parameters.Add(pIdNo);
parameters.Add(pDateExpired);
parameters.Add(pModifiedBy);
parameters.Add(pModifiedOn);
bool result = _DAL.ExecuteNonQuery(query, parameters.ToArray());
_DAL is simply my helper class for data access, it manages the query executions placing it inside a single transaction so that I get to decide whether to commit or rollback the changes of single to multiple queries.
public bool ExecuteNonQuery(string query, object[] parameters, [CallerMemberName] string callerMemberName = "")
{
bool success = false;
try
{
using (OracleCommand command = new OracleCommand())
{
command.Connection = _connection;
command.Transaction = _transaction;
command.CommandText = query;
command.CommandType = CommandType.Text;
command.Parameters.AddRange(parameters);
command.ExecuteNonQuery();
}
this.AuditSQL(query, string.Empty);
success = true;
}
catch (OracleException ex)
{
this.AuditSQL(query, ex.Message, callerMemberName);
if (ex.Number == 54) // SELECT .. FOR UPDATE NOWAIT failed.
{
throw new RowLockException();
}
}
return success;
}
I am not sure if you are interested in looking at my _DAL class but I provided you a snippet to give you an idea of what you can do.
EDIT: Since you've mentioned that the scripts already exists then it is easier, you just have to make sure that there is no ' character within the script to make oracle ask for a variable value and all batch queries must have BEGIN and END;.
Example:
BEGIN
-- INSERT
-- UPDATE
-- INSERT
... etc.
END;
Now, what exactly is your problem? You can read the query from the file and replace all SET DEFINE OFF (as well as other statements you deem unnecessary) with string.Empty and replace all ' with '' if needed. Worst case, you would be using regex to identify and remove unwanted statements.
Can you do an insert statement with multiple file blob reads in the same command?
In the code below inputfile contains the following:
string[] inputfile = {"C:\\test_blob\\blob1.pdf","C:\\test_blob\\blob2.jpg"};
I'm uncertain if cmd.Parameters can be done prior to the cmd.CommandText or if I can do more than one File.ReadAllBytes() as a cmd.Parameter.
public static void insert_blob_file(string dbname, string uid, string pwd, string[] inputfile)
{
using (var conn = new OdbcConnection("DSN=" + dbname + ";UID=" + uid + ";pwd=" + pwd))
{
conn.Open();
using (var cmd = conn.CreateCommand())
{
for (int i = 0; i < inputfile.Count();i++)
{
var inputStream = new FileStream[i];
using (inputStream[i] = File.OpenRead(inputfile[i]))
{
cmd.Parameters.AddWithValue("blob" + i.ToString(), File.ReadAllBytes(inputfile[i]));
}
cmd.CommandText = "INSERT INTO MyTable (Id, MyBlobColumn,String1,MyBlobColum1,String2,String3) VALUES (1,#blob0,SomeString,#blob1,SomeString,SomeString)";
}
cmd.ExecuteNonQuery();
}
conn.Close();
}
}
First, wich of the 2/3 Strategies for storing blobs do you even use? Here is a nice listing of the 2 common and SQL Servers special approach to try to combine them: https://www.simple-talk.com/sql/learn-sql-server/an-introduction-to-sql-server-filestream/
Secondly building queries via string conaction is just going to expose you to SQL Injections. You really should be using the SQL Parameter Syntax instead. Aside from being safer, they might even be faster as the SQL servers does not need to imply types. You can explicitly tell it the types and proper mapping.
Thirdly, I asume you are calling a function like insert_blob_file in some form of multitasking. SQL Operations are network operations and those can take really long times, one way or the other.
As for the actuall problem: When inserting or updating large amounts of data, batching is rather important. You want to do enough at once to avoid overhead. But not so much, you end up locking up the table and thus possibly the whole DB for a very long time. Especially if the network connection to the client is not the fastest. I always advice to do bulk inserts on the DBMS side just to avoid this, but it seems unlikely you can do that here.
With blobs every insert should be a seperate job. Do not even try to do bulk blob inserts.
I'm building a .NET application that talks to an Oracle 11g database. I am trying to take data from Excel files provided by a third party and upsert (UPDATE record if exists, INSERT if not), but am having some trouble with performance.
These Excel files are to replace tariff codes and descriptions, so there are a couple thousand records in each file.
| Tariff | Description |
|----------------------------------------|
| 1234567890 | 'Sample description here' |
I did some research on bulk inserting, and even wrote a function that opens a transaction in the application, executes a bunch of UPDATE or INSERT statements, then commits. Unfortunately, that takes a long time and prolongs the session between the application and the database.
public void UpsertMultipleRecords(string[] updates, string[] inserts) {
OleDbConnection conn = new OleDbConnection("connection string here");
conn.Open();
OleDbTransaction trans = conn.BeginTransaction();
try {
for (int i = 0; i < updates.Length; i++) {
OleDbCommand cmd = new OleDbCommand(updates[i], conn);
cmd.Transaction = trans;
int count = cmd.ExecuteNonQuery();
if (count < 1) {
cmd = new OleDbCommand(inserts[i], conn);
cmd.Transaction = trans;
}
}
trans.Commit();
} catch (OleDbException ex) {
trans.Rollback();
} finally {
conn.Close();
}
}
I found via Ask Tom that an efficient way of doing something like this is using an Oracle MERGE statement, implemented in 9i. From what I understand, this is only possible using two existing tables in Oracle. I've tried but don't understand temporary tables or if that's possible. If I create a new table that just holds my data when I MERGE, I still need a solid way of bulk inserting.
The way I usually upload my files to merge, is by first inserting into a load table with sql*loader and then executing a merge statement from the load table into the target table.
A temporary table will only retain it's contents for the duration of the session. I expect sql*loader to end the session upon completion, so better use a normal table that you truncate after the merge.
merge into target_table t
using load_table l on (t.key = l.key) -- brackets are mandatory
when matched then update
set t.col = l.col
, t.col2 = l.col2
, t.col3 = l.col3
when not matched then insert
(t.key, t.col, t.col2, t.col3)
values
(l.key, l.col, l.col2, l.col3)
Well i have a file.sql that contains 20,000 of insert commands
Sample From the .sql file
INSERT INTO table VALUES
(1,-400,400,3,154850,'Text',590628,'TEXT',1610,'TEXT',79);
INSERT INTO table VALUES
(39,-362,400,3,111659,'Text',74896,'TEXT',0,'TEXT',14);
And i am using the following code to create an in memory Sqlite database and pull the values into it then calculate the time elapsed
using (var conn = new SQLiteConnection(#"Data Source=:memory:"))
{
conn.Open();
var stopwatch = new Stopwatch();
stopwatch.Start();
using (var cmd = new SQLiteCommand(conn))
{
using (var transaction = conn.BeginTransaction())
{
cmd.CommandText = File.ReadAllText(#"file.sql");
cmd.ExecuteNonQuery();
transaction.Commit();
}
}
var timeelapsed = stopwatch.Elapsed.TotalSeconds <= 60
? stopwatch.Elapsed.TotalSeconds + " seconds"
: Math.Round(stopwatch.Elapsed.TotalSeconds/60) + " minutes";
MessageBox.Show(string.Format("Time elapsed {0}", timeelapsed));
conn.Close();
}
Things i have tried
Using file database instead of memory one.
Using begin transaction and commit transaction [AS SHOWN IN MY CODE].
Using Firefox's extension named SQLite Manager to test whether the
slowing down problem is from the script; However, I was surprised
that the same 20,000 lines that i am trying to process using my code
has been pulled to the database in JUST 4ms!!!.
Using PRAGMA synchronous = OFF, as well as, PRAGMA journal_mode =
MEMORY.
Appending begin transaction; and commit transaction; to the
beginning and ending of the .sql file respectively.
As the SQLite documentations says : SQLite is capable of processing 50,000 commands per seconds. And that is real and i made sure of it using the SQLite Manager [AS DESCRIPED IN THE THIRD SOMETHING THAT I'V TRIED]; However, I am getting my 20,000 commands done in 4 minutes something that tells that there is something wrong.
QUESTION : What is the problem am i facing why is the Execution done very slowly ?!
SQLite.Net documentation recommends the following construct for transactions
using (SqliteConnection conn = new SqliteConnection(#"Data Source=:memory:"))
{
conn.Open();
using(SqliteTransaction trans = conn.BeginTransaction())
{
using (SqliteCommand cmd = new SQLiteCommand(conn))
{
cmd.CommandText = File.ReadAllText(#"file.sql");
cmd.ExecuteNonQuery();
}
trans.Commit();
}
con.Close();
}
Are you able to manipulate the text file contexts to something like:
INSERT INTO table (col01, col02, col03, col04, col05, col06, col07, col08, col09, col10, col11)
SELECT 1,-400,400,3,154850,'Text',590628,'TEXT',1610,'TEXT',79
UNION ALL
SELECT 39,-362,400,3,111659,'Text',74896,'TEXT',0,'TEXT',14
;
Maybe try "batching them" into groups of 100 as a initial test.
http://sqlite.org/lang_select.html
SqlLite seems to support the UNION ALL statement.
I have C# code that cycles through .sql files and executes what's inside them to set up a database.
One .sql file is basically as follows:
DROP PROCEDURE IF EXISTS myProc;
DELIMITER $$
CREATE PROCEDURE myProc()
BEGIN
-- procedure stuff goes here
END $$
DELIMITER ;
CALL myProc();
When I input this into the MySQL Query Browser's script window, it runs perfectly... over and over again, just as one would want it to.
However, if I put the string into my IDbCommand and execute it...
connection.Open(); // An IDbConnection
IDbTransaction transaction = connection.BeginTransaction();
using (IDbCommand cmd = connection.CreateCommand())
{
cmd.Connection = connection;
cmd.Transaction = transaction;
cmd.CommandText = line;
cmd.CommandType = CommandType.Text;
try
{
cmd.ExecuteNonQuery();
}
catch (SqlException ex)
{
transaction.Rollback();
return false;
}
}
transaction.Commit();
connection.Close();
... I get the dreaded exception 1064...
You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near 'DELIMITER $$ CREATE PROCEDURE myProc() BEGIN...
So, the question is... why does MySQL let me do this with no problems, but when I try to run it from C#, it fails? And of course the second question is how I'm supposed to fix it.
For those looking for a quick snippet...
var connectionString = #"server=ChangeMe;user=ChangeMe;pwd=ChangeMe;database=ChangeMe;";
var scriptText = File.ReadAllText(#"C:\script.sql");
using (var connection = new MySqlConnection(connectionString))
{
var script = new MySqlScript(connection, scriptText);
connection.Open();
script.Execute();
}
I think what you are looking for is this: "Bug #46429: use DELIMITER command in MySql.Data.MySqlClient.MySqlScript"
Some of the commands in your script are interpreted bt the mysqlk client excutable before the SQL is sent to the server.
See mysql commands
In effect all the delimiters
To interpret the script from C# you will have to write code that knows what the command is, you cannot just pass the commands line by line to the server.
e.g. you have 3 commands in that file and so want to call ExecuteNonQuery only 3 times
and from paracycle's coment you need to use the MySqlScript class