Develop Reference

SSIS C# Script Task: (Bulk) Insert data from .csv into SQL Server database (preferably using OLEDB)

Problem Stement
I am trying to completely automate (via parametrization) my SSIS package. It uses the data flow that reads a .csv file and inserts its contents into SQL Server table. So, I need to do this without using the data flow task.
New setup
I have replaced the data flow task with a script task that does the same thing.
The .csv file is loaded into the DataTable object and then inserted into the destination table using SqlBulkCopy class and SqlConnection instance.
public void Main()
{
var atlas_source_application = (string)Dts.Variables["$Project::Atlas_SourceApplication"].Value;
var ssis_package_name = (string)Dts.Variables["System::PackageName"].Value;
var csv_path = (string)Dts.Variables["$Project::SVM_Directory"].Value;
var atlas_server_name = (string)Dts.Variables["$Project::AtlasProxy_ServerName"].Value;
var atlas_init_catalog_name = (string)Dts.Variables["$Project::AtlasProxy_InitialCatalog"].Value;
var connname = #"Data Source=" + atlas_server_name + ";Initial Catalog=" + atlas_init_catalog_name + ";Integrated Security=SSPI;";
var csv_file_path = #"" + csv_path + "\\" + ssis_package_name + ".csv";
try
{
DataTable csvData = new DataTable();
// Part I - Read
string contents = File.ReadAllText(csv_file_path, System.Text.Encoding.GetEncoding(1252));
TextFieldParser parser = new TextFieldParser(new StringReader(contents));
parser.HasFieldsEnclosedInQuotes = true;
parser.SetDelimiters(",");
string[] fields;
while (!parser.EndOfData)
{
fields = parser.ReadFields();
if (csvData.Columns.Count == 0)
{
foreach (string field in fields)
{
csvData.Columns.Add(new DataColumn(string.IsNullOrWhiteSpace(field.Trim('\"')) ? null : field.Trim('\"'), typeof(string)));
}
}
else
{
csvData.Rows.Add(fields.Select(item => string.IsNullOrWhiteSpace(item.Trim('\"')) ? null : item.Trim('\"')).ToArray());
}
}
parser.Close();
// Part II - Insert
using (SqlConnection dbConnection = new SqlConnection(connname))
{
dbConnection.Open();
using (SqlBulkCopy s = new SqlBulkCopy(dbConnection))
{
s.DestinationTableName = "[" + atlas_source_application + "].[" + ssis_package_name + "]";
foreach (var column in csvData.Columns)
{
s.ColumnMappings.Add(column.ToString(), column.ToString());
}
s.WriteToServer(csvData);
}
}
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (Exception ex)
{
Dts.Events.FireError(0, "Something went wrong ", ex.ToString(), string.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
This setup works perfectly fine on my local computer. However, once the package is deployed on the server, the insertion part breaks since the database is nowhere to be found (at least that's what it tells me).
Therefore, I tried to imitate the visual SSIS component inside the data flow task [Destination OLE DB] that uses a connection manager.
Old Setup
OLE DB connection manager setup
OLE DB destination setup
This setup uses OLE DB driver with "SQL Server Native Client 11.0" provider (or simply SQLNCLI11.1), "SSPI" integrated security, "Table or view - fast load" mode of access to data. This setup works perfectly fine locally and on the server.
Desired Setup
Armed with this knowledge I have tried to use OleDbConnection and OleDbCommand classes using this stackoverflow question, but I can't see how to use these components to bulk insert data into the DB.
I have also tried to use the visual SSIS component which is called "Bulk Insert Task", but lo luck there either.
How can I possibly insert in bulk using OLE DB?

Timeout Expired - Session - How to Fix

Hi all and thanks in advance. I am trying to fix a method that inserts information to a database table. Currently its experiencing timeouts because its running in a while loop that is taking too long to process all the contents. While I know I could just increase the command timeout I don't think that solves the problem because I think its the code. But I'm not certain what the correct fix is. I have access to Dapper and I wonder if it would be more efficient to make a method that passes the necessary variables and executes just a quick simple statement for that group then goes to get the next one? Or is that just perpetuating what's below just in a different way? Should I move this out of the code and onto the server for better performance?
UPDATE Full error message:
Exception of type 'System.Web.HttpUnhandledException' was thrown.
File: c:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\6caa4c91\19b853c6\App_Web_o3102kpb.9.cs
Method: ProcessRequest Line Number : 0
Inner Exception: {Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
File: z:\inetpub\wwwroot\SessionTransfer.aspx.cs
Method: AddSessionToDatabase Line Number : 94
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
File: z:\inetpub\wwwroot\SessionTransfer.aspx.cs Method: Page_Load Line Number : 33 }
Here is the original code:
SqlConnection con = new SqlConnection(connectionString);
SqlCommand cmd = new SqlCommand();
con.Open();
cmd.Connection = con;
int i = 0;
string strSql, guid = GetGuid();
string temp = "";
while (i < Session.Contents.Count)
{
if (Session.Contents[i] == null)
temp = "";
else {
if ((Session.Contents[i].ToString().Trim().Length) > 0)
temp = Session.Contents[i].ToString().Replace("'", "''");
else
temp = "";
}
strSql = "INSERT INTO SessionTable (GUID, SessionKey, SessionValue) " +
"VALUES ('" + guid + "', '" + Session.Contents.Keys[i].ToString() + "', '" + temp + "')";
cmd.CommandText = strSql;
cmd.ExecuteNonQuery();
i++;
}
con.Close();
cmd.Dispose();
con.Dispose();
return guid;
UPDATE - FINAL SOLUTION:
var SessionList = new List<Session>();
while (i < Session.Contents.Count)
{
string temp = "";
if (Session.Contents[i] == null)
temp = "";
else
{
temp = (Session.Contents[i].ToString().Trim().Length) > 0 ? Session.Contents[i].ToString().Replace("'", "''") : "";
}
var s = new Session
{
TempGuid = guidTemp,
Contents = Session.Contents[i] != null ? Session.Contents[i].ToString() : null,
Temp = temp
};
SessionList.Add(s);
i++;
}
mySession = SerializationUtilities.SerializeObjectToXML(SessionList);
using (var con = new SqlConnection())
{
con.ExecuteHGW("Transfer", new { mySession }, commandType: CommandType.StoredProcedure);
}
Then on the SQL side I just put the XML in a table and did one single insert statement against the table, time is significantly improved.

I would like to suggest you 2 improvements:
in the loop you are executing insert statements one by one against db. It takes much time to open connection to db then send the query, execute it and return result. It is much better to batch them. So gather like 10,000 of such insert statements, build the whole instuction with StringBuilder and execute it against db in one go. This will really increase the speed of your app. Amount of Insert instructions in one batch you should choose yourself basing on system tests.
If after applying 1) hint the problem with timeout still occurs I would suggest 2 possible solutions:
a). do not send all elements to be processed against db to web service at once but instead, as previoulsy, batch them (apply second batching). So for instance send to web service 50,000 elements to be insterted, then wait for confirmation from web service and then proceed with next batch. The big advantage is that you can show to user easily progress bar showing him current operation state.
b). send all items to be processed against db at once but do not wait for result. In your app just show that items are processed and each 10 s send to web service request to ask if the job is finished. When it is finished signal it to user.

Getting timeout errors with SqlTransaction on same table

public TransImport()
{
ConnString = ConfigurationManager.ConnectionStrings["Connection"].ConnectionString;
SqlConnection conn_new;
SqlCommand command_serial_new;
SqlConnection conn;
SqlCommand command_serial;
SqlTransaction InsertUpdateSerialNumbers;
conn = new SqlConnection(ConnString);
command_serial = conn.CreateCommand();
conn_new = new SqlConnection(ConnString);
command_serial_new = conn_new.CreateCommand();
command_serial_new.CommandText = "SELECT 1 FROM YSL00 WHERE SERLNMBR = #slnr";
var p = new SqlParameter("#slnr", SqlDbType.NVarChar, 50);
command_serial_new.Parameters.Add(p);
//Here you will start reading flat file to get serialnumber.
InsertUpdateSerialNumbers = conn.BeginTransaction();
while (!headerFileReader.EndOfStream)
{
headerRow = headerFileReader.ReadLine();
if (CheckSerialNumber(headerFields[0].Trim()))
DisplayMessage("Good serialnumber"); //this function is not copied here.
}
InsertUpdateSerialNumbers.Commit();
}
private Boolean CheckSerialNumber(string SerialNumber)
{
command_serial_new.Parameters["#slnr"].Value = SerialNumber;
try
{
var itExists = Convert.ToInt32(command_serial_new.ExecuteScalar()) > 0;
if (!itExists)
{
command_serial.Transaction = InsertUpdateSerialNumbers;
command_serial.CommandText = "INSERT INTO YSL00([Manifest_Number],[PONUMBER],[ITEMNMBR],[SERLNMBR]"
+ "VALUES ('" + Manifest + "','" + PONr + "','" + itemNumber + "','" + serialNr + "')";
var insertStatus = command_serial.ExecuteNonQuery();
return true;
}
}
catch (Exception ex)
{
LogException(ex, "Error in CheckSerialNumber =>"+ command_serial_new.CommandText.ToString());
}
return false;
}
I get error "Timeout expired. The timeout period elapsed prior to completion of the operation or server is not responding".
The CheckSerialNumber function also does an insert to YSL00 (the same table where I had executescalar. See code above).
As I mentioned earlier there are 1000s of line in a flat file that I read and update YSL000 table.
Note that I have two separate sqlcommands and also two separate connections to handle this. Reason is with sqltransaction it doesn't let me to query on the same table. I think timeout may be happening because of this?
Thanks for reading. Please suggest
Update 1: Since I have not pasted entire code, I want to mention that dispose is done using below code in the program.
if (conn != null)
{
conn.Close();
conn.Dispose();
}
if (conn_new != null)
{
conn_new.Close();
conn_new.Dispose();
}

you can increase the time out of your SqlConnection object.
you can do this with your ConnString:
string connStr = "Data Source=(local);Initial Catalog=AdventureWorks;Integrated
Security=SSPI;Connection Timeout=300";

I think default isolation level - read commited - is preventing your 'CheckSerialNumber' method from being effective. Command_serial_new will not take into consideration rows inserted in your loop - this might lead to some troubles. To be honest I would also look for some deadlock. Perhaps command_serial_new is actually completely blocked by the other transaction.
To start off:
Set command_serial_new query as:
SELECT 1 FROM YSL00 WITH (NOLOCK) WHERE SERLNMBR = #slnr
Think about using lower isolation level to query inserted rows as well (set it to read uncommited).
Close your connections and transactions.
Use just one SqlConnection - you don't need two of them.

Many of the objects you are using implement IDisposable, and you should be wrapping them with using statements. Without these using statements, .NET won't necessarily get rid of your objects until an undetermined time when the garbage collector runs, and could block subsequent queries if it's still holding a transaction open somewhere.
So for example, you'll need to wrap your connections with using statements:
using (conn_new = new SqlConnection(ConnString)) {
...

If I am not mistaken you need to merge the file content with the table content.
For this purpose I would recommend you
Copy the file content in to a temporary table (see temporary tables and BulkInsert)
Use command MERGE (http://msdn.microsoft.com/en-us/library/bb510625.aspx) to merge the temporary table content with the original table

Unable to restore SQL database, exclusive access could not be obtained (single user mode)

I am writing a simple database backup and restore routine for an application. I can backup my database without issues, however when I restore is I am unable to gain exclusive access to my database.
I am trying all the combinations of fixes on SO, putting in single user mode, taking it offline then placing it back only with no success.
I can successfully restore the database within studio manager (express)
This method is the only connection to the SQL server at the time, so I don't understand why I can't perform the restore.
Appreciate the help to point out where the issue may be.
internal void RestoreDatabase(string databaseFile)
{
//get database details
var databaseConfiguration = new DatabaseConfiguration().GetDatabaseConfiguration();
try
{
//construct server connection string
var connection = databaseConfiguration.IsSqlAuthentication
? new ServerConnection(databaseConfiguration.ServerInstance,
databaseConfiguration.SqlUsername,
databaseConfiguration.SqlPassword)
: new ServerConnection(databaseConfiguration.ServerInstance);
//set database to single user and kick everyone off
using (
var sqlconnection =
new SqlConnection(new DatabaseConfiguration().MakeConnectionString(databaseConfiguration)))
{
sqlconnection.Open();
using (
var sqlcommand = new SqlCommand("ALTER DATABASE " + databaseConfiguration.DatabaseName + " SET Single_User WITH Rollback IMMEDIATE",
sqlconnection))
{
sqlcommand.ExecuteNonQuery();
}
using (
var sqlcommand = new SqlCommand("ALTER DATABASE " + databaseConfiguration.DatabaseName + " SET OFFLINE",
sqlconnection))
{
sqlcommand.ExecuteNonQuery();
}
using (
var sqlcommand = new SqlCommand("ALTER DATABASE " + databaseConfiguration.DatabaseName + " SET ONLINE",
sqlconnection))
{
sqlcommand.ExecuteNonQuery();
}
sqlconnection.Close();
}
//setup server connection and restore
var server = new Server(connection);
var restore = new Restore();
restore.Database = databaseConfiguration.DatabaseName;
restore.Action = RestoreActionType.Database;
restore.Devices.AddDevice(databaseFile, DeviceType.File);
restore.ReplaceDatabase = true;
restore.Complete += Restore_Complete;
restore.SqlRestore(server);
}
catch (Exception ex)
{
//my bad
restoreDatabaseServerError(ex.InnerException.Message, EventArgs.Empty);
}
finally
{
//set database to multi user
using (
var sqlconnection =
new SqlConnection(new DatabaseConfiguration().MakeConnectionString(databaseConfiguration)))
{
sqlconnection.Open();
using (
var sqlcommand = new SqlCommand("ALTER DATABASE " + databaseConfiguration.DatabaseName + " SET Multi_User",
sqlconnection))
{
sqlcommand.ExecuteNonQuery();
sqlcommand.Dispose();
}
sqlconnection.Close();
}
}
}

If anybody is connected to your database, SQL Server cannot drop it, so you have to disconnect existing connections, as you have tried. The problem with single_user is, that it still allows a single user to connect. As you yourself cannot be connected to the database when dropping it you have to get out of there. That opens up that slot for someone else to connect and in turn prevent you from dropping it.
There are a few SQL Server processes that are particularly good at connecting to a database in that split second. Replication is one example. (You shouldn't really drop a database that is published anyway, bat that is another story.)
So what can we do about this? The only 100% safe way is to prevent users from connecting to the database. The only practical way is to switch the database offline and then drop it. However, that has the nasty side effect, that SQL Server does not delete the files of that database, so you have to do that manually.
Another option is to just be fast enough. In your example you bring the database back online before you drop it. That is a fairly resource intensive process that gives an "intruder" lots of time to connect.
The solution I have been using with success looks like this:
ALTER DATABASE MyDb SET RESTRICTED_USER WITH ROLLBACK IMMEDIATE;
USE MyDb;
ALTER DATABASE MyDb SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
USE tempdb;
DROP DATABASE MyDb;
This first sets the database to restricted user and connects to it. Then, while still connected it sets the database to single user. Afterwards the context is switched to tempdb and the drop is executed immediately thereafter. Important here is, to send these commands as one batch to SQL Server to minimize the time between the USE tempdb; and the DROP. Setting the database to restricted user in the beginning catches some rare edge cases, so leave it in even though it does not make sense at first glance.
While this still leaves a theoretical gap for someone else to get in, I have never seen it fail.
After the database is dropped you can run your restore as normal.
Good luck.

Your restore needs to take place on the same connection you set the DB server to single user mode.
In summary for the changes below, I moved the end of the using to below your restore code, and moved the close for the SQL connection to after the restore so it uses the same connection. Also removed set offline and online since they aren't needed. Can't test at the moment, so let me know if it works.
//set database to single user and kick everyone off
using (var sqlconnection = new SqlConnection(new DatabaseConfiguration().MakeConnectionString(databaseConfiguration)))
{
sqlconnection.Open();
using (var sqlcommand = new SqlCommand("ALTER DATABASE " + databaseConfiguration.DatabaseName + " SET Single_User WITH Rollback IMMEDIATE",sqlconnection))
{
sqlcommand.ExecuteNonQuery();
}
//setup server connection and restore
var server = new Server(sqlconnection);
var restore = new Restore();
restore.Database = databaseConfiguration.DatabaseName;
restore.Action = RestoreActionType.Database;
restore.Devices.AddDevice(databaseFile, DeviceType.File);
restore.ReplaceDatabase = true;
restore.Complete += Restore_Complete;
restore.SqlRestore(server);
sqlconnection.Close();
}

Creating a database programmatically in SQL Server

How can I create a database programmatically and what is the minimum information I need to do this?
Please no "SQL Server Management Object API " suggestions.

You can either use the SQL Server Management Object API (see task "creating, altering and removing databases"):
var srv = new Server();
var db = new Database(srv, "mydb");
db.Create();
Information on how to get started is here. During SQL server installation you need to install the client SDK, the SMO assemblies are in C:\Program Files\Microsoft SQL Server\100\SDK\Assemblies
Or if you don't want the dependency on these assemblies, you can also simply run DDL statements using ADO.Net (e.g. see this question):
using (var connection = new SqlConnection(myConnectionString))
{
connection.Open();
var command = connection.CreateCommand();
command.CommandText = "CREATE DATABASE mydb";
command.ExecuteNonQuery();
}
Obviously you need a correct connection string: known sql server instance and a user with CREATE DATABASE permission.

Create database 'Databasename'

From the creators:
// your connection string
string connectionString = "Server=(local)\\netsdk;uid=sa;pwd=;database=master";
// your query:
var query = GetDbCreationQuery();
var conn = new SqlConnection(connectionString);
var command = new SqlCommand(query, conn);
try
{
conn.Open();
command.ExecuteNonQuery();
MessageBox.Show("Database is created successfully", "MyProgram",
MessageBoxButtons.OK, MessageBoxIcon.Information);
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
finally
{
if ((conn.State == ConnectionState.Open))
{
conn.Close();
}
}
To create in default location with default settings, just:
static string GetDbCreationQuery()
{
// your db name
string dbName = "MyDatabase";
// db creation query
string query = "CREATE DATABASE " + dbName + ";";
return query;
}
Or, to create it in a specific location:
static string GetDbCreationQuery()
{
// your db name
string dbName = "MyDatabase";
// path to your db files:
// ensure that the directory exists and you have read write permission.
string[] files = { Path.Combine(Application.StartupPath, dbName + ".mdf"),
Path.Combine(Application.StartupPath, dbName + ".ldf") };
// db creation query:
// note that the data file and log file have different logical names
string query = "CREATE DATABASE " + dbName +
" ON PRIMARY" +
" (NAME = " + dbName + "_data," +
" FILENAME = '" + files[0] + "'," +
" SIZE = 3MB," +
" MAXSIZE = 10MB," +
" FILEGROWTH = 10%)" +
" LOG ON" +
" (NAME = " + dbName + "_log," +
" FILENAME = '" + files[1] + "'," +
" SIZE = 1MB," +
" MAXSIZE = 5MB," +
" FILEGROWTH = 10%)" +
";";
return query;
}
Even in case the execution fails, give it another try. The db files might have got created.

You need to open a connection to the server, i.e. you need a server and instance name.
You also need the proper access rights to create a database, so you might need some user name and password depending on the authentication settings on the server.
From the server name and authentication information you can construct a connection string and open a connection.
Then you can use the CREATE DATABASE SQL command (see here on MSDN). The only needed parameter for this command is a database name.

You need connection information: server, possibly instance, a user having create database rights on that server/instance and the corresponding password. Then you can use SMO for creating the database. Here is a small PowerShell example that you can very easily "translate" to C#, for example:
[System.Reflection.Assembly]::LoadWithPartialName('Microsoft.SqlServer.SMO') | Out-Null
$s = New-Object Microsoft.SqlServer.Management.Smo.Server($ServerInstance)
# Instantiate the database object and add the filegroups
$db = New-Object Microsoft.SqlServer.Management.Smo.Database($s, $DatabaseName)
$primaryFG = New-Object Microsoft.SqlServer.Management.Smo.FileGroup($db, 'PRIMARY')
$db.FileGroups.Add($primaryFG)
# Create Data file
$syslogname = $DatabaseName + '_SysData'
$dbdsysfile = New-Object Microsoft.SqlServer.Management.Smo.DataFile($primaryFG, $syslogname)
$primaryFG.Files.Add($dbdsysfile)
$dbdsysfile.FileName = $s.MasterDBPath + '\' + $syslogname + '.mdf'
$dbdsysfile.Size = [double](5.0 * 1024.0)
$dbdsysfile.GrowthType = 'KB'
$dbdsysfile.Growth = 10000.0
$dbdsysfile.IsPrimaryFile = 'True'
# Create Log file
$loglogname = $DatabaseName + '_Log'
$dblfile = New-Object Microsoft.SqlServer.Management.Smo.LogFile($db, $loglogname)
$db.LogFiles.Add($dblfile)
$dblfile.FileName = $s.MasterDBLogPath + '\' + $loglogname + '.ldf'
$dblfile.Size = [double](10.0 * 1024.0)
$dblfile.GrowthType = 'KB'
$dblfile.Growth = 10000.0
# Create database with READ_COMMITTED_SNAPSHOT isolation level.
# Other options can be set on $db object before calling Create.
$db.IsReadCommittedSnapshotOn = $true
$db.RecoveryModel = [Microsoft.SqlServer.Management.Smo.RecoveryModel]::Simple
$db.Create()

Assuming you have the rights to fire off a CREATE DATABASE statement you can do so as you would any other query.
I should stress that being able to do so requires quite high privileges on the server and this would be restricted to DBAs in QA and Production environments.
For that reason I would make sure that your connection uses Windows Integrated Security. That way when the appropriate DBA runs your application the app will function as requested.
Once you have created your database you will also need to fire off the T-SQL to create logins and create users. I'm taking it as obvious that CREATE TABLE/VIEW statements will be needed.