Develop Reference

SSIS C# Script Task: (Bulk) Insert data from .csv into SQL Server database (preferably using OLEDB)

Problem Stement
I am trying to completely automate (via parametrization) my SSIS package. It uses the data flow that reads a .csv file and inserts its contents into SQL Server table. So, I need to do this without using the data flow task.
New setup
I have replaced the data flow task with a script task that does the same thing.
The .csv file is loaded into the DataTable object and then inserted into the destination table using SqlBulkCopy class and SqlConnection instance.
public void Main()
{
var atlas_source_application = (string)Dts.Variables["$Project::Atlas_SourceApplication"].Value;
var ssis_package_name = (string)Dts.Variables["System::PackageName"].Value;
var csv_path = (string)Dts.Variables["$Project::SVM_Directory"].Value;
var atlas_server_name = (string)Dts.Variables["$Project::AtlasProxy_ServerName"].Value;
var atlas_init_catalog_name = (string)Dts.Variables["$Project::AtlasProxy_InitialCatalog"].Value;
var connname = #"Data Source=" + atlas_server_name + ";Initial Catalog=" + atlas_init_catalog_name + ";Integrated Security=SSPI;";
var csv_file_path = #"" + csv_path + "\\" + ssis_package_name + ".csv";
try
{
DataTable csvData = new DataTable();
// Part I - Read
string contents = File.ReadAllText(csv_file_path, System.Text.Encoding.GetEncoding(1252));
TextFieldParser parser = new TextFieldParser(new StringReader(contents));
parser.HasFieldsEnclosedInQuotes = true;
parser.SetDelimiters(",");
string[] fields;
while (!parser.EndOfData)
{
fields = parser.ReadFields();
if (csvData.Columns.Count == 0)
{
foreach (string field in fields)
{
csvData.Columns.Add(new DataColumn(string.IsNullOrWhiteSpace(field.Trim('\"')) ? null : field.Trim('\"'), typeof(string)));
}
}
else
{
csvData.Rows.Add(fields.Select(item => string.IsNullOrWhiteSpace(item.Trim('\"')) ? null : item.Trim('\"')).ToArray());
}
}
parser.Close();
// Part II - Insert
using (SqlConnection dbConnection = new SqlConnection(connname))
{
dbConnection.Open();
using (SqlBulkCopy s = new SqlBulkCopy(dbConnection))
{
s.DestinationTableName = "[" + atlas_source_application + "].[" + ssis_package_name + "]";
foreach (var column in csvData.Columns)
{
s.ColumnMappings.Add(column.ToString(), column.ToString());
}
s.WriteToServer(csvData);
}
}
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (Exception ex)
{
Dts.Events.FireError(0, "Something went wrong ", ex.ToString(), string.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
This setup works perfectly fine on my local computer. However, once the package is deployed on the server, the insertion part breaks since the database is nowhere to be found (at least that's what it tells me).
Therefore, I tried to imitate the visual SSIS component inside the data flow task [Destination OLE DB] that uses a connection manager.
Old Setup
OLE DB connection manager setup
OLE DB destination setup
This setup uses OLE DB driver with "SQL Server Native Client 11.0" provider (or simply SQLNCLI11.1), "SSPI" integrated security, "Table or view - fast load" mode of access to data. This setup works perfectly fine locally and on the server.
Desired Setup
Armed with this knowledge I have tried to use OleDbConnection and OleDbCommand classes using this stackoverflow question, but I can't see how to use these components to bulk insert data into the DB.
I have also tried to use the visual SSIS component which is called "Bulk Insert Task", but lo luck there either.
How can I possibly insert in bulk using OLE DB?

Using SMO Restore Class when there are multiple backup points in .bak file

I am attempting to use SMO to write a simple utility to backup/restore databases. This works very well when there is only one point in time in the backup file. However when there is a backup file that has multiple backup points defined (not backup sets) SMO always chooses the earliest, whereas SSMS will always choose the latest.
This leads to an incorrect restore of the data, and I would like to figure out if there is a property that I can set that will force the Restore class to always use the latest backup point.
I have already tried to set the Restore.ToPointInTime but that won't work due to the recovery model of the database being simple.
I have found a MSDN article describing how to choose your restore time, and it includes setting the database to full recovery mode:
http://technet.microsoft.com/en-us/library/ms179451(v=sql.105).aspx
Is this necessary when using SMO, and is there a way to do it using pure SMO (no C# sql commands)? I have used the Restore.ReadBackupHeaders and from that I am able to extract the available backup points in time, but not able to set the one to restore anywhere.
EDIT:
Here is the code I am using, including a recent change which attempts to set the database recovery model via smo:
public void RestoreDatabase(string databaseName, string backupPath)
{
var server = new Server(GetServerConnection());
//If the database doesn't exist, create it so that we have something
//to overwrite.
if (!server.Databases.Contains(databaseName))
{
var database = new Database(server, databaseName);
database.Create();
}
var targetDatabase = server.Databases[databaseName];
targetDatabase.RecoveryModel = RecoveryModel.Full;
targetDatabase.Alter();
Restore restore = new Restore();
var backupDeviceItem = new BackupDeviceItem(backupPath, DeviceType.File);
restore.Devices.Add(backupDeviceItem);
restore.Database = databaseName;
restore.ReplaceDatabase = true;
restore.Action = RestoreActionType.Database;
var fileList = restore.ReadFileList(server);
var dataFile = new RelocateFile();
string mdf = fileList.Rows[0][1].ToString();
dataFile.LogicalFileName = fileList.Rows[0][0].ToString();
dataFile.PhysicalFileName = server.Databases[databaseName].FileGroups[0].Files[0].FileName;
var logFile = new RelocateFile();
string ldf = fileList.Rows[1][1].ToString();
logFile.LogicalFileName = fileList.Rows[1][0].ToString();
logFile.PhysicalFileName = server.Databases[databaseName].LogFiles[0].FileName;
restore.RelocateFiles.Add(dataFile);
restore.RelocateFiles.Add(logFile);
var backupHeaderInfo = GetBackupHeaderInformation(restore, server);
var latestBackupDate = backupHeaderInfo.Max(backupInfo => backupInfo.BackupStartDate);
restore.ToPointInTime = latestBackupDate.ToString();
server.KillAllProcesses(databaseName);
restore.SqlRestore(server);
}
It seems like this should do the trick, however the line
targetDatabase.RecoveryModel = RecoveryModel.Full
does not seem to do anything to change the recovery model, leading me to still get the following exception:
The STOPAT option is not supported for databases that use the SIMPLE recovery model.
RESTORE DATABASE is terminating abnormally.
EDIT 2:
I added the line
targetDatabase.Alter();
and it fixed the not updating problem. However It now restores but leaves the database in restoring mode, so it is unable to be queried.
EDIT 3:
I got the code working by setting the Restore.FileNumber property to be the maximum value of the positions in the BackupHeaders, which seems to do the trick, though I'm still unsure why the backup file has multiple backup headers, but only a single backup set.
The working code is below.
public void RestoreDatabase(string databaseName, string backupPath)
{
var server = new Server(GetServerConnection());
//If the database doesn't exist, create it so that we have something
//to overwrite.
if (!server.Databases.Contains(databaseName))
{
var database = new Database(server, databaseName);
database.Create();
}
var targetDatabase = server.Databases[databaseName];
targetDatabase.RecoveryModel = RecoveryModel.Full;
targetDatabase.Alter();
Restore restore = new Restore();
var backupDeviceItem = new BackupDeviceItem(backupPath, DeviceType.File);
restore.Devices.Add(backupDeviceItem);
restore.Database = databaseName;
restore.ReplaceDatabase = true;
restore.NoRecovery = false;
restore.Action = RestoreActionType.Database;
var fileList = restore.ReadFileList(server);
var dataFile = new RelocateFile();
dataFile.LogicalFileName = fileList.Rows[0][0].ToString();
dataFile.PhysicalFileName = server.Databases[databaseName].FileGroups[0].Files[0].FileName;
var logFile = new RelocateFile();
logFile.LogicalFileName = fileList.Rows[1][0].ToString();
logFile.PhysicalFileName = server.Databases[databaseName].LogFiles[0].FileName;
restore.RelocateFiles.Add(dataFile);
restore.RelocateFiles.Add(logFile);
var backupHeaderInfo = GetBackupHeaderInformation(restore, server);
restore.FileNumber = backupHeaderInfo.Where(backupInfo => backupInfo.BackupType == BackupType.Database).Max(backupInfo => backupInfo.Position);
server.KillAllProcesses(databaseName);
restore.SqlRestore(server);
targetDatabase.SetOnline();
}

Despite your saying that you don't have multiple backup sets, I think you do. From the documentation for the backupset table:
A backup set contains the backup from a single, successful backup operation.
So, if you have "multiple restore points" in a single backup file, you have multiple backup sets. Verify this by querying the dbo.backupset table in msdb
Pedantry aside, I think you're looking for the FileNumber property on the Restore object. This corresponds to the FILE = n backup set option in the T-SQL restore command. In order to get the last one, just pull the last row from your ReadBackupHeaders call.
To test for yourself, go through the motions of performing a restore through SSMS and then, instead of hitting "ok", hit the "Script" button near the top. I suspect that you'll see a FILE = <some number> in there somewhere.

How to tell if SQL Server automated upgrade is successful from C# WinForms app

I'm automating an upgrade of SQL Server 2005 Express to SQL Server 2008R2 Express via a WinForms app that is used to upgrade our application. The application is deployed at some 800+ locations, so we don't want any manual steps.
I've got the following code mostly written to perform the upgrade. I need to know, what's best practice for determining if the SQL Server installer completed successfully? Should I just look for an exit code of 0 for the process? Is that good enough; i.e. could it still exit with code of 0 if the upgrade had a problem and was rolled back (I'd test this, but don't know the best way to simulate a failure)?
Is there any other way to determine if the upgrade was successful in my C# app, so I can handle it properly if there was any error encountered by the SQL Server Installer?
try
{
//First, find the version of the currently installed SQL Server Instance
string sqlString = "SELECT SUBSTRING(CONVERT(VARCHAR, SERVERPROPERTY('productversion')), 0, 5)";
string sqlInstanceVersion = string.Empty;
using (DbCommand cmd = _database.GetSqlStringCommand(sqlString))
{
sqlInstanceVersion = cmd.ExecuteScalar().ToString();
}
if (sqlInstanceVersion.Equals(String.Empty))
{
//TODO throw an exception or do something else
}
//11.00 = SQL2012, 10.50 = SQL2008R2, 10.00 = SQL2008, 9.00 = SQL2005, 8.00 = SQL2000
switch (sqlInstanceVersion)
{
case "11.00":
case "10.50":
case "10.00":
//Log that the version is already up to date and return
return;
case "9.00":
case "8.00":
//We are on SQL 2000 or 2005, so continue with upgrade to 2008R2
break;
default:
//TODO throw an exception for unsupported SQL Server version
break;
}
string upgradeArgumentString = "/Q /ACTION=upgrade /INSTANCENAME={0} /ENU /IACCEPTSQLSERVERLICENSETERMS";
string instanceName = "YourInstanceNameHere";
string installerFilePath = AppDomain.CurrentDomain.BaseDirectory + "\\SQLEXPR_x86_ENU.exe";
if (!File.Exists(installerFilePath))
{
throw new FileNotFoundException(string.Format("Unable to find installer file: {0}", installerFilePath));
}
Process process = new Process
{
StartInfo = { FileName = installerFilePath, Arguments = String.Format(upgradeArgumentString, instanceName), UseShellExecute = false }
};
process.Start();
if (process.WaitForExit(SQLSERVER_UPGRADE_TIMEOUT))
{
//Do something here when the process completes within timeout.
//What should I do here to determine if the SQL Server Installer completed successfully? Look at just the exit code?
}
else
{
//The process exceeded timeout. Do something about it; like throw exception, or whatever
}
}
catch(Exception ex)
{
//Handle your exceptions here
}

Look at the full version string, nut just the first 5 chars of it. A successful upgrade will change the version string.

How to get SQL JobStep connection strings

After I get the server jobs and I got each job steps I want to get the connection string related to each step as you could find it while opening SQL management studio in jobs like this:
is there is a suitable way to get the connection strings for each package by C# code?
ServerConnection conn = new ServerConnection("localhost");
//new SqlConnection("data source=localhost;initial catalog=CPEInventory_20101122;integrated security=True;"));
Server server = new Server(conn);
JobCollection jobs = server.JobServer.Jobs;
var stepInformationsDetailsList = new List<StepInformationsDetails>();
foreach (Job job in jobs)
{
foreach (JobStep jobstep in job.JobSteps)
{
stepInformationsDetailsList.Add(new StepInformationsDetails() {
ServerName = job.Parent.MsxServerName,
ReportName = job.Name,
StepName = jobstep.Name,
Command = jobstep.Command,
Schedual = jobstep.DatabaseName,
StepID = jobstep.ID
});
}
}
dataGridView1.DataSource = stepInformationsDetailsList;

That data will all be in your Command variable. When you override/add anything on the job step tabs, the net result is that the command line passed to dtexec has those values. The UI is simply slicing arguments out of the command column in the msdb.dbo.sysjobsteps table to link them to various tabs.
On your localhost, this query ought to return back the full command line.
SELECT
JS.command
FROM
dbo.sysjobs AS SJ
INNER JOIN dbo.sysjobsteps AS JS
ON JS.job_id = SJ.job_id
WHERE
sj.name = 'Concessions made by Simba Auto-Report';
Since you are not overriding the value of the Connection Manager 'Simba Replica...', it is not going to show up in the output of the command. If it was, you'd have a string like /SQL "\"\MyFolder\MyPackage\"" /SERVER "\"localhost\sql2008r2\"" /CONNECTION SYSDB;"\"Data Source=SQLDEV01\INT;Initial Catalog=SYSDB;Provider=SQLNCLI10.1;Integrated Security=SSPI;Auto Translate=False;\"" /CHECKPOINTING OFF /REPORTING E The /CONNECTION section would correlate to your 'Simba Replica...' value.
I suspect, but do not know, that the UI is examining the SSIS package via the API Microsoft.SqlServer.Dts.Runtime and identifying all the Connection Manager, via Connections property to build out that dialog.

How to test connection to a data source in SSAS using C#

I have a database in Analysis Services on a remote server. This contains a data source for another database located on another remote server.
I am trying to write a connectivity test using C# which will check the database connection between the two databases.
I have been unable to do this using ADOMD.NET. I'm currently looking at using SMO to do this but I haven't had any luck so far.
I would greatly appreciate any advice or suggestions.
Update:
After further research, I have come up with the below test (Please note that I intend to add more try..catch blocks and Assertions later).
Also, this uses C:\Program Files\Microsoft SQL Server\100\SDK\Assemblies\Microsoft.AnalysisServices.DLL to access the Server, Database and
DataSource classes.
class ConnectivityTests
{
// Variables
String serverName = "";
String databaseName = "";
String dataSourceName = "";
[Test]
public void TestDataSourceConnection()
{
// Creates an instance of the Server
Server server = new Server();
server.Connect(serverName);
// Gets the Database from the Server
Database database = server.Databases[databaseName];
// Get the DataSource from the Database
DataSource dataSource = database.DataSources.FindByName(dataSourceName);
// Attempt to open a connection to the dataSource. Fail test if unsuccessful
OleDbConnection connection = new OleDbConnection(dataSource.ConnectionString);
try
{
connection.Open();
}
catch (OleDbException e)
{
Assert.Fail(e.ToString());
}
finally
{
connection.Close();
}
}
I believe that this test is sufficient for my testing (Once I've added some more try..catch blocks and Assertions). If the test passes, it means there are no connectivity issues between my machine and both servers, which implies that there shouldn't be any connectivity issues between the servers.
However, I have been unable to work out how to test the connection between the two servers directly and I am interested if anyone knows a way of doing this.

The best solution I have come across to doing this connectivity test is below:
Please note that this requires the Microsoft.AnalysisServices.DLL to be added as a reference.
class ConnectivityTests
{
// Variables
String serverName = "";
String databaseName = "";
String dataSourceName = "";
[Test]
public void TestDataSourceConnection()
{
try
{
// Creates an instance of the Server
Server server = new Server();
server.Connect(serverName);
// Gets the Database from the Server
Database database = server.Databases[databaseName];
// Get the DataSource from the Database
DataSource dataSource = database.DataSources.FindByName(dataSourceName);
// Attempt to open a connection to the dataSource. Fail test if unsuccessful
OleDbConnection connection = new OleDbConnection(dataSource.ConnectionString);
connection.Open();
}
catch (Exception e)
{
Assert.Fail(e.ToString());
}
finally
{
connection.Close();
}
}
}