Develop Reference

How use C# in "Script Task" to get files from Input folder in Order by "Date Modified" [duplicate]

i have an folder, in which i receive .csv files for every half hour with time stamps. Now, i need to take the latest file from the available files and import it into sql server.
For Example
in my source folder, i have
test_01112012_120122.csv
test_01112012_123022.csv
test_01112012_123555.csv
now i need to fetch the latest file and import that file into sql server with the help of SSIS.
Thanks
satish

The code from #garry Vass, or one like it, is going to be needed even if you're using SSIS as your import tool.
Within SSIS, you will need to update the connection string to your flat file connection manager to point to the new file. Ergo, you need to determine what is the most recent file.
Finding the most recent file
Whether you do it by file attributes (Garry's code) or slicing and dicing of file names is going to be dependent upon what your business rules are. Is it always the most recently modified file (attribute) or does it need to be based off the file name being interpreted as a sequence. This matters if the test_01112012_120122.csv had a mistake in it and the contents are updated. The modified date will change but the file name will not and those changes wouldn't get ported back into the database.
I would suggest you create 2 variables of type String and scoped to the package named RootFolder and CurrentFile. Optionally, you can create one called FileMask if you are restricting to a particular type like *.csv. RootFolder would be the base folder you expect to find files in C:\ssisdata\MyProject. CurrentFile will be assigned a value from a script of the fully qualified path to the most recently modified file. I find it helpful at this point to assign a design-time value to CurrentFile, usually to the oldest file in the collection.
Drag a Script Task onto the Control Flow and set as your ReadOnlyVariable User::RootFolder (optionally User::FileMask). Your ReadWriteVariable would be User::CurrentFile.
This script would go inside the public partial class ScriptMain: ... braces
/// <summary>
/// This verbose script identifies the most recently modified file of type fileMask
/// living in RootFolder and assigns that to a DTS level variable.
/// </summary>
public void Main()
{
string fileMask = "*.csv";
string mostRecentFile = string.Empty;
string rootFolder = string.Empty;
// Assign values from the DTS variables collection.
// This is case sensitive. User:: is not required
// but you must convert it from the Object type to a strong type
rootFolder = Dts.Variables["User::RootFolder"].Value.ToString();
// Repeat the above pattern to assign a value to fileMask if you wish
// to make it a more flexible approach
// Determine the most recent file, this could be null
System.IO.FileInfo candidate = ScriptMain.GetLatestFile(rootFolder, fileMask);
if (candidate != null)
{
mostRecentFile = candidate.FullName;
}
// Push the results back onto the variable
Dts.Variables["CurrentFile"].Value = mostRecentFile;
Dts.TaskResult = (int)ScriptResults.Success;
}
/// <summary>
/// Find the most recent file matching a pattern
/// </summary>
/// <param name="directoryName">Folder to begin searching in</param>
/// <param name="fileExtension">Extension to search, e.g. *.csv</param>
/// <returns></returns>
private static System.IO.FileInfo GetLatestFile(string directoryName, string fileExtension)
{
System.IO.DirectoryInfo directoryInfo = new System.IO.DirectoryInfo(directoryName);
System.IO.FileInfo mostRecent = null;
// Change the SearchOption to AllDirectories if you need to search subfolders
System.IO.FileInfo[] legacyArray = directoryInfo.GetFiles(fileExtension, System.IO.SearchOption.TopDirectoryOnly);
foreach (System.IO.FileInfo current in legacyArray)
{
if (mostRecent == null)
{
mostRecent = current;
}
if (current.LastWriteTimeUtc >= mostRecent.LastWriteTimeUtc)
{
mostRecent = current;
}
}
return mostRecent;
// To make the below code work, you'd need to edit the properties of the project
// change the TargetFramework to probably 3.5 or 4. Not sure
// Current error is the OrderByDescending doesn't exist for 2.0 framework
//return directoryInfo.GetFiles(fileExtension)
// .OrderByDescending(q => q.LastWriteTimeUtc)
// .FirstOrDefault();
}
#region ScriptResults declaration
/// <summary>
/// This enum provides a convenient shorthand within the scope of this class for setting the
/// result of the script.
///
/// This code was generated automatically.
/// </summary>
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
}
Updating a Connection Manager
At this point, our script has assigned a value to the CurrentFile variable. The next step is to tell SSIS we need to use that file. In your Connection Manager for your CSV, you will need to set an Expression (F4 or right click and select Properties) for the ConnectionString. The value you want to assign is our CurrentFile variable and the way that's expressed is #[User::CurrentFile]
Finally, these screen shots are based on the upcoming release of SQL Server 2012 so the icons may appear different but the functionality remains the same.

Assuming that you wanted to use C#, to get the newest file in a given directory, you can use a method like this...
private static FileInfo GetLatestFile(string directoryName, string fileExtension)
{
DirectoryInfo directoryInfo = new DirectoryInfo(directoryName);
return directoryInfo.GetFiles(fileExtension)
.OrderByDescending(q => q.LastWriteTimeUtc)
.FirstOrDefault();
}
This method is called like...
FileInfo file = GetLatestFile( "C:\myDirectory", "*.csv");
And it returns a FileInfo instance (or null) of the file with the most recent write time. You can then use the FileInfo instance to get the name of the file and so on for your processing...

PowerShell Object returns null

I have below command and it returns me null object . When I run the command separately in PowerShell window I get the right result. Below is my PowerShell method which is calling the command and the also the PowerShell command which I have defined. I am basically looking to return a string value. Please let me know what wrong am I doing?
C# method:
public string RunScript( string contentScript, Dictionary<string, EntityProperty> parameters)
{
List<string> parameterList = new List<string>();
foreach( var item in parameters )
{
parameterList.Add( item.Value.ToString() );
}
using( PowerShell ps = PowerShell.Create() )
{
ps.AddScript( contentScript );
// in ContentScript I get "Get-RowAndPartitionKey" on debugging
ps.AddParameters( parameterList );//I get list of strings
IAsyncResult async = ps.BeginInvoke();
StringBuilder stringBuilder = new StringBuilder();
foreach( PSObject result in ps.EndInvoke( async ) )
// here i get result empty in ps.EndInvoke(async)
{
stringBuilder.AppendLine( result.ToString() );
}
return stringBuilder.ToString();
}
}
}
My Powershell GetRowAndPartitionKey cmdlet definition, which the code above is trying to call:
public abstract class GetRowAndPartitionKey : PSCmdlet
{
[Parameter]
public List<string> Properties { get; set; } = new List<string>();
}
[Cmdlet( VerbsCommon.Get, "RowAndPartitionKey" )]
public class GetRowAndPartitionKeyCmd : GetRowAndPartitionKey
{
protected override void ProcessRecord()
{
string rowKey = string.Join( "_", Properties );
string pKey = string.Empty;
WriteObject( new
{
RowKey = rowKey,
PartitionKey = pKey
} );
}
}
}

When using the PowerShell SDK, if you want to pass parameters to a single command with .AddParameter() / .AddParameters() / AddArgument(), use .AddCommand(), not .AddScript()
.AddScript() is for passing arbitrary pieces of PowerShell code that is executed as a script block to which the parameters added with .AddParameters() are passed.
That is, your invocation is equivalent to & { Get-RowAndPartitionKey } <your-parameters>, and as you can see, your Get-RowAndPartitionKey command therefore doesn't receive the parameter values.
See this answer or more information.
Note: As a prerequisite for calling your custom Get-RowAndPartitionKey cmdlet, you may have to explicitly import the module (DLL) that contains it, which you can do:
either: with a separate, synchronous Import-Module call executed beforehand (for simplicity, I'm using .AddArgument() here, with passes an argument positionally, which binds to the -Name parameter (which also accepts paths)):
ps.AddCommand("Import-Module").AddArgument(#"<your-module-path-here>").Invoke();
or: as part of a single (in this case asynchronous) invocation - note the required .AddStatement() call to separate the two commands:
IAsyncResult async =
ps.AddCommand("Import-Module").AddArgument(#"<your-module-path-here>")
.AddStatement()
.AddCommand("GetRowAndPartitionKey").AddParameter("Properties", parameterList)
.BeginInvoke();
"<your-module-path-here>" refers to the full file-system path of the module that contains the Get-RowAndPartitionKey cmdlet; depending on how that module is implemented, it is either a path to the module's directory, its .psd1 module manifest, or to its .dll, if it is a stand-alone assembly.
Alternative import method, using the PowerShell SDK's dedicated .ImportPSModule() method:
This method obviates the need for an in-session Import-Module call, but requires extra setup:
Create a default session state.
Call .ImportPSModule() on it to import the module.
Pass this session state to PowerShell.Create()
var iss = InitialSessionState.CreateDefault();
iss.ImportPSModule(new string[] { #"<your-module-path-here>" });
var ps = PowerShell.Create(iss);
// Now the PowerShell commands submitted to the `ps` instance
// will see the module's exported commands.
Caveat: A PowerShell instance reflects its initial session state in .Runspace.InitialSessionState, but as a conceptually read-only property; the tricky part is that it is technically still modifiable, so that mistaken attempts to modify it are quietly ignored rather than resulting in exceptions.
To troubleshoot these calls:
Check ps.HadErrors after .Invoke() / .EndInvoke() to see if the PowerShell commands reported any (non-terminating) errors.
Enumerate ps.Streams.Errors to inspect the specific errors that occurred.
See this answer to a follow-up question for self-contained sample code that demonstrates these techniques.

Programmatically script SSIS package - Dynamic XML to OLE DB

I've been trying to piece together how other users have finished their projects, but my understanding is still limited.
I want to take any given XML source, make a Data Flow Task, and pass its data to an OLE DB destination matching the table name of the XML file. Running it with the visual tool means I cannot do dynamic data flow tasks because the Metadata does not refresh.
I have created a script that creates a package, but when I open the package in Visual Studio, it has a red-x saying that there cannot be zero Input Columns. When I drill down and look at the mappings of the OLE DB Destination, then click OK - it corrects it for me. I cannot figure out how to do that programmatically.
I've seen others solve it by using foreach loops and going through the Input columns, but I cannot seem to figure it out.
I also have a separate script that I tried to mimic several people's scripts with, and it has different issues. Not sure how to post it as an attachment
Thank you in advance for the help :)
EDIT
I've been getting positive feedback for trying out BIML, and I will...but I want to know if in the short term anyone can help me figure out why this doesn't fill in ExternalMetaDataColumnId for my input. I've posted my updated code below with foreach loops that aren't doing what I expect them to.
Thank you
#region
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.SqlServer.Dts.Runtime;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using System.Xml;
#endregion
namespace ConsoleApplication3
{
class Program
{
static void Main(string[] args)
{
#region Initial Setup
Application a = new Application();
Package p = new Package();
TaskHost t = p.Executables.Add("DTS.Pipeline") as TaskHost;
t.Name = "DataFlow Task";
t.Description = "Flat File to Database";
MainPipe mp = t.InnerObject as MainPipe;
#endregion
#region Flat File Source in Dataflow Task
IDTSComponentMetaData100 md = mp.ComponentMetaDataCollection.New();
md.ComponentClassID = "Microsoft.XmlSourceAdapter";
md.Name = "XML Source";
CManagedComponentWrapper wrp = md.Instantiate();
wrp.ProvideComponentProperties();
#endregion
#region Add connection manager to OLE DB
ConnectionManager conn = p.Connections.Add("OLEDB");
conn.Name = "westcoastuserDBO";
conn.ConnectionString = "Data Source=SERVER;Initial Catalog=DBO;Provider=SQLNCLI11.1;Integrated Security=SSPI;Auto Translate=False;";
#endregion
#region XML Source Properties
wrp.SetComponentProperty("XMLData", #"C:\Users\file.xml");
wrp.SetComponentProperty("XMLSchemaDefinition", #"C:\Users\file.xsd");
wrp.SetComponentProperty("AccessMode", 0);
wrp.SetComponentProperty("UseInlineSchema", false);
//below does not work
//wrp.SetComponentProperty("XMLIntegerMapping", 0).TypeConverter = "Microsoft.SqlServer.Dts.Pipeline.XmlSourceAdapter + XMLIntegerMappingConverter";
wrp.ReinitializeMetaData();
wrp.ReleaseConnections();
IDTSComponentMetaData100 md2 = mp.ComponentMetaDataCollection.New();
md2.ComponentClassID = "Microsoft.OLEDBDestination";
CManagedComponentWrapper wrp2 = md2.Instantiate();
wrp2.ProvideComponentProperties();
md2.Name = "OLE DB Connection";
md2.UsesDispositions = true;
md2.Version = 4;
wrp2.SetComponentProperty("OpenRowset", "dbo.authorizations");
#endregion
IDTSPath100 path = mp.PathCollection.New();
path.AttachPathAndPropagateNotifications(md.OutputCollection[0], md2.InputCollection[0]);
IDTSInput100 input = md2.InputCollection[0];
IDTSVirtualInput100 vInput = input.GetVirtualInput();
//below taken from https://stackoverflow.com/questions/12587709/c-sharp-ssis-data-flow-component-creating-custom-input-columns
IDTSExternalMetadataColumnCollection100 externalColumnCollection = input.ExternalMetadataColumnCollection;
// Iterate through the virtual input column collection.
foreach (IDTSVirtualInputColumn100 vColumn in vInput.VirtualInputColumnCollection)
{
// Call the SetUsageType method of the destination
// to add each available virtual input column as an input column.
wrp2.SetUsageType(
input.ID, vInput, vColumn.LineageID, DTSUsageType.UT_READONLY);
}
// Get the destination's default output collection
IDTSOutputCollection100 outColl = md2.OutputCollection;
// Iterate through the outputs in default output collection
foreach (IDTSOutput100 output in outColl)
{
// Iterate through the default output columns in the output
int count = output.OutputColumnCollection.Count;
foreach (IDTSOutputColumn100 outputColumn in output.OutputColumnCollection)
{
// Get the output's external metadata column collection
IDTSExternalMetadataColumnCollection100 extMetadataColumnColl = output.ExternalMetadataColumnCollection;
// Iterate through the external metadata column collection's external metadata columns
foreach (IDTSExternalMetadataColumn100 extMetadataColumn in extMetadataColumnColl)
{
// Call the MapOutPutColumn method of the destination to map
// each available output column to an external metadata column
wrp2.MapOutputColumn(
output.ID, outputColumn.ID, extMetadataColumn.ID, true);
}
}
}
md2.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.GetExtendedInterface(conn);
md2.RuntimeConnectionCollection[0].ConnectionManagerID = conn.ID;
conn.AcquireConnection(null);
#region Save Package to FileSystem
string packageXml = #"C:\Users\test.dtsx";
XmlDocument myPkgDocument = new XmlDocument();
p.SaveToXML(ref myPkgDocument, null, null);
a.SaveToXml(packageXml, p, null);
#endregion
}
}
}

I think the problem that you are not mapping the input columns to the OLEDB Destination, and after opening the package, if you click on the OLEDB Destination and go to the Mapping section, it will automatically map the columns based on their names. The Foreach loop that is used by others are to loop over columns and map them to the related Destination columns.
There are many articles talking about creating SSIS package dynamically, you can refer to them for more information:
Dynamic Data Flow in SSIS using .NET/C#
Programmatically map the columns of a flat file destination?
Building Packages Programmatically
Samples for creating SSIS packages programmatically
Generating SSIS Packages Programmatically (Part I)

Delete files based on variable SSIS Script

I have a script in ssis which deletes the file and I need to modify the script by adding variable so that process can be executed dynamically. I would appreciate if some can help me out by showing how to add variable in the script below:-
enter public void Main()
{
int RetentionPeriod = 0;
string directoryPath = #"\\ABCD\EFG\HIJ";--need to add location variable
string[] oldFiles = System.IO.Directory.GetFiles(directoryPath, "*.csv");
foreach (string currFile in oldFiles)
{
FileInfo currFileInfo = new FileInfo(currFile);
if (currFileInfo.LastWriteTime < (DateTime.Now.AddDays(-RetentionPeriod)))---need to add date variable here
{
currFileInfo.Delete();
}
}
// TODO: Add your code here
Dts.TaskResult = (int)ScriptResults.Success;
}
}
}
As shown in the script I need to add two variable namely location variable and Date variable.Both the variables have string data type
I know this question is very silly but I have no exp in writing SSIS scripts

string directoryPath = Dts.Variables["User::NameOfStringVariable"].Value
You also have to add the variable to the ReadOnly list on the script task configuration. here is a screen shot from another question that shows you where to make the variable accessible to the script:
and in case you don't know where/how to add a variable to a package. One easy way is to right click on the grey area of the control flow and choose variables and that will bring up the variables window then simply add the variable with the appropriate datatype that you want.

this is over coded: try this:
set a variable as answered above to contain the file path
Add this to main after setting a System.IO; reference
public void Main()
{
string FilePath = Dts.Variables["UserControl::File"].Value.ToString();
if (File.Exists(FilePath))
{
File.Delete(FilePath);
}
Dts.TaskResult = (int)ScriptResults.Success;
}

What is the EzAPI equivalent for using an OLE DB Source command from variable?

tl;dr
What is the EzAPI code to use an OLE DB Source with data access mode of "SQL command from variable" and assign a variable?
Preamble
Once a month, we need to refresh our public test site with subsets of production data. We have determined that for our needs, an SSIS solution provides the best fit for accomplishing this task.
My goal is to systematically build a large number (100+) of "replication" packages. EzAPI is a friendly wrapper to the SSIS object model and it seems like a great way to save mouse-clicks.
I would like for my packages to look like
Variable - "tableName"; [Schema].[TableName]
Variable - "sourceQuery"; SELECT * FROM [Schema].[TableName]
DataFlow - "Replicate Schema_TableName"
OLE DB Source - "Src Schema_TableName"; Data Access Mode: SQL command from variable; Variable name: User::sourceQuery
OLE DB Destination - "Dest Schema_TableName"; Table or view name variable- fast load; Variable name - User::tableName
Code
This is the code for my table to table replication package.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.SqlServer.SSIS.EzAPI;
using Microsoft.SqlServer.Dts.Runtime;
namespace EzApiDemo
{
public class TableToTable : EzSrcDestPackage<EzOleDbSource, EzSqlOleDbCM, EzOleDbDestination, EzSqlOleDbCM>
{
public TableToTable(Package p) : base(p) { }
public static implicit operator TableToTable(Package p) { return new TableToTable(p); }
public TableToTable(string sourceServer, string database, string table, string destinationServer) : base()
{
string saniName = TableToTable.SanitizeName(table);
string sourceQuery = string.Format("SELECT D.* FROM {0} D", table);
// Define package variables
this.Variables.Add("sourceQuery", false, "User", sourceQuery);
this.Variables.Add("tableName", false, "User", table);
// Configure DataFlow properties
this.DataFlow.Name = "Replicate " + saniName;
this.DataFlow.Description = "Scripted replication";
// Connection manager configuration
this.SrcConn.SetConnectionString(sourceServer, database);
this.SrcConn.Name = "PROD";
this.SrcConn.Description = string.Empty;
this.DestConn.SetConnectionString(destinationServer, database);
this.DestConn.Name = "PREPROD";
this.DestConn.Description = string.Empty;
// Configure Dataflow's Source properties
this.Source.Name = "Src " + saniName;
this.Source.Description = string.Empty;
this.Source.SqlCommand = sourceQuery;
// Configure Dataflow's Destination properties
this.Dest.Name = "Dest " + saniName;
this.Dest.Description = string.Empty;
this.Dest.Table = table;
this.Dest.FastLoadKeepIdentity = true;
this.Dest.FastLoadKeepNulls = true;
this.Dest.DataSourceVariable = this.Variables["tableName"].QualifiedName;
this.Dest.AccessMode = AccessMode.AM_OPENROWSET_FASTLOAD_VARIABLE;
this.Dest.LinkAllInputsToOutputs();
}
/// <summary>
/// Sanitize a name so that it is valid for SSIS objects.
/// Strips []/\:=
/// Replaces . with _
/// </summary>
/// <param name="name"></param>
/// <returns></returns>
public static string SanitizeName(string name)
{
string saniName = name.Replace("[", String.Empty).Replace("]", string.Empty).Replace(".", "_").Replace("/", string.Empty).Replace("\\", string.Empty).Replace(":", string.Empty);
return saniName;
}
}
}
Invocation looks like TableToTable s2 = new TableToTable(#"localhost\localsqla", "AdventureWorks", "[HumanResources].[Department]", #"localhost\localsqlb"); and that builds a package that does what I want except for using a variable in the source.
Problem
The above code supplies the access mode as SQL Query and the query is embedded in the OLE Source. The desire it to use "SQL Command From Variable" and that variable being #[User::sourceQuery] What I'm stuck on is using a variable in the source.
It should be a simple matter of assigning something like
this.Source.DataSourceVariable = this.Variables["sourceQuery"].QualifiedName;
this.Source.AccessMode = AccessMode.AM_SQLCOMMAND_VARIABLE;
This results in the correct data access mode selected but the variable isn't populated.
You can observe that I perform a similar step in the destination which does accept the variable and works "right."
this.Dest.DataSourceVariable = this.Variables["tableName"].QualifiedName;
this.Dest.AccessMode = AccessMode.AM_OPENROWSET_FASTLOAD_VARIABLE;
What doesn't work
Listing out the permutations I've attempted
this.Source.AccessMode = AccessMode.AM_OPENROWSET;
Results in Data Access Mode set to Table or View and name of table or the view is blank.
this.Source.AccessMode = AccessMode.AM_OPENROWSET_VARIABLE;
Results in Data Access Mode set to "Table or view name variable" and variable name is sourceQuery. Very close to what I want, except the access mode is not correct. Were this package to run, it'd blow up as the OpenRowSet would expect a straight table name.
this.Source.AccessMode = AccessMode.AM_SQLCOMMAND;
Results in Data Access Mode set to "SQL Command" and the SQL command text is "User::sourceQuery" That's the literal value of the variable name so it's the right thing but since the access mode is wrong, it doesn't work.
this.Source.AccessMode = AccessMode.AM_OPENROWSET_FASTLOAD;
this.Source.AccessMode = AccessMode.AM_OPENROWSET_FASTLOAD_VARIABLE;
Niether of these are correct access modes as they are for destinations (I still tried them but they didn't work as expected).
At this point, I thought I'd try to work backwards by creating a package that has the OLE DB source defined as I want it and then inspect the source object's properties.
Application app = new Application();
Package p = app.LoadPackage(#"C:\sandbox\SSISHackAndSlash\SSISHackAndSlash\EzApiPackage.dtsx", null);
TableToTable to = new TableToTable(p);
My code has set both SqlCommand and DataSourceVarible with the variable's qualified name. I've pulled down changeset 65381 and compiled that (after fixing some references to the SQL Server 2012 dlls) in hopes there might have been a fix since the Dec 30 2008 Stable build but to no avail.
Have I found a bug in their code or am I just missing something?

The current, stable build of EzAPI does not support the assignment of a variable as an OleDB Source property. I opened a similar discussion over on CodePlex and ended up learning more about how all of this works.
The root problem is the related property "SqlCommandVariable" should be set when the access mode is set to "SQL Command from Variable." Currently, the code only covers destination variables.
My resolution was to download the source code and modify the setter for the property DataSourceVariable in EzComponents.cs (line 1027 for changeset 65381)
set
{
m_comp.SetComponentProperty("OpenRowsetVariable", value);
if (AccessMode == AccessMode.AM_SQLCOMMAND_VARIABLE)
{
m_comp.SetComponentProperty("SqlCommandVariable", value);
}
ReinitializeMetaData();
}
If you're looking to get this problem resolved properly, you may upvote the Issue

Try swapping around
this.Source.DataSourceVariable = this.Variables["sourceQuery"].QualifiedName;
this.Source.AccessMode = AccessMode.AM_SQLCOMMAND_VARIABLE;
to
this.Source.AccessMode = AccessMode.AM_SQLCOMMAND_VARIABLE;
this.Source.DataSourceVariable = this.Variables["sourceQuery"].QualifiedName;
I've discovered that the order matters more than it does with a typical API.