Develop Reference

SSIS - File Existence check is not controlling the package task flow properly

There is a pre-existing SSIS package which performs multiple file manipulations based on a source file and often fails when that file is not found at the expected directory. I just want to build some smarts into it so that it will email notification of the missing file, rather than fail.
I have attempted MANY different Script Tasks, using VB, C and SQL, but NOTHING is controlling the process flow consistently. It works sometimes, and when it doesn't work others. I have listed my variables are below --- the top three are all I expected to use, but I added FullPath to simplify things. Unfortunately, that made no change.
My tests: I remove the source files from the directory and execute the package in VS, it DOES call the send email task letting me know the file does not exist. Then I put the files into place and execute the package, it calls the send email task again, as though the file is not there. I am not very good with breakpoints and watch windows, so I put two message boxes into place for Filepath and FileExists -- the Filepath returned IS correct, with the filename, yet the FileExists message box value returned immediately thereafter returns a 0. Note, at the very same time this is telling me it doesn't see the file, I have checked the disk and can see it is there.
Here's the kicker: I've been on this for days and was testing yesterday -- suddenly it works! I put the files into the source directory, it ran the whole process correctly. I removed the files from the source directory, it called the send mail task and completed successfully. I tested both conditions multiple times successfully -- and now it is failing again. I do not understand and have no desire or time to keep testing this file existence check script task that only works intermittently. I even tried today to get the File Properties task that I am hearing so much about (https://archive.codeplex.com/?p=filepropertiestask) but it is incompatible with the current versions of the software. I've tried VS 2019 and SSDT 2017, File Properties is incompatible/unsupported in either. Or, I just don't know how to install it.
Can anyone advise?
Variables -
FileName string, fileName.txt
FilePath string, C:\directory path\
FileExists boolean, False (though I've tried int32, even char N/Y)
FullPath string, C:\Directory path\filename.txt
C Script Task attempts -
public void Main()
{
// TODO: Add your code here
String Filepath = Dts.Variables["User::FilePath"].Value.ToString() + Dts.Variables["User::FileName"].Value.ToString();
if (
File.Exists(Filepath))
{
Dts.Variables["User::FileExists"].Value = 1;
}
else
Dts.Variables["User::FileExists"].Value = 0;
Dts.TaskResult = (int)ScriptResults.Success;
}
OR
//TODO: Add your code here
String Filepath = Dts.Variables["User::FilePath"].Value.ToString() + Dts.Variables["User::FileName"].Value.ToString();
if (
FileExists(Filepath))
{
Dts.Variables["User::FileExists"].Value = 1;
}
MessageBox.Show(Filepath);
MessageBox.Show(Dts.Variables{"User::FileExists"].Value.ToString());
Dts.TaskResult = (int)ScriptResults.Success;
}
or even just as simple as this:
Dts.Variables("FileExists").Value = File.Exists(Dts.Variables("FilePath").Value).ToString
Dts.TaskResult = (int)ScriptResults.Success;
VB Script Task -
Public Sub Main()
' Fill WriteVariable with value from ReadVariable
Dts.Variables("User::FileExists").Value = Dts.Variables("User::FullPath").Value
Dts.TaskResult = ScriptResults.Success
End Sub
Exec SQL Task -
DECLARE
#FilesExist BIT = 0,
#FolderPath VARCHAR(100) = 'C:\directory path\'
DECLARE #Files TABLE ([FileName] VARCHAR(100), Depth INT, [File] INT)
INSERT INTO #Files
EXEC master.sys.xp_dirtree #FolderPath,1,1;
IF EXISTS(
SELECT 1 FROM #Files
WHERE [FileName] = 'fileName.txt'
AND Depth = 1
AND [File] = 1
)
SET #FilesExist = 1
RETURN;
Script Task Precedent constraints:
Evaluation Operation: Expression and Constraint
Value: Success
Expression: #[User::FileExists]==1
Logical AND
Evaluation Operation: Expression and Constraint
Value: Success
Expression: #[User::FileExists]==0
Logical AND
This is a dummied screenshot of my control flow. Where the script task file existence check is the 7th item in the flow. The filename has no date in it. It is always 'filename.txt'. This file is created by the 4th task in the flow by merging other files, and I have just learned that I need to add a similar check here -- but there are multiple files, so I will need to do a wildcard check before the 3rd task in the package as well.

While I like the elegance of Script Tasks, given the current set of tasks, I think you can get by with the out of the box tooling.
Foreach Loop Container
This is the work horse in the solution. It has a built in file enumerator so if your source file is actually SourceFile_YYYYMMDD.txt or something like that, you can just use sourcefile*.txt and it'll find it no problem.
The reason I like this, is that you can put all of your logic inside this container and if the file is found, it's just going to do the expected work. No precursor/successor stuff has to be defined to make it go.
I created another variable called CurrentFile and initialized it to the empty string. This, empty string starting value, is crucial for the success of the package. When the Foreach Loop finds a file, it is going to put the full file name into this variable.
When the package runs and all the file manipulation work is done, if a file was found, the current value of #[User::CurrentFile] is not going to be the empty string. If no file is found, then the value of CurrentFile remains the starting value.
I modified your existing FileExists boolean SSIS Variable to be Expression driven. In the properties, I used the following expression #[User::CurrentFile] != "" If the value of CurrentFile isn't our starting value, then it evaluates to true. Otherwise, it remains false.
Using those two "tricks", that leads us to an SSIS package like the following
I have two paths leading out of the Foreach loop container. Both specify success as the constraint and then we use #[User::FileExists] for the happy path (file found) and the inversion of that !#[User::FileExists] for the not exists path. Put your notification logic inside the container that says "No file found path"
Because I love me some Biml, I am attaching the Biml to create this solution.
Slightly less useful as you need to patch this into an existing package, but you should be able to create a minimum viable package that handles checking and alerting if file not found. And then you can compare a working example to your current implmentation.
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="so_62505561">
<Variables>
<Variable Name="FileExists" DataType="Boolean" EvaluateAsExpression="true">#[User::CurrentFile] != ""</Variable>
<Variable Name="FileName" DataType="String">so_62505561.txt</Variable>
<Variable Name="FilePath" DataType="String">C:\ssisdata\input</Variable>
<Variable Name="FileSpecification" DataType="String">so_62505561*.txt</Variable>
<Variable Name="FullPath" DataType="String"></Variable>
<Variable Name="CurrentFile" DataType="String"></Variable>
</Variables>
<Tasks>
<ForEachFileLoop Name="FELC Do File Work" FileSpecification="*.txt" ConstraintMode="LinearOnSuccess" Folder="C:\tmp">
<Expressions>
<Expression ExternalProperty="FileSpec">#[User::FileSpecification]</Expression>
<Expression ExternalProperty="Directory">#[User::FilePath]</Expression>
</Expressions>
<VariableMappings>
<VariableMapping Name="0" VariableName="User.CurrentFile" />
</VariableMappings>
<Tasks>
<Container Name="Placeholder for work">
</Container>
</Tasks>
</ForEachFileLoop>
<!-- this is the unhappy path -->
<Container Name="No file found path">
<PrecedenceConstraints>
<Inputs>
<Input EvaluationOperation="ExpressionAndConstraint" EvaluationValue="Success" Expression="!#[User::FileExists]" OutputPathName="FELC Do File Work.Output" />
</Inputs>
</PrecedenceConstraints>
</Container>
<Container Name="File found path">
<PrecedenceConstraints>
<Inputs>
<Input EvaluationOperation="ExpressionAndConstraint" EvaluationValue="Success" Expression="#[User::FileExists]" OutputPathName="FELC Do File Work.Output" />
</Inputs>
</PrecedenceConstraints>
</Container>
</Tasks>
</Package>
</Packages>
</Biml>
Using the script task to test for file existence
Given your assumptions
FileName string, fileName.txt
FilePath string, C:\directory path\
FileExists boolean, False
The Script Task should have FileName and FilePath as read only variables, FileExists is a read/write variable.
// I favor System.IO.Path.Combine for path manipulation as it figures out the correct separator to use
string filepath = System.IO.Path.Combine(Dts.Variables["User::FilePath"].Value.ToString(), Dts.Variables["User::FileName"].Value.ToString());
if (System.IO.File.Exists(filepath))
{
Dts.Variables["User::FileExists"].Value = true;
}
// I favor emitting to the log I can audit it. Also, GUI events are not allowed when run from jobs
// Log the state of all our SSIS variables
bool fireAgain = false;
string message = "{0}::{1} : {2}";
foreach (var item in Dts.Variables)
{
Dts.Events.FireInformation(0, "SCR Echo Back", string.Format(message, item.Namespace, item.Name, item.Value), string.Empty, 0, ref fireAgain);
}
// Log the file path that was built
Dts.Events.FireInformation(0, "SCR Echo Back", string.Format(message, "local", "filepath", filepath), string.Empty, 0, ref fireAgain);
Dts.TaskResult = (int)ScriptResults.Success;
At this point, when the script runs, you'll have 4 information events in your log: the state of our 3 SSIS scoped variables and the state of our built path. In the event the downstream bits that depend on #[User::FileExists] are not working as expected, you have all your values in one convenient place.
At this point, the file check script task has exited and the precedent constraints should be flagged as Expression and Constraint using the following combos
Success + #[User::FileExists] (File exists path)
Success + !#[User::FileExists] (Alert path)
Logging note
Firing information events results in the information being emitted in two different locations for the a package run in Visual Studio.
The Output Window and the Progress tab.
Output Window would show something like
SSIS package "C:\Users\bfellows\source\repos\Integration Services Project1\Integration Services Project1\NewPackage.dtsx" starting.
Information: 0x0 at Script Task 1, SCR Echo Back: User::FileExists : False
Information: 0x0 at Script Task 1, SCR Echo Back: User::FileName : fileName.txt
Information: 0x0 at Script Task 1, SCR Echo Back: User::FilePath : C:\directory path\
Information: 0x0 at Script Task 1, SCR Echo Back: local::filepath : C:\directory path\fileName.txt
SSIS package "C:\Users\bfellows\source\repos\Integration Services Project1\Integration Services Project1\NewPackage.dtsx" finished: Success.
whereas the progress tab is a gui element
Final resolution
#sqldba discovered that in the 4th step the VBA was using a local variable FilesExist and not FileExists and since VB is forgiving of non-declared variables, that's where things got "weird"

Why does Dart wrap process arguments in quotes where C# does not?

I'm porting some C# code to Dart on Windows and just wasted a couple of hours on a frustrating difference in how Dart passes command line arguments through to the process.
Here's a portion of the C# code being ported, which executes just fine:
var startInfo = new ProcessStartInfo("C:\\Program Files\\Inkscape\\inkscape.exe", "--file=\"C:\\Users\\Kent\\test.svg\" --export-png=\"C:\\Users\\Kent\\test.png\" --export-width=100 --export-area-page")
{
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true,
};
var process = Process.Start(startInfo);
process.WaitForExit();
if (process.ExitCode != 0)
{
var stdout = process.StandardOutput.ReadToEnd();
var stderr = process.StandardError.ReadToEnd();
Console.WriteLine(stdout);
Console.WriteLine(stderr);
}
Running this correctly produces an output PNG file. However, the Dart equivalent does not:
final inkscapeResult = await Process.run(
r'C:\Program Files\Inkscape\inkscape.exe',
[
'--file="C:\\Users\\Kent\\test.svg"',
'--export-png="C:\\Users\\Kent\\test.png"',
'--export-width=100',
'--export-area-page',
],
);
final exitCode = inkscapeResult.exitCode;
if (exitCode != 0) {
writeError(inkscapeResult.stdout);
writeError(inkscapeResult.stderr);
}
Instead, the process exits with code 1 and the program outputs:
** (inkscape.exe:5604): WARNING **: 17:30:09.785: Can't open file: "C:\Users\Kent\test.svg" (doesn't exist)
** (inkscape.exe:5604): WARNING **: 17:30:09.785: Can't open file: "C:\Users\Kent\test.svg" (doesn't exist)
** (inkscape.exe:5604): WARNING **: 17:30:09.785: Specified document "C:\Users\Kent\test.svg" cannot be opened (does not exist or not a valid SVG file)
It is pretty well-known that Inkscape is a bit finicky when it comes to executing on the command line, but the advice I've read is to simply pass in full paths, which I am doing.
I found that if I eschew arguments, it works:
final inkscapeResult = await Process.run(
r'C:\Program Files\Inkscape\inkscape.exe --file="C:\\Users\\Kent\\test.svg" --export-png="C:\\Users\\Kent\\test.png" --export-width=100 --export-area-page',
[],
);
So I inferred that Dart must be doing something with the arguments that Inkscape does not like. I managed to catch it in the act using Process Explorer and it has the following command line:
"C:\Program Files\Inkscape\inkscape.exe" "--file=\"C:\Users\Kent\test.svg\"" "--export-png=\"C:\Users\Kent\test.png\"" --export-width=100 --export-area-page
If I run this directly from the command line, I get no output but the image also does not get generated, so presumably it's failing in the same fashion. Compare this with the command line produced by the C# code:
"C:\Program Files\Inkscape\inkscape.exe" --file="C:\Users\Kent\test.svg" --export-png="C:\Users\Kent\test.png" --export-width=100 --export-area-page
Dart is wrapping each individual argument with quotes, where C# is not (though C#'s API does not even separate individual arguments). This is somehow breaking Inkscape's parser.
Who is in the wrong here? Me, Dart, C#, or Inkscape?

I think the simplest way is to split the argument name and the argument value like that:
final inkscapeResult = await Process.run(r'C:\Program Files\Inkscape\inkscape.exe', [
'--file', 'C:\\Users\\Kent\\test.svg',
'--export-png', 'C:\\Users\\Kent\\test.png',
'--export-width', '100',
'--export-area-page',
]);
That way you don't have to worry about escaping or quoting arguments.

Handling Jenkins Groovy script exceptions

I am executing a C# exe, CRS.exe, that I expect to return a non-zero value, such as -1. I am using ps to get the value back, and have this within a stage:
try{
pcode = (powershell(returnStdout: true, script: 'return Invoke-Expression -Command \" .\\perfmon\\CRS.exe hello \"'))''
echo "Pcode = ${pcode} "
}
catch (err) {echo err.message }
echo "Pcode = ${pcode} "
Based on this post, "Normally, a script which exits with a nonzero status code will cause the step to fail with an exception." --
Jenkins pipeline bubble up the shell exit code to fail the stage
I want to handle this non-zero result, is the exception handler the only way?
Results of above run:
Running PowerShell script
tester arg = hello
[Pipeline] echo
script returned exit code -1
[Pipeline] echo
Pcode = null
Interestingly enough, a char return seems to be fine? This returns without throwing an exception
icode = (powershell(returnStdout: true, script: 'return Invoke-Expression -Command \'.\\perfmon\\zipInstaller.ps1 -urlString ' + fileContents + "'"))
echo "icode = ${icode} "
Results in
[Pipeline] {
[Pipeline] powershell
[Chris] Running PowerShell script
[Pipeline] echo
icode = -5
I would like to catch the return codes from the exe's and manage my groovy pipelines flow based on that. Any insight would be greatly appreciated.

Instead of using returnStdout: true, try using returnStatus: true. It should always return the exit code. Not as helpful if you need the output from the powershell command all at once (without returnStdout, it will just print output to the Jenkins log), but as a workaround for that you could pipe output to a file and then print that out.
Another option (but it's ugly, so I don't recommend it) is to call Powershell from a bat command. bat should mask the error code for the powershell command, so Jenkins won't get excited and fail automatically, and you'll still get stdout. Obviously not too helpful if you actually wanted the error code though. Your line would look something like
pcode = (bat(returnStdout: true, script: 'Powershell.exe "return Invoke-Expression -Command \" .\\perfmon\\CRS.exe hello \"'"))
That line will need a bit of refining, but it might be another option.

C# process's exit code always returning Zero while running ADB commands? [duplicate]

We have an android device and as part of testing I need to excute a console test application on the target device. If the test application detects an error it returns -1.
I can use adb shell to run the test applications remotely on the target but I can't find a way of getting back the return code. I need this so I that I can build this into an automated test suite.
I could try grepping the console output for some failure text but that is a bit grubby. Does anyone know of a more elegant solution?

This is a workaround to get the exit code:
adb shell '{your command here} > /dev/null 2>&1; echo $?'
This is a wrapper around adb in Ruby:
def adb(opt)
input = "#{adb_command} #{opt[:command]} #{opt[:params]}"
puts "Executing #{input}...\n"
output = nil
exit_code = 0
def wait_for(secs)
if secs
begin
Timeout::timeout(secs) { yield }
rescue
print 'execution expired'
end
else
yield
end
end
wait_for(opt[:timeout]) do
case opt[:command]
when :install, :push, :uninstall
output, exit_code = `#{input}`, $?.to_i
when :shell
input = "#{adb_command} shell \"#{opt[:params]}; echo \\$?\""
output = `#{input}`.split("\n")
exit_code = output.pop.to_i
output = output.join("\n")
else
raise 'Error: param command to adb not defined!'
end
end
return if opt[:ignore_fail] and output =~ /#{opt[:ignore_fail]}/
raise output unless exit_code == 0
end

You could use Facebook's fb-adb, a "A better shell for Android devices" which "propagates program exit status instead of always exiting with status 0".

Redirect standard output of exe to my Console application

There is an executable called jlink.exe that I will like to make of use on my console application. Because jlink.exe is not a .net application it is not possible for me to reference it and call it's methods. As a result I will like to use it reading it's output
When I start that exe from windows explorer this is how it looks:
It then waits for a command. If I type mem 88 2 then I get back:
Now I will like to do the same thing with my .net console application reading it's standard output but for some reason I cannot read the output. This is what I have:
// change working directory
Directory.SetCurrentDirectory(#"C:\Program Files (x86)\IAR Systems\Embedded Workbench 6.4 Evaluation\arm\bin");
Process p = new Process( )
{
StartInfo = new ProcessStartInfo( #"jlink.exe" )
{
UseShellExecute = false ,
RedirectStandardOutput = true ,
RedirectStandardInput = true
}
};
p.Start( );
StreamWriter standardInput = p.StandardInput;
StreamReader standardOutput = p.StandardOutput;
var line = string.Empty;
while ( ( line = standardOutput.ReadLine( ) ) != null )
{
Console.WriteLine( line );
}
why when I run that code I just get a black window... I do not get the same output as when running that executable. When I run that code against other executable it works great.
If I then type mem 88 2 and press ENTER nothing happens. If I press enter one more time then I finally get what I want with extra content. Why am I getting this behavior with that executable?
Edit
I have looked at similar question such as this one:
process.standardoutput.ReadToEnd() always empty?
I don't think I am doing something wrong. I guess there is something weird with that executable. To use that executable you may download it at: http://supp.iar.com/Download/SW/?item=EWARM-EVAL or at http://www.iar.com/en/Products/IAR-Embedded-Workbench/ARM/ . Then go to C:\Program Files (x86)\IAR Systems\Embedded Workbench 6.4 Evaluation\arm\bin and execute jlink.exe (If you do not have a board connected via usb you will get an error but it does not matter. the error message does not show up either).