Develop Reference

File counter expression in SSIS using execute process task output variable

I have a console application in c# that downloads files from a website.
I have created a variable named Filecount that counts the number of files downloaded in that instance.
In SSIS, I have set the StandardOutputVariable in the Execute process task configuration as User::FileCount that should pass through the number of files that it has downloaded.
I want to create an SQL task that will truncate the table if the file count is greater than 0.
However, when I try to evaluate my expression, it always comes back as true however, this should not occur as no counts have been passed through yet meaning it should be evaluated as false.
Can someone explain if I have either written the expression wrong or set the variable incorrectly?
int fileCount = 0;
using (var client = new HttpClient(handler))
{
client.DefaultRequestHeaders.Clear();
client.DefaultRequestHeaders.Add("Accept", "application/json;odata=verbose");
var response = await client.GetAsync(siteUrl + "/_api/Web/GetFolderByServerRelativeUrl('XXX')/Files");
response.EnsureSuccessStatusCode();
json = await response.Content.ReadAsStringAsync();
Root deserializedResults = JsonConvert.DeserializeObject<Root>(json);
foreach (Result result in deserializedResults.d.results)
{
if (result.TimeCreated > DateTime.Today.AddDays(-7))
{
DownloadFileViaRestAPI(siteUrl, credentials, "XXX", result.Name, "XXX");
fileCount++;
}
}
}
Console.Write(fileCount);
SSIS Execute Process Task;
StandardOutVariable = User::FileCount
SQL Task expression;
Property: Disable
Expression: Len(Trim(#[User::FileCount])) > 0 ? False : True

tl;dr; What you've described and my implementation work fine so something else is going awry in your situation.
Set up
I built out a simple SSIS package. Execute Package Task -> Script task (to dump variable values to console) -> A Sequence Container that has your disable logic on it => Another script task (to dump to console if the disable works/does not work)
I have your original variable, User::FileCount of type String and added User::HasFile of type Boolean with an expression of Len(Trim(#[User::FileCount])) > 0 to test your logic.
SO_71067359.bat
I created a batch file that will serve as your C# app. When needed, I'll remove line 2 so no output is generated. Contents are
#echo off
echo 1
Results
C:\ssisdata>SO_71067359.bat
1
EPT Download File
Execute Package Task. Working directory set to the location of my batch file. Executable is the batch script. StandardOutputVariable is User::FileCount
SCR Echo Back
This is my standard echo back script and I'll add User::FileCount and User::HasFile. When the package runs, this will dump the values to the Console output which is something I can copy and paste unlike the pretty Results tab most are familiar with
Content in case my blog goes away is
bool fireAgain = false;
string message = "{0}::{1} : {2}";
foreach (var item in Dts.Variables)
{
Dts.Events.FireInformation(0, "SCR Echo Back", string.Format(message, item.Namespace, item.Name, item.Value), "", 0, ref fireAgain);
}
SEQC Simulate Other
This is a sequence container that has your Disable logic directly added to it. Yes, I have the variable but since you have the inversion of the results, I didn't want to sully the test.
Yes it ran
Inside the sequence container is a copy/paste of the SCR Echo Back except I renamed it to "Yes it ran" and specified the only variable is System::TaskName If the task runs, the console will print the name of the task.
Test round 1
The output of the batch script will be "1" so we expect to see the Inner task fire. Let's check the console
SSIS package "C:\Users\bfellows\source\repos\SO_Trash\SO_Trash\SO_71067359.dtsx" starting.
Information: 0x0 at SCR Echo Back, SCR Echo Back: User::FileCount->1
Information: 0x0 at SCR Echo Back, SCR Echo Back: User::HasFiles->True
Information: 0x0 at Yes it ran, SCR Echo Back: System::TaskName->Yes it ran
SSIS package "C:\Users\bfellows\source\repos\SO_Trash\SO_Trash\SO_71067359.dtsx" finished: Success.
The program '[2268] DtsDebugHost.exe: DTS' has exited with code 0 (0x0).
Test Round 2
Clearing line 2 of the bat file
SSIS package "C:\Users\bfellows\source\repos\SO_Trash\SO_Trash\SO_71067359.dtsx" starting.
Information: 0x0 at SCR Echo Back, SCR Echo Back: User::FileCount->
Information: 0x0 at SCR Echo Back, SCR Echo Back: User::HasFiles->False
SSIS package "C:\Users\bfellows\source\repos\SO_Trash\SO_Trash\SO_71067359.dtsx" finished: Success.
The program '[75252] DtsDebugHost.exe: DTS' has exited with code 0 (0x0).

ArgumentException : "the path is not of a legal form" when using the CallerFilePath attribute in a script

Context
I am trying to run a C# script (.csx file) programmatically from a program I will call ScriptRunner.exe here and that I wrote myself (because csi.exe doesn't output what I want).ScriptRunner.exe is a simple console application and its most interesting feature is to have the following line :
var state = await CSharpScript.RunAsync<int>(script, referencesAndUsings, globalArgs);
ScriptRunner.exe works great ! However...
Problem
The moment my script contains the following line :
static string GetCurrentFileName([System.Runtime.CompilerServices.CallerFilePath] string fileName = null) { return fileName; }
and in particular [System.Runtime.CompilerServices.CallerFilePath], I get an ArgumentException : "the path is not of a legal form" ; note that the latter doesn't appear if I use the same line from a C# Interactive through a #load command - which correctly shows the path of my .csx file.
Investigated elements until now
The stacktrace shows at System.IO.Path.LegacyNormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)
I checked what seems to be the implementation
I checked by hand the path of my csx file ; there's no invalid path characters in the path to my csx, and no wildcards in it either.
I checked there was no reference issue with mscorlib
Maybe something is missing in the ScriptOptions (referencesAndUsings in my first sample code), I looked at it but... I don't seem to understand everything well enough
The way I created my ScriptOptions ("referencesAndUsings") looks like the following:
var myOptions = ScriptOptions.Default;
myOptions.AddReferences(new List<string>() { ... });
myOptions.AddImports(new List<string>() { ... });
This is the documentation for the CallerFilePath attribute
This is the documentaiont for the concept of Caller Information
What really saddens me is that it works in C# Interactive.
Question
Does anyone know why it wouldn't want to work when interpreted by my ScriptRunner.exe ; and how to make it work ?

In
var state = await CSharpScript.RunAsync<int>(script, referencesAndUsings, globalArgs);
script is the script itself (the contents of the csx file). As such it has no path. CSharpScript knows nothing about the path of your csx file. When you call GetCurrentFileName from within the script, there is no path information.
You need to specify a FilePath in ScriptOptions using WithFilePath(csxFilePath)

Build visual studio solution using msbuild from c# code

I want to build my solution file from other c# code using msbuid I have tried
var msbuild_path = #"C:\Windows\Microsoft.NET\Framework\v4.0.30319\msbuild.exe";
var solution_path = #"D:\Sumit\WorkingCopy\Final\Final.sln";
Process.Start(msbuild_path + " " + solution_path);
but this one throws an error Please help me out!!

According to https://msdn.microsoft.com/en-us/library/h6ak8zt5(v=vs.110).aspx , the Process.Start method takes two arguments:
public static Process Start(string fileName, string arguments)
So you should change your code to
Process.Start(msbuild_path, solution_path);
What you were doing before was actually trying to run a file named "C:\Windows\Microsoft.NET\Framework\v4.0.30319\msbuild.exe(space)D:\Sumit\WorkingCopy\Final\Final.sln", but no such file exists with that name. The msbuild.exe may exist, but "msbuild.exe D:\Sumit...\Final.sln" is not the filename you meant to pass as the command filename. Also, the argument string was empty, so the system assumed you did not want to pass any arguments to "msbuild.exe D:\Sumit...\Final.sln". But the error message was because the two filenames were mashed into one filename.
Windows allows filenames to contain embedded spaces, which frequently causes problems in dealing with command-line arguments.

File existence check after a folder rename returns an incorrect value on UNC share

On a Windows 7 (or server) box, we have a folder on a UNC share (cross machine UNC, not localhost). We rename that folder, and then check for the existence of a file at the new folder location. Even though it exists, it takes almost 5 seconds for File.Exists to return true on it.
Full repro can be found on https://github.com/davidebbo/NpmFolderRenameIssue. Here is the core code:
// This file doesn't exist yet
// Note that the presence of this existence check is what triggers the bug below!!
Console.WriteLine("Exists (should be false): " + File.Exists("test/test2/myfile"));
// Create a directory, with a file in it
Directory.CreateDirectory("test/subdir/test");
File.WriteAllText("test/subdir/test/myfile", "Hello");
// Rename the directory
Directory.Move("test/subdir/test", "test/test2");
var start = DateTime.UtcNow;
// List the files at the new location. Here, our file shows up fine
foreach (var path in Directory.GetFiles("test/test2"))
{
Console.WriteLine(path);
}
for (; ; )
{
// Now do a simple existence test. It should also be true, but when
// running on a (cross machine) UNC share, it takes almost 5 seconds to become true!
if (File.Exists("test/test2/myfile")) break;
Console.WriteLine("After {0} milliseconds, test/test2/myfile doesn't show as existing",
(DateTime.UtcNow - start).TotalMilliseconds);
Thread.Sleep(100);
}
Console.WriteLine("After {0} milliseconds, test/test2/myfile correctly shows as existing!",
(DateTime.UtcNow - start).TotalMilliseconds);
So it seems like the initial existence check causes the existence value to be cached, causing this bogus behavior.
Questions: what is the explanation for this? What's the best way to avoid it?
NOTE: this issue initially arose when using npm (Node Package Manager) on Windows. The code I have here is a C# port of the repro. See https://github.com/isaacs/npm/issues/2230 for the original Node/npm issue. The goal is to find a way to address it.

David,
The redirector implements a negative "File Not Found" cache which prevents a client from flooding a server with file not found requests. The default cache time is 5 seconds but you can modify the FileNotFoundCacheLifetime registry value to control the cache or disable it by setting this value to 0.
Details: http://technet.microsoft.com/en-us/library/ff686200(v=WS.10).aspx

There are multiple levels of caching in network code. This could slow down the time the file existence finally shows up.
A solution would be not to use file shares but create a simple client/server architecture where the server returns the file existence from the local file system. That should really speed up item detection times.
My guess would be that if you tried to open the file even if File.Exists says it doesn't exist yet it should be opened correctly so you can use the server existence information. If that won't work you can simply add a download option to the server/client architecture.

Once I knew about the "File Not Found" cache, I was able to get around the problem by using a FileInfo object, which implements a Refresh() method. Your code could do this instead:
FileInfo testFile = new FileInfo("test/test2/myfile");
Console.WriteLine("Exists (should be false): " + testFile .Exists);
Directory.Move("test/subdir/test", "test/test2");
testFile.Refresh();
// the FileInfo object should now be refreshed, and a second call to Exists will return a valid value
if (testFile.Exists)
{
...
}

Changing environment variables of the calling process

This one seems trivial but the answer has eluded me for a few days now.
I have a Windows batch file, that calls a C# program to do an extra verification that cannot be done in a batch file. After the verification is complete I need to return a status and a string back to the calling shell.
Now the return value is trivial and my C# console app simply sets a return value (exit code if you will). And I thought the string will also be a piece of cake. I attempted to define a new shell variable using the:
Environment.SetEnvironmentVariable("ERR", "Some text");
This call should (and does) define a shell variable within the current process - that is the very C# process that created the variable. The value is lost as soon as the C# app terminates and the shell that created the C# app knows nothing about the variable. So... A call with no particular use... At all... Unless perhaps if I created a child process from the C3 app, perhaps it would inherit my variables.
The EnvironmentVariableTarget.Machine and EnvironmentVariableTarget.User targets for the SetEnvironmentVariable call don't solve the problem either, as only a newly created process will get these new values from the registry.
So the only working solution I can think of is:
write to stdout
write to a file
encode extra meaning into the return value
The first two are a bit ugly and the last one has its limitations and problems.
Any other ideas (how to set a shell variable in the parent process)? Maybe such shell variable modifications are a security concern (think PATH)...
Thank-you for your time.

I had the same problem as Ryan and the only thing that came to my mind as a work-around was to write a batch in error out to set the variable and to call it from the batch.
ConsoleApplication1.exe:
'put some sensible code here
'put result in variable myResult
Dim myResult As String = Guid.NewGuid().ToString("D").ToUpperInvariant()
Console.WriteLine("Normal output from the consonle app")
Console.Error.WriteLine("#ECHO OFF")
Console.Error.WriteLine("SET zzzResult={0}", myResult)
Test.cmd (the calling batch):
#ECHO OFF
:Jump to folder of batch file
PUSHD %~d0%~p0
:Define a temp file
SET zzzTempFile=%TEMP%\TMP%Random%.CMD
:Call .NET console app
ConsoleApplication1.exe 2>%zzzTempFile%
:Call the generated batch file
CALL %zzzTempFile%
:Clean up temp file
DEL %zzzTempFile%
:Clean up variable
SET zzzTempFile=
:Do something with the result
ECHO Yeah, we finally got it!
ECHO:
ECHO The value is "%zzzResult%".
ECHO:
:Clean up result variable
SET zzzResult=
:Go back to original folder
POPD
That should do the trick. And yes, I do know this is an old post and Ryan is solving other issues by now, but there might be still somebody else out there having the same problem...

What you are asking is to be able to arbitrarily write to the memory space of a running process. For good reason, this is not possible without SeDebugPrivilege.
Any of the three solutions you list will work. Stdout is the standard way to communicate with a batch script.
By the way, you're writing a Windows batch file. I'm pretty sure the ship has already sailed on "a bit ugly".

If you want to put a value of some output into a variable in the batch you can use the following construct:
FOR /F "usebackq tokens=4 delims=\[\] " %i IN (`ver`) DO set VERSION=%i
ECHO %VERSION%
Output on my OS:
6.1.7601
'usebackq' means we are using back quotes which gives the ability to use a fileset in the command quoted with double quotes. You may not need this. 'tokens' means the index in the resulting string array to select (it can be a range M-N). If you need to skip lines use 'skip=X'). 'delims' are the string separators to use (like string-Split() in .Net).
You will put your console app instead of 'ver' and adapt the delimiters and tokens to match your specific output. If you have more variables to fill you will need to make the if a bit more complex but that should make a good start.

My BAT is a bit rusty, but I think it's possible to retrieve the 'exit' code from processes you've run externally, perhaps via %ERRORLEVEL%. If that's the case, make sure to exit your program via
Environment.Exit(123); // where 123 = error code
You can't add any messages, so you'll have to do that in the .bat file.
If this isn't the case, stdout is probably the best way.

After stumbling on this myself as well recently, I came up with this approach. What I did is run the bat file using the Process class, i.e.
// Spawn your process as you normally would... but also have it dump the environment varaibles
Process process = new Process();
process.StartInfo.FileName = mybatfile.bat;
process.StartInfo.Arguments = #"&&set>>envirodump.txt";
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = false;
process.Start();
string output = process.StandardOutput.ReadToEnd();
process.WaitForExit();
// Read the environment variable lines into a string array
string[] envirolines = File.ReadAllLines("envirodump.txt");
File.Delete("envirodump.txt");
// Now simply set the environment variables in the parent process
foreach(string line in a)
{
string var = line.Split('=')[0];
string val = line.Split('=')[1];
Environment.SetEnvironmentVariable(var, val);
}
This seems to have worked for me. It's not the cleanest approach, but will work in a bind. :)