Develop Reference

ScriptBuffer throws NullReferenceException when called from an Event as part of a loop - SSIS Script Component [duplicate]

This question already has answers here:
What is a NullReferenceException, and how do I fix it?
(27 answers)
Closed 1 year ago.
I have an SSIS package which calls a Data Flow Task as part of a loop which iterates different end-point addresses (out of scope).
The Data Flow Task has a source Script Component responsible for calling a REST API and creating a row for each result.
There are 3 output buffers;
1. actual data row
2. error row
3. monitoring
The monitoring buffer used for telemetry and is populated through an event (EventHander) that is fired every time the API makes a request.
During the first iteration of the ForEach int the Control Flow loop, everything runs as expected, all the buffers produce the correct rows.
However, during the next iterations, the monitoring buffer which is populated within the event throws;
System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.HandleUserException(Exception e)
at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.PrimeOutput(Int32 outputs, Int32[] outputIDs, PipelineBuffer[] buffers)
at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostPrimeOutput(IDTSManagedComponentWrapper100 wrapper, Int32 outputs, Int32[] outputIDs, IDTSBuffer100[] buffers, IntPtr ppBufferWirePacket)
I don't understand why the MonitoringBuffer is not initialised in the proceeding iterations.
The exception occurs while calling MonitoringBuffer.AddRow();.
Here's the whole Script Component simplified for readability:
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
private string ClientCode { get { return Variables.ErplyClientCode; } }
private string Username { get { return Variables.ErplyUsername; } }
private string Password { get { return Variables.ErplyPassword; } }
private bool IsTest { get { return Variables.IsTest; } }
private int ErplyRecordsPerPage { get { return Variables.ErplyRecordsPerPage; } }
private string ErplyDebugOutputPath { get { return Variables.ErplyDebugOutputPath; } }
private DateTime ChangeSince { get { return Variables.ChangeSince; } }
private int records { get; set; }
private int errors { get; set; }
private string rawFolder { get; set; }
public override void PreExecute()
{
base.PreExecute();
}
public override void PostExecute()
{
base.PostExecute();
}
public override void CreateNewOutputRows()
{
ErplyAPI.OnPreRequestEvent += new EventHandler<EAPIEvent>(ErplyAPI_OnPreRequestEvent);
var staff = ErplyAPI.getStaff(ClientCode, Username, Password, ChangeSince, ErplyRecordsPerPage, IsTest);
foreach (var p in staff.List)
{
try
{
if (!p.IsError)
{
EmployeeBuffer.AddRow();
EmployeeBuffer.employeeID = p.employeeID;
}
else
{
ErrorBuffer.AddRow();
ErrorBuffer.employeeID = p.employeeID;
ErrorBuffer.Error = p.Error.Message.Trim() + "\n" + p.Error.StackTrace;
errors++;
}
records++;
}
catch (Exception ex)
{
this.ComponentMetaData.FireWarning(0, "Script", ex.Message + "\n" + ex.StackTrace, string.Empty, 0);
}
}
EmployeeBuffer.SetEndOfRowset();
ErrorBuffer.SetEndOfRowset();
}
private void ErplyAPI_OnPreRequestEvent(object sender, EAPIEvent e)
{
var request = string.Empty;
var sessionKey = string.Empty;
bool fireAgain = true;
if (e == null)
{
ComponentMetaData.FireWarning(0, "SC_ERPLY_API", string.Format("EAPIEvent is NULL in ErplyAPI_OnPreRequestEvent. Amonit did not log the Erply request."), string.Empty, 0);
return;
}
if (e.eAPI == null)
{
ComponentMetaData.FireWarning(0, "SC_ERPLY_API", string.Format("EAPIEvent.eAPI is NULL in ErplyAPI_OnPreRequestEvent. Amonit did not log the Erply request."), string.Empty, 0);
return;
}
try
{
if (e.Parameters != null && e.Parameters.ContainsKey("request"))
request = e.Parameters["request"].ToString();
if (request != "verifyUser" && e.Parameters != null && e.Parameters.ContainsKey("sessionKey"))
sessionKey = e.Parameters["sessionKey"].ToString();
}
catch (Exception ex)
{
ComponentMetaData.FireWarning(0, "SC_ERPLY_API", string.Format("Error occurred assigning variables from EAPIEvent parameters in ErplyAPI_OnPreRequestEvent. {0} {1}", ex.Message, ex.StackTrace), string.Empty, 0);
}
try
{
MonitoringBuffer.AddRow(); // Exception occurs here
MonitoringBuffer.Request = ResizeString(request, 255);
MonitoringBuffer.SessionKey = ResizeString(sessionKey, 128);
}
catch (Exception ex)
{
var message = string.Format("Error occurred outputting Erply request in ErplyAPI_OnPreRequestEvent. {0} {1}", ex.Message, ex.StackTrace);
MonitoringBuffer.ErrorMessage = ResizeString(message, 8000);
ComponentMetaData.FireWarning(0, "SC_ERPLY_API", message, string.Empty, 0);
}
finally
{
MonitoringBuffer.EndOfRowset();
}
}
}

I sorted the problem out.
The exception was being raised when the variable dispenser was being accessed from the Event. For some reason the GetValueWithContext(ScriptComponent.EvaluatorContext) is being dropped during the second call. Why this happens is beyond me.
The solution is simple, assign the variables from the variables dispenser to a local property or variable in the OnPreExecute function.
It's also good practice to not call the variable dispenser in the CreateNewOutputRows as it cause variable locking.

I ran into this issue too, but my solution was a little different -- moving the variable assignments into PreExecute() didn't help.
Instead, what I'd done is that I wanted to parse three different files, and read each of them with a Script Component. Their columns were kinda similar, so I created one Data Flow task, made sure it worked, then copied it and modified each copy to reflect the differences in the files. Running each individual Data Flow task was successful, but when I tried to run two of them, one after the other in a loop, I got a NullReferenceException from HostPrimeOutput() after calling the OutputBuffer.AddRow() method in my Script Component.
It turns out that when I copied each Data Flow task, the Script Components all kept the same namespace, and I guess it doesn't like that. So, I created brand new Script Components, set up all the output columns again (ugh!), copied the body of the script over, and it's happy.

Can a method within a class inherit a subroutine that executes at the start and end of the method?

I have a logger that I am adding to my project. Right now for every method I'm going to have to write Logger.DebugLog("Starting Method") at the start of each method and Logger.DebugLog("Completed Method")
this logger - when Debug enabled - allows me to track exactly what methods were called on this run so that if there is an issue I can see how far it got before breaking making it easy to debug. Assume that the method name and line are being captured - my goal here is I do not want to add those two lines on every one of the +100 public or private methods
namespace myProject
{
public class myClass
{
public bool MyPublicMethod(string Message = "someRandomMessage")
{
try
{
myPrivateMethod(1);
writeToLog(Message);
return true;
}
catch(){
return false;
}
}
private bool myPrivateMethod(int passedNumber)
{
try
{
writeToLog(passedNumber);
return true;
}
catch(){
return false;
}
}
}
}
my log file should look like this:
04:00:00 - Starting Method:MyPublicMethod
04:00:00 - Starting Method:myPrivateMethod
04:00:01 - 1
04:00:01 - Completed Method:myPrivateMethod
04:00:02 - someRandomMessage
04:00:02 - Completed Method:MyPublicMethod
What I am forced to do now which looks cluttered is:
namespace myProject
{
public class myClass
{
public bool MyPublicMethod(string Message = "someRandomMessage")
{
try
{
writeToLog("Starting Method");
myPrivateMethod(1);
writeToLog(Message);
writeToLog("Completed Method");
return true;
}
catch(){
return false;
}
}
private bool myPrivateMethod(int passedNumber)
{
try
{
writeToLog("Starting Method");
writeToLog(passedNumber);
writeToLog("Completed Method");
return true;
}
catch(){
return false;
}
}
}
}
is this even possible in .NET or do I have to explicitly list that logging method if I want to use it?
4/6/18 Edit: It is possible - see AOP. Heres a pretty decent article on it http://www.dotnetcurry.com/patterns-practices/1305/aspect-oriented-programming-aop-csharp-using-solid
Here is a good summary of exactly what I was looking for:
Consider the following code:
public class DocumentSource : IDocumentSource
{
//..
public Document[] GetDocuments(string format)
{
try
{
using (var context = CreateEFContext())
{
var documents =
context
.Documents
.Where(c => c.Name.EndsWith("." + format))
.ToArray();
logger.LogSuccess(
"Obtained " + documents.Length + " documents of type " + format +
Environment.NewLine +
"Connection String: " + connectionString);
return documents;
}
}
catch (Exception ex)
{
logger.LogError(
"Error obtaining documents of type " + format +
Environment.NewLine +
"Connection String: " + connectionString, ex);
throw;
}
}
//..
}
Here is how the method would look like without logging:
public Document[] GetDocuments(string format)
{
using (var context = CreateEFContext())
{
return
context
.Documents
.Where(c => c.Name.EndsWith("." + format))
.ToArray();
}
}
Clearly, the logging code has made the original method less readable. It has tangled the real method code with logging code.
This is also a violation of the Single Responsibility Principle.
Also, we expect to find the same logging pattern in many methods all over the code base. Basically, we expect to find the following pattern:
try
{
//Do something here
logger.LogSuccess(…
//..
}
catch (Exception ex)
{
logger.LogError(…
throw;
}

There are some Fody Add-ins that will allow you to add this kind of code into your compiled output at compile-time rather than having to write it yourself.
For example, MethodDecorator allows you to define a specific attribute, and any method you decorate with that attribute will call specific methods prior to entering and leaving the method.
I should note that for a project of any reasonable size, logging the entry and exit for every single method is going to produce more log messages than anyone can reasonably expect to read. I'd suggest you be judicious in which methods add log messages, and what information you include in those log messages.
The vast majority of the time a more useful strategy is to use guard statements to test your assumptions along the way, throwing an exception the moment anything is out of place, and then wrapping exceptions with more useful information (via InnerException) as they go up the call chain, then finally logging the results of those exceptions at the top level of your application. That way, you only produce log messages when something appears to go in a way you don't expect, and the log message in that case has all the information you're likely to need.

I'm not sure this is exactly what you're looking for, but it might point you in the right direction. For demonstration purposes I'm logging to the Console, but you could log to a file instead.
You could create a method that takes in another method to execute, which executes the function and returns the value, and which wraps the beginning and ending of the method call with writes to your log file:
// For methods that return a value
private static TResult LogMethod<TResult>(string displayName, Func<TResult> method)
{
Console.WriteLine($"{DateTime.Now} - Starting method: {displayName}");
TResult result = method();
Console.WriteLine($"{DateTime.Now} - Completed method: {displayName}");
return result;
}
// For void methods
private static void LogMethod(string displayName, Action method)
{
Console.WriteLine($"{DateTime.Now} - Starting method: {displayName}");
method();
Console.WriteLine($"{DateTime.Now} - Completed method: {displayName}");
}
As an example of how to use this, let's say we have the following methods:
private static int GetNumberFromUser(string prompt)
{
int result;
do
{
Console.Write(prompt);
} while (!int.TryParse(Console.ReadLine(), out result));
return result;
}
private static int AddNumbers(int first, int second)
{
return first + second;
}
private static void Tell(string message)
{
Console.WriteLine(message);
}
Now, instead of calling these methods directly, we can call them through our LogMethod function:
private static void Main()
{
var firstNumber = LogMethod("GetNumber",
() => GetNumberFromUser("Enter first number: "));
var secondNumber = LogMethod("GetNumber",
() => GetNumberFromUser("Enter second number: "));
var result = LogMethod("AddNumber",
() => AddNumbers(firstNumber, secondNumber));
LogMethod("Tell", () => Tell($"{firstNumber} + {secondNumber} = {result}"));
GetKeyFromUser("\nDone!! Press any key to exit...");
}
Output

Can you programmatically call a method as if it were a class field?

I am writing an automated test suite for my program, and have been looking for ways to simplify and re factor a lot of the code.
Currently, I have a number of try/catch blocks (because I want to be able to move from one test to the next, even in the case of failure) that can log information otherwise. It looks a bit like this:
try
{
PerformTest1();
}
catch(Exception e)
{
file.WriteLine("Unable to perform test number 1.");
file.WriteLine("Error: " + e.Message);
file.WriteLine("StackTrace: " + e.StackTrace);
}
try
{
PerformTest2();
}
catch(Exception e)
{
file.WriteLine("Unable to perform test number 2.");
file.WriteLine("Error: " + e.Message);
file.WriteLine("StackTrace: " + e.StackTrace);
}
And so on. I was wondering if I could make an array of these tests, so I could use a loop. Something like:
foreach(Test t in testsArray)
{
try
{
t.RunTest();
}
catch(Exception e)
{
file.WriteLine(t.failDescription);
file.WriteLine("Error: " + e.Message);
file.WriteLine("StackTrace: " + e.StackTrace);
}
}
How can I execute this, but without making each test it's own class (with a RunTest() method)? I was thinking about making one test class, that has the necessary fields, one of them being a method. Then I could create a test object, and call that objects method. Is such a thing even possible? Or would I have to make multiple test classes?

You would need a Test interface with a runTest() method, but you can implement the logic with anonymous classes or lambda expressions (if you are using Java 8), which would save you the need to create an explicit class for each test.
Example :
Test test1 = new Test () {
public void runTest () {
// logic of first test here
}
};
Actually, you don't need to create your own interface. Use the existing Runnable interface with its run() method.
Runnable test1 = new Runnable () {
public void run () {
// logic of first test here
}
};
or with a lambda expression :
Runnable test1 = () -> {
// logic of first test here
};
Then you can add each of these Runnables to a list (or array) and run them in a loop, as you wanted :
for (Runnable test in testsArray)
{
try {
test.run();
}
catch(Exception e)
{
...
}
}

You could use some Lambda tricks:
var actions = new List<Action>();
actions.Add(PerformTest1); // requires a void PerformTest1() method
actions.Add(PerformTest2);
//... and so on...
and then finally fire all actions:
foreach(var action in actions)
{
action();
}

Have you looked into the available testing frameworks for .Net/ Java? I believe Nunit (or JUnit) provides much of the functionality you are looking for, including writing out the stack trace and exception details in the event of a failure.
The also encapsulate multiple tests within a test fixture, so will allow you to still execute subsequent tests after a failure without bleeding data or state between your individual tests.

public interface Testable {
void doTest();
}
public class Test1 implements Testable {
// implement doTest()
}
private static Testable[] testables = new Testabable[] {
new Test1()
};
// now iterate your testables and call doTest()
Ok, seems to be old school, but it's the first thing, one should learn about interfaces.

In C# can you return a default int value from main?

I want to return a default int value from main.
Consider the following:
using System;
class Class1
{
static int Main(string[] args)
{
int intReturnCode = 1;
int intRandNumber;
Random myRandom = new Random();
intRandNumber = myRandom.Next(0,2);
if(intRandNumber ==1)
{
throw new Exception("ErrorError");
}
return intReturnCode;
}
}
When the exception is reached I don't get to set the returncode.
Is it possible to have a default return code inside of main?
Clarification: I have a program that is throwing Unhandled Exceptions. I have the application inside a try catch, however some errors (probably out of memory, stackoverflow etc) are still bubling up and causing my application to fail in production.
To fix this I've added code to capture unhandled exceptions.
This was added to main:
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(OnUnhandledException);
And now I have this method that is reached when an unhandled exception occurs.
public static void OnUnhandledException(object sender, UnhandledExceptionEventArgs e)
{
//here's how you get the exception
Exception exception = (Exception)e.ExceptionObject;
//bail out in a tidy way and perform your logging
}
The prblem is that I'm no longer in Main and I want to exit with a non-zero exit code.

An unhandled exception is implementation defined behaviour. Anything can happen; the CLR can decide to set the return code of the process as it sees fit, it can start a debugger, it can do whatever it wants. You cannot rely on any behaviour of any program that contains an unhandled exception.
If you want to have predictable behaviour, like determining what the return code is when the process ends, then there must be a total of zero unhandled exceptions.
If you have a third party component that is throwing unhandled out of memory exceptions then your best bet is: fix the bug in that component. If you can't do that then isolate the component into its own process or its own appdomain.

The question is really why you are throwing an exception in main instead of providing a return code that indicates an error? Instead of what you're doing, my code would look as follows:
static int Main(string[] args)
{
int intRandNumber;
try
{
Random myRandom = new Random();
intRandNumber = myRandom.Next(0,2);
if(intRandNumber ==1)
{
Console.WriteLine("Got invalid random number!");
return 0;
}
}
catch (Exception exp)
{
Console.WriteLine("Strange ... an error occurred! " + exp.ToString());
return -1;
}
return 1;
}
As a rule of thumb you should never throw exceptions to control program flow. If you can, handle conditions like oops, I got the wrong number without throwing an exception.

Throwing an exception in the main thread ends execution without reaching the return: that's when you get the "Console application has stopped working, would you like to debug?" dialog from the operating system. The Main cannot return anything under these conditions, because there is nothing to return.
If you would like to return something when you get an exception, code your program like this:
// Put your implementation into a private method
private static int DoWork(string[] args) {
... // Do actual work, and throw exceptions if you need to
return intReturnCode;
}
// The Main method calls the implementation, and returns the value
// returned from the implementation if everything goes well.
// If an exception is thrown, however, Main could return another value
// of its choice.
public static int Main(string[] args) {
try {
return DoWork(args);
} catch {
return myDefaultReturnCode;
}
}

catch your exception and set return statement in finally block
using System;
class Class1
{
static int Main(string[] args)
{
try
{
int intReturnCode = 1;
int intRandNumber;
Random myRandom = new Random();
intRandNumber = myRandom.Next(0,2);
if(intRandNumber ==1)
{
throw new Exception("ErrorError");
}
}
catch(Exception e)
{
// ...
}
finally
{
return intReturnCode;
}
}
}

Exception handling in threads

Recently i have attended an interview . A code snippet is given to me.I know,the interviewer took it from albhari's threading sample.
public static void Main()
{
try
{
new Thread (Go).Start();
}
catch (Exception ex)
{
// We'll never get here!
Console.WriteLine ("Exception!");
}
}
static void Go() { throw null; }
The modification of the above code as
public static void Main()
{
new Thread (Go).Start();
}
static void Go()
{
try
{
...
throw null; // this exception will get caught below
...
}
catch (Exception ex)
{
Typically log the exception, and/or signal another thread
that we've come unstuck
...
}
}
would be the good candidate to handle the exception.
I have been asked, "Except the above trail what are the other alternatives would fit as good solution?. It was hard to find the alternative,so i raise it here to gather your suggestion.

Exception thrown in a thread normally couldn't be caught in another thread.
You'd better to catch it in function Go and pass it to main thread explicitly.
However, if you just want to log all unhandled messages from all threads, you may use AppDomain.UnhandledException event or equivalent events at Application class if you are developing WinForms or WPF app.

what are the other alternatives would fit as good solution?.
Solution to what? What problem are you trying to solve?
If you use BackgroundWorker, as opposed to Thread, it has an RunWorkerCompleted event, and within that you can check the RunWorkerCompletedEventArgs param for the Error property. This generally is used in WinForms or WPF apps, because there is good support for BackgroundWorker in the Visual Studio designer.
You could also define a delegate for Go(), and call BeginInvoke() on it. Of course you need the EndInvoke() too.
Also, it's generally not a good idea to start up random threads. ThreadPool.QueueUserWorkItem, BackgroundWorker, or asynch delegates all use the ThreadPool, and are recommended.

There are alternatives listed on Joe Albahari's website:
http://www.albahari.com/threading/#_Exception_Handling
"There are, however, some cases where you don’t need to handle exceptions on a worker thread, because the .NET Framework does it for you. These are covered in upcoming sections, and are:
-Asynchronous delegates
-BackgroundWorker
-The Task Parallel Library (conditions apply)"

You can use the AppDomain.UnhandledException event

I think this is the easiest way is:
BackgroundWorker bw = new BackgroundWorker();
bw.DoWork += new DoWorkEventHandler((object sender2, DoWorkEventArgs e2) =>
{
throw new Exception("something bad");
e2.Result = 1 + 1;
});
bw.RunWorkerCompleted += new RunWorkerCompletedEventHandler((object sender2, RunWorkerCompletedEventArgs e2) =>
{
if (e2.Error != null)
{
Console.WriteLine("Error: " + e2.Error.Message);
}
});
bw.RunWorkerAsync();
but there is another way that some might prefer if you want to synchronize the thread (perhaps this is on a thread other than the GUI thread):
private class FileCopier
{
public bool failed = false;
public Exception ex = null;
public string localPath;
public string dstPath;
public FileCopier(string localPath, string dstPath)
{
this.localPath = localPath;
this.dstPath = dstPath;
}
public void Copy()
{
try{
throw new Exception("bad path");
}catch(Exception ex2)
{
ex = ex2;
failed = true;
}
}
}
public static void Main()
{
FileCopier fc = new FileCopier("some path", "some path");
Thread t = new Thread(fc.Copy);
t.Start();
t.Join();
if (fc.failed)
Console.WriteLine(fc.ex.Message);
}
Note that the second example would make more sense if you have several threads and you loop through them and join all...but I kept the example simple.
the 3rd pattern would be using Task Factory which is cleaner:
private static test(){
List<Task<float>> tasks = new List<Task<float>>();
for (float i = -3.0f; i <= 3.0f; i+=1.0f)
{
float num = i;
Console.WriteLine("sent " + i);
Task<float> task = Task.Factory.StartNew<float>(() => Div(5.0f, num));
tasks.Add(task);
}
foreach(Task<float> t in tasks)
{
try
{
t.Wait();
if (t.IsFaulted)
{
Console.WriteLine("Something went wrong: " + t.Exception.Message);
}
else
{
Console.WriteLine("result: " + t.Result);
}
}catch(Exception ex)
{
Console.WriteLine("Error: " + ex.Message);
}
}
}
private static float Div(float a, float b)
{
Console.WriteLine("got " + b);
if (b == 0) throw new Exception("Divide by zero");
return a / b;
}