Develop Reference

ScriptBuffer throws NullReferenceException when called from an Event as part of a loop - SSIS Script Component [duplicate]

This question already has answers here:
What is a NullReferenceException, and how do I fix it?
(27 answers)
Closed 1 year ago.
I have an SSIS package which calls a Data Flow Task as part of a loop which iterates different end-point addresses (out of scope).
The Data Flow Task has a source Script Component responsible for calling a REST API and creating a row for each result.
There are 3 output buffers;
1. actual data row
2. error row
3. monitoring
The monitoring buffer used for telemetry and is populated through an event (EventHander) that is fired every time the API makes a request.
During the first iteration of the ForEach int the Control Flow loop, everything runs as expected, all the buffers produce the correct rows.
However, during the next iterations, the monitoring buffer which is populated within the event throws;
System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.HandleUserException(Exception e)
at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.PrimeOutput(Int32 outputs, Int32[] outputIDs, PipelineBuffer[] buffers)
at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostPrimeOutput(IDTSManagedComponentWrapper100 wrapper, Int32 outputs, Int32[] outputIDs, IDTSBuffer100[] buffers, IntPtr ppBufferWirePacket)
I don't understand why the MonitoringBuffer is not initialised in the proceeding iterations.
The exception occurs while calling MonitoringBuffer.AddRow();.
Here's the whole Script Component simplified for readability:
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
private string ClientCode { get { return Variables.ErplyClientCode; } }
private string Username { get { return Variables.ErplyUsername; } }
private string Password { get { return Variables.ErplyPassword; } }
private bool IsTest { get { return Variables.IsTest; } }
private int ErplyRecordsPerPage { get { return Variables.ErplyRecordsPerPage; } }
private string ErplyDebugOutputPath { get { return Variables.ErplyDebugOutputPath; } }
private DateTime ChangeSince { get { return Variables.ChangeSince; } }
private int records { get; set; }
private int errors { get; set; }
private string rawFolder { get; set; }
public override void PreExecute()
{
base.PreExecute();
}
public override void PostExecute()
{
base.PostExecute();
}
public override void CreateNewOutputRows()
{
ErplyAPI.OnPreRequestEvent += new EventHandler<EAPIEvent>(ErplyAPI_OnPreRequestEvent);
var staff = ErplyAPI.getStaff(ClientCode, Username, Password, ChangeSince, ErplyRecordsPerPage, IsTest);
foreach (var p in staff.List)
{
try
{
if (!p.IsError)
{
EmployeeBuffer.AddRow();
EmployeeBuffer.employeeID = p.employeeID;
}
else
{
ErrorBuffer.AddRow();
ErrorBuffer.employeeID = p.employeeID;
ErrorBuffer.Error = p.Error.Message.Trim() + "\n" + p.Error.StackTrace;
errors++;
}
records++;
}
catch (Exception ex)
{
this.ComponentMetaData.FireWarning(0, "Script", ex.Message + "\n" + ex.StackTrace, string.Empty, 0);
}
}
EmployeeBuffer.SetEndOfRowset();
ErrorBuffer.SetEndOfRowset();
}
private void ErplyAPI_OnPreRequestEvent(object sender, EAPIEvent e)
{
var request = string.Empty;
var sessionKey = string.Empty;
bool fireAgain = true;
if (e == null)
{
ComponentMetaData.FireWarning(0, "SC_ERPLY_API", string.Format("EAPIEvent is NULL in ErplyAPI_OnPreRequestEvent. Amonit did not log the Erply request."), string.Empty, 0);
return;
}
if (e.eAPI == null)
{
ComponentMetaData.FireWarning(0, "SC_ERPLY_API", string.Format("EAPIEvent.eAPI is NULL in ErplyAPI_OnPreRequestEvent. Amonit did not log the Erply request."), string.Empty, 0);
return;
}
try
{
if (e.Parameters != null && e.Parameters.ContainsKey("request"))
request = e.Parameters["request"].ToString();
if (request != "verifyUser" && e.Parameters != null && e.Parameters.ContainsKey("sessionKey"))
sessionKey = e.Parameters["sessionKey"].ToString();
}
catch (Exception ex)
{
ComponentMetaData.FireWarning(0, "SC_ERPLY_API", string.Format("Error occurred assigning variables from EAPIEvent parameters in ErplyAPI_OnPreRequestEvent. {0} {1}", ex.Message, ex.StackTrace), string.Empty, 0);
}
try
{
MonitoringBuffer.AddRow(); // Exception occurs here
MonitoringBuffer.Request = ResizeString(request, 255);
MonitoringBuffer.SessionKey = ResizeString(sessionKey, 128);
}
catch (Exception ex)
{
var message = string.Format("Error occurred outputting Erply request in ErplyAPI_OnPreRequestEvent. {0} {1}", ex.Message, ex.StackTrace);
MonitoringBuffer.ErrorMessage = ResizeString(message, 8000);
ComponentMetaData.FireWarning(0, "SC_ERPLY_API", message, string.Empty, 0);
}
finally
{
MonitoringBuffer.EndOfRowset();
}
}
}

I sorted the problem out.
The exception was being raised when the variable dispenser was being accessed from the Event. For some reason the GetValueWithContext(ScriptComponent.EvaluatorContext) is being dropped during the second call. Why this happens is beyond me.
The solution is simple, assign the variables from the variables dispenser to a local property or variable in the OnPreExecute function.
It's also good practice to not call the variable dispenser in the CreateNewOutputRows as it cause variable locking.

I ran into this issue too, but my solution was a little different -- moving the variable assignments into PreExecute() didn't help.
Instead, what I'd done is that I wanted to parse three different files, and read each of them with a Script Component. Their columns were kinda similar, so I created one Data Flow task, made sure it worked, then copied it and modified each copy to reflect the differences in the files. Running each individual Data Flow task was successful, but when I tried to run two of them, one after the other in a loop, I got a NullReferenceException from HostPrimeOutput() after calling the OutputBuffer.AddRow() method in my Script Component.
It turns out that when I copied each Data Flow task, the Script Components all kept the same namespace, and I guess it doesn't like that. So, I created brand new Script Components, set up all the output columns again (ugh!), copied the body of the script over, and it's happy.

how to pass an error message back before an exception is thrown

i have the following code that gets an error message.i want to pass it into a string before an exception is thrown,this is my code
ValidateError(authDeserialized, "Succeed", "error", "failed"); //the validateError is a function as indicated below
Model.Response= authResponse.Content;
protected static void ValidateError(dynamic response, string validStatus,string categoryMatch, string message)
{
if (response.result.status != validStatus)
{
try
{
var category = response.result.category;
if (category == categoryMatch)
message=ErrorCodes.MessageFor(code,description);
//so i get the message back fine here but now how do i pass it back to this line Model.Response= authResponse.Content; so that it can get saved?
}
catch (Exception) { }
throw new Exception(message ?? "Request was not successfull");
}
}

As you are already sending message to the ValidateError() method, pass that parameter as a out parameter, it will update value of message if you assign new value to it then it will update message and will be accessible to outside environment.
string failureMessage = "failed";
ValidateError(authDeserialized, "Succeed", "error", out failureMessage);
//^^^ This is what you have to change
//Now you can assign failureMessage to any other value
Model.Response= authResponse.Content;
protected static void ValidateError(dynamic response, string validStatus,string categoryMatch, out string message)
{ //^^^ This is what you have to change
if (response.result.status != validStatus)
{
try
{
var category = response.result.category;
if (category == categoryMatch)
message=ErrorCodes.MessageFor(code,description); //so i get the message back fine here but now how do i pass it back to this line Model.Response= authResponse.Content; so that it can get saved?
}
catch (Exception) { }
throw new Exception(message ?? "Request was not successfull");
}
}
In this way you can assign value to failure message before throwing an error.
Try out online

Limit Dns.GetHostAddresses by time

i'm writing a script that resolve ip address for domain using C#
the problem is that i have a lot of domains that does not resolve to an IP, so the code (Dns.GetHostAddresses) is running for a long time trying to resolve an IP for a domain that doesn't have an IP.
this is the code:
public string getIPfromHost(string host)
{
try
{
var domain = Dns.GetHostAddresses(host)[0];
return domain.ToString();
}
catch (Exception)
{
return "No IP";
}
}
what i want to do is if there is no IP after 1 sec i want to return "No IP"
how can i achieve that?

You can achieve this by using TPL(Task Parallel Library). You can create new task and wait for 1 sec, if it is succeed then return true otherwise false.
just use below code in getIPfromHost(string host) this method.
(This is solution for your question which needs to wait for 1 sec please ensure your previous method was working fine.)
public string getIPfromHost(string host)
{
try
{
Task<string> task = Task<string>.Factory.StartNew(() =>
{
var domain = Dns.GetHostAddresses(host)[0];
return domain.ToString();
});
bool success = task.Wait(1000);
if (success)
{
return task.Result;
}
else
{
return "No IP";
}
}
catch (Exception)
{
return "No IP";
}
}

Error with telegrambot in webapi

I have written a telegram bot with webapi in asp.net and it receives updates but in the response the bot sends me this error:
error:System.NullReferenceException: Object reference not set to an instance of an object.
at WebApiNew.Controllers.WebhookController.Post(Update update)
and my code:
[HttpPost]
public string Post(Update update)
{
if (update != null)
{
try
{
var chatid = update.Message.Chat.Id;
var text = update.Message.Text;
bot.SendTextMessage(chatId: chatid, text: text);
}
catch (Exception ex)
{
bot.SendTextMessage(chatId: 158272989, text: "error :" + ex.ToString());
}
}
else {
bot.SendTextMessage(chatId: 158272989, text: "error ");
}
return "";
}
Where is the problem?

Updates come as a collection of Update objects
Maybe modify your code so post handles an array of updates

limit # of web service requests simultaneously

I have an Excel Add-In written in C#, .NET 4.5. It will send many web service requests to a web server to get data. E.g. it sends 30,000 requests to web service server. When data of a request comes back, the addin will plot the data in Excel.
Originally I did all the requests asynchronously, but sometime I will get OutOfMemoryException
So I changed, sent the requests one by one, but it is too slow, takes long time to finish all requests.
I wonder if there is a way that I can do 100 requests at a time asynchronously, once the data of all the 100 requests come back and plot in Excel, then send the next 100 requests.
Thanks
Edit
On my addin, there is a ribbon button "Refresh", when it is clicked, refresh process starts.
On main UI thread, ribbon/button is clicked, it will call web service BuildMetaData,
once it is returned back, in its callback MetaDataCompleteCallback, another web service call is sent
Once it is returned back, in its callback DataRequestJobFinished, it will call plot to plot data on Excel. see below
RefreshBtn_Click()
{
if (cells == null) return;
Range firstOccurence = null;
firstOccurence = cells.Find(functionPattern, null,
null, null,
XlSearchOrder.xlByRows,
XlSearchDirection.xlNext,
null, null, null);
DataRequest request = null;
_reportObj = null;
Range currentOccurence = null;
while (!Helper.RefreshCancelled)
{
if(firstOccurence == null ||IsRangeEqual(firstOccurence, currentOccurence)) break;
found = true;
currentOccurence = cells.FindNext(currentOccurence ?? firstOccurence);
try
{
var excelFormulaCell = new ExcelFormulaCell(currentOccurence);
if (excelFormulaCell.HasValidFormulaCell)
{
request = new DataRequest(_unityContainer, XLApp, excelFormulaCell);
request.IsRefreshClicked = true;
request.Workbook = Workbook;
request.Worksheets = Worksheets;
_reportObj = new ReportBuilder(_unityContainer, XLApp, request, index, false);
_reportObj.ParseParameters();
_reportObj.GenerateReport();
//this is necessary b/c error message is wrapped in valid object DataResponse
//if (!string.IsNullOrEmpty(_reportObj.ErrorMessage)) //Clear previous error message
{
ErrorMessage = _reportObj.ErrorMessage;
Errors.Add(ErrorMessage);
AddCommentToCell(_reportObj);
Errors.Remove(ErrorMessage);
}
}
}
catch (Exception ex)
{
ErrorMessage = ex.Message;
Errors.Add(ErrorMessage);
_reportObj.ErrorMessage = ErrorMessage;
AddCommentToCell(_reportObj);
Errors.Remove(ErrorMessage);
Helper.LogError(ex);
}
}
}
on Class to GenerateReport
public void GenerateReport()
{
Request.ParseFunction();
Request.MetacompleteCallBack = MetaDataCompleteCallback;
Request.BuildMetaData();
}
public void MetaDataCompleteCallback(int id)
{
try
{
if (Request.IsRequestCancelled)
{
Request.FormulaCell.Dispose();
return;
}
ErrorMessage = Request.ErrorMessage;
if (string.IsNullOrEmpty(Request.ErrorMessage))
{
_queryJob = new DataQueryJob(UnityContainer, Request.BuildQueryString(), DataRequestJobFinished, Request);
}
else
{
ModifyCommentOnFormulaCellPublishRefreshEvent();
}
}
catch (Exception ex)
{
ErrorMessage = ex.Message;
ModifyCommentOnFormulaCellPublishRefreshEvent();
}
finally
{
Request.MetacompleteCallBack = null;
}
}
public void DataRequestJobFinished(DataRequestResponse response)
{
Dispatcher.Invoke(new Action<DataRequestResponse>(DataRequestJobFinishedUI), response);
}
public void DataRequestJobFinished(DataRequestResponse response)
{
try
{
if (Request.IsRequestCancelled)
{
return;
}
if (response.status != Status.COMPLETE)
{
ErrorMessage = ManipulateStatusMsg(response);
}
else // COMPLETE
{
var tmpReq = Request as DataRequest;
if (tmpReq == null) return;
new VerticalTemplate(tmpReq, response).Plot();
}
}
catch (Exception e)
{
ErrorMessage = e.Message;
Helper.LogError(e);
}
finally
{
//if (token != null)
// this.UnityContainer.Resolve<IEventAggregator>().GetEvent<DataQueryJobComplete>().Unsubscribe(token);
ModifyCommentOnFormulaCellPublishRefreshEvent();
Request.FormulaCell.Dispose();
}
}
on plot class
public void Plot()
{
...
attributeRange.Value2 = headerArray;
DataRange.Value2 = ....
DataRange.NumberFormat = ...
}

OutOfMemoryException is not about the too many requests sent simultaneously. It is about freeing your resources right way. In my practice there are two main problems when you are getting such exception:
Wrong working with immutable structures or System.String class
Not disposing your disposable resources, especially graphic objects and WCF requests.
In case of reporting, for my opinion, you got a second one type of a problem. DataRequest and DataRequestResponse are good point to start the investigation for the such objects.
If this doesn't help, try to use the Tasks library with async/await pattern, you can find good examples here:
// Signature specifies Task<TResult>
async Task<int> TaskOfTResult_MethodAsync()
{
int hours;
// . . .
// Return statement specifies an integer result.
return hours;
}
// Calls to TaskOfTResult_MethodAsync
Task<int> returnedTaskTResult = TaskOfTResult_MethodAsync();
int intResult = await returnedTaskTResult;
// or, in a single statement
int intResult = await TaskOfTResult_MethodAsync();
// Signature specifies Task
async Task Task_MethodAsync()
{
// . . .
// The method has no return statement.
}
// Calls to Task_MethodAsync
Task returnedTask = Task_MethodAsync();
await returnedTask;
// or, in a single statement
await Task_MethodAsync();
In your code I see a while loop, in which you can store your Task[] of size of 100, for which you can use the WaitAll method, and the problem should be solved. Sorry, but your code is huge enough, and I can't provide you a more straight example.

I'm having a lot of trouble parsing your code to figure out is being iterated for your request but the basic template for batching asynchronously is going to be something like this:
static const int batchSize = 100;
public async Task<IEnumerable<Results>> GetDataInBatches(IEnumerable<RequestParameters> parameters) {
if(!parameters.Any())
return Enumerable.Empty<Result>();
var batchResults = await Task.WhenAll(parameters.Take(batchSize).Select(doQuery));
return batchResults.Concat(await GetDataInBatches(parameters.Skip(batchSize));
}
where doQuery is something with the signature
Task<Results> async doQuery(RequestParameters parameters) {
//.. however you do the query
}
I wouldn't use this for a million requests since its recursive, but your case should would generate a callstack only 300 deep so you'll be fine.
Note that this also assumes that your data request stuff is done asynchronously and returns a Task. Most libraries have been updated to do this (look for methods with the Async suffix). If it doesn't expose that api you might want to create a separate question for how to specifically get your library to play nice with the TPL.