Develop Reference

Parallel Tasks & lock local variables

I'm new in multithreading programming and I have some questions about it. I'm developing a scheduled job that loads a .xls that every row with data needs to be validated and after open a request in a application, and I'm implementing multithreading to process every row, aiming win some performance.
In my job I have something like this example:
public class Job
{
private readonly IUserBLO _userBLO;
private readonly IRequestBLO _requestBLO;
private readonly object _lockerSheet;
public Job()
{
_userBLO = Factory.CreateInstance("UserBLO");
_requestBLO = Factory.CreateInstance("RequestBLO");
_lockerSheet = new object();
}
public void ProcessJob(BinaryWriter binary)
{
using (ExcelPackage package = new ExcelPackage(binary))
{
var sheet = package.Workbook.Worksheet[1];
var rows = Enumerable.Range(sheet.Dimensions.Start.Row, sheet.Dimensions.End.Row);
Parallel.ForEach(rows, row =>
{
try
{
var excelData = GetExcelData(sheet, row);
ValidadeData(excelData);
_requestBLO.OpenRequestByExcelData(excelData);
}
catch (Exception ex)
{
LogError(ex);
}
});
}
}
public List<string> GetExcelData(ExcelWorksheet sheet, int row)
{
var excelData = new List<string>();
for (int i = sheet.Dimensions.Start.Column; i <= sheet.Dimensions.End.Column; i++)
{
lock (_lockerSheet)
{
var cellValue = sheet.Cells[row, i].Text;
}
excelData.Add(cellValue);
}
return excelData;
}
public void ValidadeData(List<string> excelData)
{
var userLogin = excelData.First();
var user = _userBLO.FindByLogin(userLogin);
if (user == null)
throw new Exception("User not found!");
}
}
When I was coding, I had some issues that when I tried to get the value from sheet in a random position, sometimes, it returned empty, even though this position had value. After some search in google, I tried to use a locker in the moment I was getting the value from sheet, and apparently worked!
lock (_lockerSheet)
{
var cellValue = sheet.Cells[row, i].Text;
}
After this situation, I had some questions:
For every "local" variable that will be parallel used in the tasks needs to be locked, even if that variable is readonly. I'm correct?
If the question above is true, do I need to use a locker in the _userBLO and _requestBLO variables, which are interfaces that contais some specific methods?
If the first question is false, do I just need use a locker if the variable will recive some value in task? When do I really need to use a locker?
PS.: Sorry for my bad english! Brazilian here! :)

C# System.AggregateException has been thrown

I am currently working on a console application that uses the Last10kAPI to retrieve financial data from various documents that they provide but whenever I build the application it compiles and opens up the terminal I get a weird error saying "System.AggregateException has been thrown.One or more errors occurred. (Cannot perform runtime binding on a null reference)" For the sake of simplicity I some of my code from my Program.cs below. The formatting is a bit off when adding it to stackoverflow so just assume proper indentation. Any help would be appreciated. Thanks in Advance!
class Program
{
static void Main(string[] args)
{
RunAsync().Wait();
Console.WriteLine("Financial Data from Ratios, Balance Sheet, Cash Flow, and Income Statement");
}
static private async Task RunAsync()
{
//Grab our ticker List;
List<string> tickers = new List<string>();
tickers = await GetTickers();
//Grab ratios for all companies in ticker list and store the results in a list of Ratios
List<Ratios> ratiosfortickers = new List<Ratios>();
foreach (string ticker in tickers)
{
Ratios tempRatio = new Ratios();
tempRatio = await GetRatio(ticker);
ratiosfortickers.Add(tempRatio);
}
//Grab all balance sheet
List<BalanceSheets> balancesheetfortickers = new List<BalanceSheets>();
foreach (string ticker in tickers)
{
BalanceSheets tempBalanceSheets = new BalanceSheets();
tempBalanceSheets = await GetBalanceSheet(ticker);
//balancesheetsfortickers.Add(tempBalanceSheets);
}
//Grab all Cash Flow
List<CashFlows> cashflowsfortickers = new List<CashFlows>();
foreach (string ticker in tickers)
{
CashFlows tempCashFlow = new CashFlows();
tempCashFlow = await GetCashFlows(ticker);
cashflowsfortickers.Add(tempCashFlow);
}
//Grab all Income Statement
List<Income> incomefortickers = new List<Income>();
foreach (string ticker in tickers)
{
Income tempIncome = new Income();
tempIncome = await GetIncome(ticker);
incomefortickers.Add(tempIncome);
}
}
}
}

looking for the best way to see when my task is complete C#

So I'm pulling in a list of items and for each item I'm creating an instance of an object to run a task on that item. All the objects are the same, they updated based off of a received message every three seconds. This update does not all occur at once though, sometimes it takes 3.1 seconds, etc. This is data I need to serialize in XML once it all exists so I'm looking for a way to see when its all done.
I've explored tasks in .net 4.6 but that initiates a task and it reports complete and then to run again the task class would initiate it again but in my case that won't work because each instance stays alive and initiates itself when a new message comes in.
What is the best way to have it report it reached the last line of code and then look at a list of these instances and say when all of them show as complete then run task to serialize?
I've included code below of the instance that is running.
private void OnMessageReceived(object sender, MessageReceivedEventArgs e)
{
var eventArgs = new CallDataReceivedEventArgs();
this.OnCallDataReceived(eventArgs);
try
{
List<Tuple<String, TimeSpan>> availInItems = new List<Tuple<string, TimeSpan>>();
List<Tuple<string, int, TimeSpan, string, string, string>> agentlist = new List<Tuple<string, int, TimeSpan, string, string, string>>();
if (e == null)
{
return;
}
List<TimeSpan> listOfTimeSpans = new List<TimeSpan>();
if (e.CmsData != null)
{
#region Gathering Agent Information
// Create a list of all timespans for all _agents in a queue using the property AgentTimeInState
foreach (var item in e.CmsData.Agents)
{
//AgentData = new ScoreBoardAgentDataModel(AgentName, AgentExtension, AgentTimeInState, AgentAuxReason, AgentId, AgentAdcState);
_agentData.AgentName = item.AgName;
_agentData.AgentExtension = item.Extension;
_agentData.AgentAuxReason = item.AuxReasonDescription;
_agentData.AgentId = item.LoginId;
_agentData.AgentAcdState = item.WorkModeDirectionDescription;
_agentData.AgentTimeInState = DateTime.Now - item.DateTimeUpdated;
_agentData.TimeSubmitted = DateTime.Now;
agentlist.Add(Tuple.Create(_agentData.AgentName, _agentData.AgentExtension, _agentData.AgentTimeInState, _agentData.AgentId, _agentData.AgentAcdState, _agentData.AgentAuxReason));
if (_agentData.AgentAcdState == "AVAIL")
{
listOfTimeSpans.Add(_agentData.AgentTimeInState);
availInItems.Add(Tuple.Create(_agentData.AgentName, _agentData.AgentTimeInState));
}
availInItems.Sort((t1, t2) => t1.Item2.CompareTo(t2.Item2));
}
var availInAgents =
agentlist
.Where(ag => ag.Item5 == "AVAIL")
.ToList();
availInAgents.Sort((t1, t2) =>
t1.Item3.CompareTo(t2.Item3));
var max3 = availInAgents.Skip(availInAgents.Count - 3);
max3.Reverse();
_agents.AgentsOnBreak = 0;
foreach (var agent in agentlist)
{
if (!string.IsNullOrEmpty(agent.Item6) && agent.Item6.StartsWith("Break"))
{
_agents.AgentsOnBreak++;
}
}
_agents.AgentsOnLunch = 0;
foreach (var agent in agentlist)
{
//If the current agent's aux reason is Lunch
if (!string.IsNullOrEmpty(agent.Item6) && agent.Item6.StartsWith("Lunch"))
{
//add one to agentsonlunch
_agents.AgentsOnLunch++;
}
}
_agents.NextInLine = string.Empty;
foreach (var agent in max3.Reverse())
{
//assign agent to NextInLine and start a new line
_agents.NextInLine += agent.Item1 + Environment.NewLine;
//reverse NextInLine
_agents.NextInLine.Reverse();
}
_agents.TimeSubmitted = DateTime.Now;
#endregion
#region Gathering Skill Information
_skillData.OldestCall = e.CmsData.Skill.OldestCall;
_skillData.AgentsStaffed = e.CmsData.Skill.AgentsStaffed;
_skillData.AgentsAuxed = e.CmsData.Skill.AgentsInAux;
_skillData.AgentsAvailable = e.CmsData.Skill.AgentsAvailable;
_skillData.AgentsOnCalls = e.CmsData.Skill.AgentsOnAcdCall;
_skillData.CallsWaitingInQueue = e.CmsData.Skill.InQueueInRing;
_skillData.Asa = e.CmsData.Skill.AnswerTimePerAcdCall;
_skillData.TimeSubmitted = DateTime.Now;
_skillData.EstimatedHoldTimeLow = e.CmsData.Skill.ExpectedWaitTimeLow;
_skillData.EstimatedHoldTimeMedium = e.CmsData.Skill.ExpectedWaitTimeMedium;
_skillData.EstimatedHoldTimeHigh = e.CmsData.Skill.ExpectedWaitTimeHigh;
#endregion
}
}
catch (Exception ex)
{
_logger.Info(ex.Message, ex);
}
}

With tasks you can start many at the same time and wait for them all to finish like this:
var taskList = new List<Task>();
foreach (var thingToDo in work)
{
taskList.Add(thingToDo.StartTask());
}
Task.WaitAll(taskList.ToArray());
This way you can run everything in parallel and wont get after the last line until everything is done.
Edit following your comment
You can embed your work in a task with this:
public async Task DoWork()
{
var taskList = new List<Task>();
foreach (var thingToDo in work)
{
taskList.Add(thingToDo.StartTask());
}
await Task.WhenAll(taskList.ToArray());
}

JSON Array to Entity Framework Core VERY Slow?

I'm working on a utility to read through a JSON file I've been given and to transform it into SQL Server. My weapon of choice is a .NET Core Console App (I'm trying to do all of my new work with .NET Core unless there is a compelling reason not to). I have the whole thing "working" but there is clearly a problem somewhere because the performance is truly horrifying almost to the point of being unusable.
The JSON file is approximately 27MB and contains a main array of 214 elements and each of those contains a couple of fields along with an array of from 150-350 records (that array has several fields and potentially a small <5 record array or two). Total records are approximately 35,000.
In the code below I've changed some names and stripped out a few of the fields to keep it more readable but all of the logic and code that does actual work is unchanged.
Keep in mind, I've done a lot of testing with the placement and number of calls to SaveChanges() think initially that number of trips to the Db was the problem. Although the version below is calling SaveChanges() once for each iteration of the 214-record loop, I've tried moving it outside of the entire looping structure and there is no discernible change in performance. In other words, with zero trips to the Db, this is still SLOW. How slow you ask, how does > 24 hours to run hit you? I'm willing to try anything at this point and am even considering moving the whole process into SQL Server but would much reather work in C# than TSQL.
static void Main(string[] args)
{
string statusMsg = String.Empty;
JArray sets = JArray.Parse(File.ReadAllText(#"C:\Users\Public\Downloads\ImportFile.json"));
try
{
using (var _db = new WidgetDb())
{
for (int s = 0; s < sets.Count; s++)
{
Console.WriteLine($"{s.ToString()}: {sets[s]["name"]}");
// First we create the Set
Set eSet = new Set()
{
SetCode = (string)sets[s]["code"],
SetName = (string)sets[s]["name"],
Type = (string)sets[s]["type"],
Block = (string)sets[s]["block"] ?? ""
};
_db.Entry(eSet).State = Microsoft.EntityFrameworkCore.EntityState.Added;
JArray widgets = sets[s]["widgets"].ToObject<JArray>();
for (int c = 0; c < widgets.Count; c++)
{
Widget eWidget = new Widget()
{
WidgetId = (string)widgets[c]["id"],
Layout = (string)widgets[c]["layout"] ?? "",
WidgetName = (string)widgets[c]["name"],
WidgetNames = "",
ReleaseDate = releaseDate,
SetCode = (string)sets[s]["code"]
};
// WidgetColors
if (widgets[c]["colors"] != null)
{
JArray widgetColors = widgets[c]["colors"].ToObject<JArray>();
for (int cc = 0; cc < widgetColors.Count; cc++)
{
WidgetColor eWidgetColor = new WidgetColor()
{
WidgetId = eWidget.WidgetId,
Color = (string)widgets[c]["colors"][cc]
};
_db.Entry(eWidgetColor).State = Microsoft.EntityFrameworkCore.EntityState.Added;
}
}
// WidgetTypes
if (widgets[c]["types"] != null)
{
JArray widgetTypes = widgets[c]["types"].ToObject<JArray>();
for (int ct = 0; ct < widgetTypes.Count; ct++)
{
WidgetType eWidgetType = new WidgetType()
{
WidgetId = eWidget.WidgetId,
Type = (string)widgets[c]["types"][ct]
};
_db.Entry(eWidgetType).State = Microsoft.EntityFrameworkCore.EntityState.Added;
}
}
// WidgetVariations
if (widgets[c]["variations"] != null)
{
JArray widgetVariations = widgets[c]["variations"].ToObject<JArray>();
for (int cv = 0; cv < widgetVariations.Count; cv++)
{
WidgetVariation eWidgetVariation = new WidgetVariation()
{
WidgetId = eWidget.WidgetId,
Variation = (string)widgets[c]["variations"][cv]
};
_db.Entry(eWidgetVariation).State = Microsoft.EntityFrameworkCore.EntityState.Added;
}
}
}
_db.SaveChanges();
}
}
statusMsg = "Import Complete";
}
catch (Exception ex)
{
statusMsg = ex.Message + " (" + ex.InnerException + ")";
}
Console.WriteLine(statusMsg);
Console.ReadKey();
}

I had an issue with that kind of code, lots of loops and tons of changing state.
Any change / manipulation you make in _db context, will generate a "trace" of it. And it making your context slower each time. Read more here.
The fix for me was to create new EF context(_db) at some key points. It saved me a few hours per run!
You could try to create a new instance of _db each iteration in this loop
contains a main array of 214 elements
If it make no change, try to add some stopwatch to get a best idea of what/where is taking so long.

If you're making thousands of updates then EF is not really the way to go. Something like SQLBulkCopy will do the trick.
You could try the bulkwriter library.
IEnumerable<string> ReadFile(string path)
{
using (var stream = File.OpenRead(path))
using (var reader = new StreamReader(stream))
{
while (reader.Peek() >= 0)
{
yield return reader.ReadLine();
}
}
}
var items =
from line in ReadFile(#"C:\products.csv")
let values = line.Split(',')
select new Product {Sku = values[0], Name = values[1]};
then
using (var bulkWriter = new BulkWriter<Product>(connectionString)) {
bulkWriter.WriteToDatabase(items);
}

CRM Dynamics 2013 SDK Update Current Accounts With 2 Values

I have a scenario in CRM where I need to update existing accounts with their Vat and Registration number. There is well over 30 thousand accounts in the system. I am trying to update using the CRM SDK API but I am battling to figure out how to perform the actual update. The vat number and reg have been provided to me in a spreadsheet with their corresponding number, please note that the accounts are already in CRM so I just need to update the correct account with its Vat and Reg number, How can I do this in CRM, please advice on my code below:
public static void UpdateAllCRMAccountsWithVATAndRegistrationNumber(IOrganizationService service)
{
QueryExpression qe = new QueryExpression();
qe.EntityName = "account";
qe.ColumnSet = new ColumnSet("account", "new_vatno", "new_registrationnumber");
qe.Criteria.AddCondition("accountnumber", ConditionOperator.In,"TA10024846", "TA10028471", "TA20014015", "TA4011652", "TA4011557");
EntityCollection response = service.RetrieveMultiple(qe);
foreach (var acc in response.Entities)
{
acc.Attributes["new_vatno"] = //this is where I am struggling to figure out how I am gong to match the records up,
acc.Attributes["new_registrationnumber"] = //this is where I am struggling to figure out how I am gong to match the records up,
service.Update(acc);
}
}
How am I going to ensure that I update the correct records. I have the vat and reg numbers for the accounts in a spreadsheet, please see example image below. Can I please get advised here. Thanks.

I would load the list of VAT updates from the spreadsheet into a dictionary and then load the 30k record from CRM into memory. Then I would match them up and use ExecuteMultipleRequest to do the updates. Alternatively, you could query CRM using the account numbers (if the list is small enough.) I made the assumption you had thousands of updates to do across the record set of 30k. Note, if the Account record size was very large and couldn't be loaded into memory you would need to do account number queries.
Here is the rough code for the basic solution (I haven't tested, method should be split up, and there is minimal error handling):
public class VatInfo
{
public string RegistrationNumber;
public string TaxNumber;
public static Dictionary<string, VatInfo> GetVatList()
{
//TODO: Implement logic to load CSV file into a list. Dictionary key value should be Account Number
throw new NotImplementedException();
}
}
public class UpdateVatDemo
{
public const int maxBatchSize = 100;
public static void RunVatUpdate(IOrganizationService conn)
{
var vats = VatInfo.GetVatList();
var pagingQuery = new QueryExpression("account");
pagingQuery.ColumnSet = new ColumnSet("accountnumber");
Queue<Entity> allEnts = new Queue<Entity>();
while (true)
{
var results = conn.RetrieveMultiple(pagingQuery);
if (results.Entities != null && results.Entities.Any())
results.Entities.ToList().ForEach(allEnts.Enqueue);
if (!results.MoreRecords) break;
pagingQuery.PageInfo.PageNumber++;
pagingQuery.PageInfo.PagingCookie = results.PagingCookie;
}
ExecuteMultipleRequest emr = null;
while (allEnts.Any())
{
if (emr == null)
emr = new ExecuteMultipleRequest()
{
Settings = new ExecuteMultipleSettings()
{
ContinueOnError = true,
ReturnResponses = true
},
Requests = new OrganizationRequestCollection()
};
var ent = allEnts.Dequeue();
if (vats.ContainsKey(ent.GetAttributeValue<string>("accountnumber")))
{
var newEnt = new Entity("account", ent.Id);
newEnt.Attributes.Add("new_vatno", vats[ent.GetAttributeValue<string>("accountnumber")].TaxNumber);
newEnt.Attributes.Add("new_registrationnumber", vats[ent.GetAttributeValue<string>("accountnumber")].RegistrationNumber);
emr.Requests.Add(new UpdateRequest() { Target = newEnt });
}
if (emr.Requests.Count >= maxBatchSize)
{
try
{
var emResponse = (ExecuteMultipleResponse) conn.Execute(emr);
foreach (
var responseItem in emResponse.Responses.Where(responseItem => responseItem.Fault != null))
DisplayFault(emr.Requests[responseItem.RequestIndex],
responseItem.RequestIndex, responseItem.Fault);
}
catch (Exception ex)
{
Console.WriteLine($"Exception during ExecuteMultiple: {ex.Message}");
throw;
}
emr = null;
}
}
}
private static void DisplayFault(OrganizationRequest organizationRequest, int count,
OrganizationServiceFault organizationServiceFault)
{
Console.WriteLine(
"A fault occurred when processing {1} request, at index {0} in the request collection with a fault message: {2}",
count + 1,
organizationRequest.RequestName,
organizationServiceFault.Message);
}
}

Updating the fetched entity is bound to fail because of its entity state, which would not be null.
To update the fetched entities, you need to new up the entity:
foreach (var acc in response.Entities)
{
var updateAccount = new Entity("account") { Id = acc.Id };
updateAccount .Attributes["new_vatno"] = null; //using null as an example.
updateAccount .Attributes["new_registrationnumber"] = null;
service.Update(acc);
}

Code below shows how I managed to get it righy. forst let me explain. I imported my records into a seperate SQL table, in my code I read that table into a list in memory, I then query CRM accounts that need to be updated, I then loop though each account and check if the account number in CRM matches the account number from my sql database, if it matches, I then update the relevant Reg no and Vat no, See code below:
List<Sheet1_> crmAccountList = new List<Sheet1_>();
//var crmAccount = db.Sheet1_.Select(x => x).ToList().Take(2);
var crmAccounts = db.Sheet1_.Select(x => x).ToList();
foreach (var dbAccount in crmAccounts)
{
CRMDataObject modelObject = new CRMDataObject()
{
ID = dbAccount.ID,
Account_No = dbAccount.Account_No,
Tax_No = dbAccount.Tax_No.ToString(),
Reg_No = dbAccount.Reg_No
//Tarsus_Country = dbAccount.Main_Phone
};
}
var officialDatabaseList = crmAccounts;
foreach (var crmAcc in officialDatabaseList)
{
QueryExpression qe = new QueryExpression();
qe.EntityName = "account";
qe.ColumnSet = new ColumnSet("accountnumber", "new_vatno", "new_registrationnumber");
qe.Criteria.AddCondition("accountnumber", ConditionOperator.In,'list of account numbers go here'
EntityCollection response = service.RetrieveMultiple(qe);
foreach (var acc in response.Entities)
{
if (crmAcc.Account_No == acc.Attributes["accountnumber"].ToString())
{
//acc.Attributes["new_vatno"] = crmAcc.VAT_No.ToString();
acc.Attributes["new_registrationnumber"] = crmAcc.Reg_No.ToString();
service.Update(acc);
}
}
}