Develop Reference

Unable to implement data parsing in a multi-threaded context using lock

I've built a program that
takes in a list of record data from a file
parses and cleans up each record in a parsing object
outputs it to an output file
So far this has worked on a single thread, but considering the fact that records can exceed 1 million in some cases, we want to implement this in a multi threading context. Multi threading is new to me in .Net, and I've given it a shot but its not working. Below I will provide more details and code:
Main Class (simplified):
public class MainClass
{
parseObject[] parseObjects;
Thread[] threads;
List<InputLineItem> inputList = new List<InputLineItem>();
FileUtils fileUtils = new FileUtils();
public GenParseUtilsThreaded(int threadCount)
{
this.threadCount = threadCount;
Init();
}
public void Init()
{
inputList = fileUtils.GetInputList();
parseObjects = new parseObject[threadCount - 1];
threads = new Thread[threadCount - 1];
InitParseObjects();
Parse();
}
private void InitParseObjects()
{
//using a ref of fileUtils to use as my lock expression
parseObjects[0] = new ParseObject(ref fileUtils);
parseObjects[0].InitValues();
for (int i = 1; i < threadCount - 1; i++)
{
parseObjects[i] = new parseObject(ref fileUtils);
parseObjects[i].InitValues();
}
}
private void InitThreads()
{
for (int i = 0; i < threadCount - 1; i++)
{
Thread t = new Thread(new ThreadStart(parseObjects[0].CleanupAndParseInput));
threads[i] = t;
}
}
public void Parse()
{
try
{
InitThreads();
int objectIndex = 0;
foreach (InputLineItem inputLineItem in inputList)
{
parseObjects[0].inputLineItem = inputLineItem;
threads[objectIndex].Start();
objectIndex++;
if (objectIndex == threadCount)
{
objectIndex = 0;
InitThreads(); //do i need to re-init the threads after I've already used them all once?
}
}
}
catch (Exception e)
{
Console.WriteLine("(286) The following error occured: " + e);
}
}
}
}
And my Parse object class (also simplified):
public class ParseObject
{
public ParserLibrary parser { get; set; }
public FileUtils fileUtils { get; set; }
public InputLineItem inputLineItem { get; set; }
public ParseObject( ref FileUtils fileUtils)
{
this.fileUtils = fileUtils;
}
public void InitValues()
{
//relevant config of parser library object occurs here
}
public void CleanupFields()
{
parser.Clean(inputLineItem.nameValue);
inputLineItem.nameValue = GetCleanupUpValueFromParser();
}
private string GetCleanupFieldValue()
{
//code to extract cleanup up value from parses
}
public void CleanupAndParseInput()
{
CleanupFields();
ParseInput();
}
public void ParseInput()
{
try
{
parser.Parse(InputLineItem.NameValue);
}
catch (Exception e)
{
}
try
{
lock (fileUtils)
{
WriteOutputToFile(inputLineItem);
}
}
catch (Exception e)
{
Console.WriteLine("(414) Failed to write to output: " + e);
}
}
public void WriteOutputToFile(InputLineItem inputLineItem)
{
//writes updated value to output file
}
}
The error I get is when trying to run the Parse function, I get this message:
An unhandled exception of type 'System.AccessViolationException' occurred in GenParse.NET.dll
Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
That being said, I feel like there's a whole lot more that I'm doing wrong here aside from what is causing that error.
I also have further questions:
Do I create multiple parse objects and iteratively feed them to each thread as I'm attempting to do, or should I use one Parse object that gets shared or cloned across each thread?
If, outside the thread, I change a value in the object that I'm passing to the thread, will that change reflect in the object passed to the thread? i.e, is the object passed by value or reference?
Is there a more efficient way for each record to be assigned to a thread and its parse object than I am currently doing with the objectIndex iterator?
THANKS!

Do I create multiple parse objects and iteratively feed them to each thread as I'm attempting to do, or should I use one Parse object that gets shared or cloned across each thread?
You initialize each thread with new ThreadStart(parseObjects[0].CleanupAndParseInput) so all threads will share the same parse object. It is a fairly safe bet that the parse objects are not threadsafe. So each thread should have a separate object. Note that this might not be sufficient, if the parse library uses any global fields it might be non-threadsafe even when using separate objects.
If, outside the thread, I change a value in the object that I'm passing to the thread, will that change reflect in the object passed to the thread? i.e, is the object passed by value or reference?
Objects (i.e. classes) are passed by reference. But any changes to an object are not guaranteed to be visible in other threads unless a memoryBarrier is issued. Most synchronization code (like lock) will issue memory barriers. Keep in mind that any non-atomic operation is unsafe if a field is written an read concurrently.
Is there a more efficient way for each record to be assigned to a thread and its parse object than I am currently doing with the objectIndex iterator?
Using manual threads in this way is very old-school. The modern, easier, and probably faster way is to use a parallel-for loop. This will try to be smart about how many threads it will use and try to adapt chunk sizes to keep the synchronization overhead low.
var items = new List<int>();
ParseObject LocalInit()
{
// Do initalization, This is run once for each thread used
return new ParseObject();
}
ParseObject ThreadMain(int value, ParallelLoopState state, ParseObject threadLocalObject)
{
// Do whatever you need to do
// This is run on multiple threads
return threadLocalObject;
}
void LocalFinally(ParseObject obj)
{
// Do Cleanup for each thread
}
Parallel.ForEach(items, LocalInit, ThreadMain, LocalFinally);
As a final note, I would advice against using multithreading unless you are familiar with the potential dangers and pitfalls it involves, at least for any project where the result is important. There are many ways to screw up and make a program that will work 99.9% of the time, and silently corrupt data the remaining 0.1% of the time.

How to make SQlite async in UWP?

I'm fairly new to databases and asynchronous programing. I'm making a POS app that will eventually have hundreds of customers and possibly thousands of transactions. When I want to do a customer search or look up a previous ticket I don't want my program to hang waiting on the results.
Here is the method that shows the search result:
private void searchCritiria_TextChanging(TextBox sender, TextBoxTextChangingEventArgs args)
{
FilteredCustomer.Clear();
if(searchCritiria.Text.Length >= 3)
{
SQLiteConnection dbConnection = new SQLiteConnection("Customers.db");
string sSQL = null;
sSQL = #"SELECT [first],[last],[spouse],[home],[work],[cell] FROM Customers";
ISQLiteStatement dbState = dbConnection.Prepare(sSQL);
while (dbState.Step() == SQLiteResult.ROW)
{
string sFirst = dbState["first"] as string;
string sLast = dbState["last"] as string;
string sSpouse = dbState["spouse"] as string;
string sHome = dbState["home"] as string;
string sWork = dbState["work"] as string;
string sCell = dbState["cell"] as string;
//Load into observable collection
if (searchType.SelectedIndex == 0)//name search
{
if(sFirst.Contains(searchCritiria.Text) || sLast.Contains(searchCritiria.Text) || sSpouse.Contains(searchCritiria.Text))
FilteredCustomer.Add(new Customer {first = sFirst, last = sLast, spouse = sSpouse, home = sHome, work = sWork, cell = sCell});
}
else//number search
{
if(sWork.Contains(searchCritiria.Text)|| sHome.Contains(searchCritiria.Text) || sCell.Contains(searchCritiria.Text))
FilteredCustomer.Add(new Customer { first = sFirst, last = sLast, spouse = sSpouse, home = sHome, work = sWork, cell = sCell });
}
}
}
}
Throughout my program I have void methods that are structured similar to this.
I'm not sure how to solve this issue. I tried doing some research but no success. Any advice would be greatly appreciated!

Edit
As correctly pointed out by #AndriyK, the answer would not have made your code asynchronous.
What it means:
Although the code works and you're able to push in data, if in case you reach a condition where two threads are trying to write into the database (Database is Locked condition, your application would still freeze until the database is not locked anymore or it times out. This means any loaders running would also freeze.
Making it asynchronous
To do so, you must run your SQLite code in a Task.Run() block. This would ensure your database runs on a different Thread and ensure your UI would never hang. This would also allow you to show ProgressRing (or other loaders) to the user while the application waits for the database to unlock. Below is the code:
public void PerformSQLTasks()
{
// your SQL code code here!
}
and run the above code by using:
Task.Run(() => PerformSQLTasks());
// or even
Task.Run(PerformSQLTasks);
Accepted Answer:
To make your UI not hang, you can make your method return a Task instead of void to make it awaitable. This will not make your code async but would allow the caller to await this method completion. To achieve this, follow the method signature below:
public Task PerformSQLTasks()
{
// your code here!
return Task.CompletedTask;
}
and you can call it by:
private async void searchCritiria_TextChanging(TextBox sender, TextBoxTextChangingEventArgs args)
{
await PerformSQLTasks();
}
Remember this won't make your code Asynchronous

uwp c# async method waiting data not completely loaded exception

I will try to tell my problem in as simple words as possible.
In my UWP app, I am loading the data async wise on my Mainpage.xaml.cs`
public MainPage()
{
this.InitializeComponent();
LoadVideoLibrary();
}
private async void LoadVideoLibrary()
{
FoldersData = new List<FolderData>();
var folders = (await Windows.Storage.StorageLibrary.GetLibraryAsync
(Windows.Storage.KnownLibraryId.Videos)).Folders;
foreach (var folder in folders)
{
var files = (await folder.GetFilesAsync(Windows.Storage.Search.CommonFileQuery.OrderByDate)).ToList();
FoldersData.Add(new FolderData { files = files, foldername = folder.DisplayName, folderid = folder.FolderRelativeId });
}
}
so this is the code where I am loading up a List of FolderData objects.
There in my other page Library.xaml.cs I am using that data to load up my gridview with binding data.
protected override void OnNavigatedTo(NavigationEventArgs e)
{
try
{
LoadLibraryMenuGrid();
}
catch { }
}
private async void LoadLibraryMenuGrid()
{
MenuGridItems = new ObservableCollection<MenuItemModel>();
var data = MainPage.FoldersData;
foreach (var folder in data)
{
var image = new BitmapImage();
if (folder.files.Count == 0)
{
image.UriSource = new Uri("ms-appx:///Assets/StoreLogo.png");
}
else
{
for (int i = 0; i < folder.files.Count; i++)
{
var thumb = (await folder.files[i].GetThumbnailAsync(Windows.Storage.FileProperties.ThumbnailMode.VideosView));
if (thumb != null) { await image.SetSourceAsync(thumb); break; }
}
}
MenuGridItems.Add(new MenuItemModel
{
numberofvideos = folder.files.Count.ToString(),
folder = folder.foldername,
folderid = folder.folderid,
image = image
});
}
GridHeader = "Library";
}
the problem I am facing is that when i launch my application, wait for a few seconds and then i navigate to my library page, all data loads up properly.
but when i try to navigate to library page instantly after launching the app, it gives an exception that
"collection was modified so it cannot be iterated"
I used the breakpoint and i came to know that if i give it a few seconds the List Folder Data is already loaded properly asyncornously, but when i dnt give it a few seconds, that async method is on half way of loading the data so it causes exception, how can i handle this async situation? thanks

What you need is a way to wait for data to arrive. How you fit that in with the rest of the application (e.g. MVVM or not) is a different story, and not important right now. Don't overcomplicate things. For example, you only need an ObservableCollection if you expect the data to change while the user it looking at it.
Anyway, you need to wait. So how do you wait for that data to arrive?
Use a static class that can be reached from everywhere. In there put a method to get your data. Make sure it returns a task that you cache for future calls. For example:
internal class Data { /* whatever */ }
internal static class DataLoader
{
private static Task<Data> loaderTask;
public static Task<Data> LoadDataAsync(bool refresh = false)
{
if (refresh || loaderTask == null)
{
loaderTask = LoadDataCoreAsync();
}
return loaderTask;
}
private static async Task<Data> LoadDataCoreAsync()
{
// your actual logic goes here
}
}
With this, you can start the download as soon as you start the application.
await DataLoader.LoadDataAsync();
When you need the data in that other screen, just call that method again. It will not download the data again (unless you set refresh is true), but will simply wait for the work that you started earlier to finish, if it is not finished yet.

I get that you don't have enough experience.There are multiple issues and no solution the way you are loading the data.
What you need is a Service that can give you ObservableCollection of FolderData. I think MVVM might be out of bounds at this instance unless you are willing to spend a few hours on it. Though MVVM will make things lot easier in this instance.
The main issue at hand is this
You are using foreach to iterate the folders and the FolderData list. Foreach cannot continue if the underlying collection changes.
Firstly you need to start using a for loop as opposed to foreach. 2ndly add a state which denotes whether loading has finished or not. Finally use observable data source. In my early days I used to create static properties in App.xaml.cs and I used to use them to share / observe other data.

How to make a static class update its own variables constantly?

I have a user control that displays information from the database. This user control has to update these information constantly(let's say every 5 seconds). A few instances of this user control is generated programmatically during run time in a single page. In the code behind of this user control I added a code that sends a query to the database to get the needed information (which means every single instance of the user control is doing this). But this seems to slow down the processing of queries so I am making a static class that will do the querying and store the information in its variables and let the instances of my user control access those variables. Now I need this static class to do queries every 5 seconds to update its variables. I tried using a new thread to do this but the variables don't seem to be updated since I always get a NullReferenceException whenever I access them from a different class.
Here's my static class:
public static class SessionManager
{
public static volatile List<int> activeSessionsPCIDs;
public static volatile List<int> sessionsThatChangedStatus;
public static volatile List<SessionObject> allSessions;
public static void Initialize() {
Thread t = new Thread(SetProperties);
t.Start();
}
public static void SetProperties() {
SessionDataAccess sd = new SessionDataAccess();
while (true) {
allSessions = sd.GetAllSessions();
activeSessionsPCIDs = new List<int>();
sessionsThatChangedStatus = new List<int>();
foreach (SessionObject session in allSessions) {
if (session.status == 1) { //if session is active
activeSessionsPCIDs.Add(session.pcid);
}
if (session.status != session.prevStat) { //if current status doesn't match the previous status
sessionsThatChangedStatus.Add(session.pcid);
}
}
Thread.Sleep(5000);
}
}
And this is how I am trying to access the variables in my static class:
protected void Page_Load(object sender, EventArgs e)
{
SessionManager.Initialize();
loadSessions();
}
private void loadSessions()
{ // refresh the current_sessions table
List<int> pcIds = pcl.GetPCIds(); //get the ids of all computers
foreach (SessionObject s in SessionManager.allSessions)
{
SessionInfo sesInf = (SessionInfo)LoadControl("~/UserControls/SessionInfo.ascx");
sesInf.session = s;
pnlMonitoring.Controls.Add(sesInf);
}
}
Any help, please? Thanks

Multiple threads problem
You have one thread that gets created for each and every call to SessionManager.Initialize.
That happens more than once in the lifetime of the process.
IIS recycles your app at some point, after a period of time should you have absolutely no requests.
Until that happens, all your created threads continue to run.
After the first PageLoad you will have one thread which updates stuff every 5 seconds.
If you refresh the page again you'll have two threads, possibly with different offsets in time but each of which, doing the same thing at 5 second intervals.
You should atomically check to see if your background thread is started already. You need at least an extra bool static field and a object static field which you should use like a Monitor (using the lock keyword).
You should also stop relying on volatile and simply using lock to make sure that other threads "observe" updated values for your static List<..> fields.
It may be the case that the other threads don't observe a change field and thusly, for them, the field is still null - therefore you get the NullReferenceException.
About volatile
Using volatile is bad, at least in .NET. There is a 90% chance that you think you know what it is doing and it's not true and there's a 99% chance that you feel relief because you used volatile and you aren't checking for other multitasking hazards the way you should.
RX to the rescue
I strongly suggest you take a look at this wonderful thing called Reactive Extensions.
Believe me, a couple of days' research combined with the fact that you're in a perfect position to use RX will pay of, not just now but in the future as well.
You get to keep your static class, but instead of materialised values that get stored within that class you create pipes that carry information. The information flows when you want it to flow. You get to have subscribers to those pipes. The number of subscribers does not affect the overall performance of your app.
Your app will be more scalable, and more robust.
Good luck!

There are few solution for this approach:
One of them is:
It's better in Global.asax in Application_start or Session_Start (depends on your case) create Thread to call your method:
Use below code :
var t = Task.Factory.StartNew(() => {
while(true)
{
SessionManager.SetProperties();
Task.Delay(5);
}
});
Second solution is using Job Scheduler for ASP.NET (that's my ideal solution).
for more info you can check this link How to run Background Tasks in ASP.NET
and third solution is rewrite your static class as follow:
public static class SessionManager
{
public static volatile List<int> activeSessionsPCIDs;
public static volatile List<int> sessionsThatChangedStatus;
public static volatile List<SessionObject> allSessions;
static SessionManager()
{
Initialize();
}
public static void Initialize() {
var t = Task.Factory.StartNew(() => {
while(true)
{
SetProperties();
Task.Delay(5);
}
});
}
public static void SetProperties() {
SessionDataAccess sd = new SessionDataAccess();
while (true) {
allSessions = sd.GetAllSessions();
activeSessionsPCIDs = new List<int>();
sessionsThatChangedStatus = new List<int>();
foreach (SessionObject session in allSessions) {
if (session.status == 1) { //if session is active
activeSessionsPCIDs.Add(session.pcid);
}
if (session.status != session.prevStat) { //if current status doesn't match the previous status
sessionsThatChangedStatus.Add(session.pcid);
}
}
Thread.Sleep(5000);
}
}

This is a solution that is a change in approach, but I kept the solution in Web Forms, to make it more directly applicable to your use case.
SignalR is a technology that enables real-time, two way communication between server and clients (browsers), which can replace your static session data class. Below, I have implemented a simple example to demonstrate the concept.
As a sample, create a new ASP.NET Web Forms application and add the SignalR package from nuget.
Install-Package Microsoft.AspNet.SignalR
You will need to add a new Owin Startup class and add these 2 lines:
using Microsoft.AspNet.SignalR;
... and within the method
app.MapSignalR();
Add some UI elements to Default.aspx:
<div class="jumbotron">
<H3 class="MyName">Loading...</H3>
<p class="stats">
</p>
</div>
Add the following JavaScript to the Site.Master. This code references signalr, and implement client-side event handlers and initiates contact with the signalr hub from the browser. here's the code:
<script src="Scripts/jquery.signalR-2.2.0.min.js"></script>
<script src="signalr/hubs"></script>
<script >
var hub = $.connection.sessiondata;
hub.client.someOneJoined = function (name) {
var current = $(".stats").text();
current = current + '\nuser ' + name + ' joined.';
$(".stats").text(current);
};
hub.client.myNameIs = function (name) {
$(".MyName").text("Your user id: " + name);
};
$.connection.hub.start().done(function () { });
</script>
Finally, add a SignalR Hub to the solution and use this code for the SessionDataHub implementation:
[HubName("sessiondata")]
public class SessionDataHub : Hub
{
private ObservableCollection<string> sessions = new ObservableCollection<string>();
public SessionDataHub()
{
sessions.CollectionChanged += sessions_CollectionChanged;
}
private void sessions_CollectionChanged(object sender, NotifyCollectionChangedEventArgs e)
{
if (e.Action == NotifyCollectionChangedAction.Add)
{
Clients.All.someOneJoined(e.NewItems.Cast<string>().First());
}
}
public override Task OnConnected()
{
return Task.Factory.StartNew(() =>
{
var youAre = Context.ConnectionId;
Clients.Caller.myNameIs(youAre);
sessions.Add(youAre);
});
}
public override Task OnDisconnected(bool stopCalled)
{
// TODO: implement this as well.
return base.OnDisconnected(stopCalled);
}
}
For more information about SignalR, go to http://asp.net/signalr
Link to source code: https://lsscloud.blob.core.windows.net/downloads/WebApplication1.zip

Locking with Constructors causing Null Reference?

So I encountered a strange issue today - I had a simple creation of an instance inside the critical section of a lock, and it would throw a null reference exception when I manually dragged the next line to execute. To illustrate:
public class SearchEngineOptimizationParser
{
protected static ConcurrentDictionary<string, SearchEngineOptimizationInfo> _referralInformation = null;
protected static DateTime _lastRecordingDate;
protected static object _lockRecordingObject = new object();
protected static Dictionary<string, string> _searchProviderLookups = null;
static SearchEngineOptimizationParser()
{
_referralInformation = new ConcurrentDictionary<string, SearchEngineOptimizationInfo>();
_lastRecordingDate = DateTime.Now;
_searchProviderLookups = new Dictionary<string, string>();
_searchProviderLookups.Add("google.com", "q");
_searchProviderLookups.Add("yahoo.com", "p");
_searchProviderLookups.Add("bing.com", "q");
}
public SearchEngineOptimizationParser()
{
}
public virtual void ParseReferrer(Uri requestUrl, NameValueCollection serverVariables, ISession session)
{
string corePath = requestUrl.PathAndQuery.SmartSplit('?')[0].ToLower();
string referrer = serverVariables["HTTP_REFERER"];
if (!string.IsNullOrWhiteSpace(referrer))
{
NameValueCollection queryString = HttpUtility.ParseQueryString(referrer);
string dictionaryKey = session.AffiliateID + "|" + corePath;
foreach (var searchProvider in _searchProviderLookups)
{
if (referrer.Contains(searchProvider.Key))
{
if (queryString[searchProvider.Value] != null)
{
string keywords = queryString[searchProvider.Value];
SearchEngineOptimizationInfo info = new SearchEngineOptimizationInfo
{
Count = 1,
CorePath = corePath,
AffiliateId = session.AffiliateID,
Keywords = keywords
};
_referralInformation.AddOrUpdate(dictionaryKey, info, (key, oldValue) =>
{
oldValue.Count++;
return oldValue;
});
break;
}
}
}
}
if (DateTime.Now > _lastRecordingDate.AddHours(1))
{
lock (_lockRecordingObject)
{
if (DateTime.Now > _lastRecordingDate.AddHours(1))
{
SearchEngineKeywordRepository repository = new SearchEngineKeywordRepository();
List<KeyValuePair<string, SearchEngineOptimizationInfo>> currentInfo = _referralInformation.ToList();
Action logData = () =>
{
foreach (var item in currentInfo)
repository.LogKeyword(item.Value);
};
Thread logThread = new Thread(new ThreadStart(logData));
logThread.Start();
_lastRecordingDate = DateTime.Now;
_referralInformation.Clear();
}
}
}
}
EDIT: Updated Real Object
public class SearchEngineKeywordRepository
{
public virtual void LogKeyword(SearchEngineOptimizationInfo keywordInfo)
{
LogSearchEngineKeywords procedure = new LogSearchEngineKeywords();
procedure.Execute(keywordInfo.CorePath, keywordInfo.AffiliateId, keywordInfo.Keywords, keywordInfo.Count);
}
}
The general pattern being that I want to do this 'something' only every hour (in the context of a website application that gets a lot of traffic). I would breakpoint my first if statement, and then step the next line to execute inside the second if statement. When doing so, the act of initializing the SomeObject instance would cause a null reference exception. It had a completely 100% default constructor - I didn't even specify one.
However, when I let the code go through naturally, it would execute without problem. For some reason, it seems that when I skipped over the lock call into the critical section to just test run that code, it caused all kinds of errors.
I'm curious to know why that is; I understand the lock keyword is just syntactic sugar for a Monitor.Enter(o) try / finally block, but that seems to be that when invoking the constructor, something else was happening.
Anyone have any ideas?
EDIT: I've added the actual code to this. I'm able to reproduce this at will, but I still don't understand why this is happening. I've tried copying this code to another solution and the problem does not seem to occur.

I've tried to reproduce your situation, but as I expected I could not. I've tried both the 2.0 and 4.0 runtime, in 32 and 64 bit mode (debugging sometimes behaves differently under x64).
Is the code shown a simplification? Have you checked all your assumptions? I understand you're skipping 3 lines of code, both the if statements and the lock? In that case, even setting the lock object to null does not cause the exception you describe.
(Having _lockRecordingObject set to null causes an ArgumentNullException when leaving the lock scope)