Develop Reference

EFCore - How to run simple query on a loop?

I'm working on a project where users can queue up actions to happen in the future. Each future action is saved in the database. I'm trying to find a way to get all the actions that have already passed and act on them. Currently I'm doing this:
public class TaskingManager
{
private static readonly System.Timers.Timer RefreshEventsLoop = new(1000);
public void Initialize()
{
RefreshEventsLoop.Elapsed += RefreshEventsLoop_Elapsed;
RefreshEventsLoop.AutoReset = false;
RefreshEventsLoop.Start();
}
private void RefreshEventsLoop_Elapsed(object? sender, System.Timers.ElapsedEventArgs e)
{
RefreshEventsLoop.Enabled = false;
ProcessEventsQueue();
RefreshEventsLoop.Enabled = true;
}
private void ProcessEventsQueue()
{
var timeNow = DateTime.UtcNow;
var context = new GalaxyDbContext(_dbOptions);
var dbFunctions = new DatabaseFunctions(context);
var eventsToProcess = context.FutureEvents
.Where(futureEvent => futureEvent.TriggerTime <= timeNow)
.ToList();
if (eventsToProcess.Any())
{
context.FutureEvents.RemoveRange(eventsToProcess);
context.SaveChanges();
foreach (var pastEvent in eventsToProcess)
{
_messageMap[pastEvent.MessageType].Invoke(dbFunctions).Execute(pastEvent);
}
}
else
{
dbFunctions.CreateFutureScienceTask();
}
}
}
This seems to work ok. But the problem is that after the app has been running for a while, I can see the LINQ part is taking up a huge amount of memory:
524MB used
And if I leave it running for a few days then it's up in the gigs.
I'm guessing running this query every second is wasteful on resources. But I don't know of a better way to do this. Is there a way to continuously check the database for something like this?

The first thing i can see is that you are not disposing the databasecontextt afer you used it.
Read this for more info about entityframework context lifetime.
To properly dispose it use a using statement.
using (var context = new GalaxyDbContext(_dbOptions))
{
//your code that uses the context
}
Or with new using syntax:
using (var context = new GalaxyDbContext(_dbOptions));
//your code that uses the context
Right know the problem could be that you create a context everytime you call the function and never dispose it and it still keeps references to the data etc..

Correct way to check performance of piece of code in C#

Just assume I have two pieces of code and I want to check CPU usage and Memory of these codes and compare together, is this a good way to check performance:
public class CodeChecker: IDisposable
{
public PerformanceResult Check(Action<int> codeToTest, int loopLength)
{
var stopWatch = new Stopwatch();
stopWatch.Start();
for(var i = 0; i < loopLength; i++)
{
codeToTest.Invoke(i);
}
stopWatch.Stop();
var process = Process.GetCurrentProcess();
var result = new PerformanceResult(stopWatch.ElapsedMilliseconds, process.PrivateMemorySize64);
return result;
}
}
public class PerformanceResult
{
public long DurationMilliseconds { get; set; }
public long PrivateMemoryBytes { get; set; }
public PerformanceResult(long durationMilliseconds, long privateMemoryBytes)
{
DurationMilliseconds = durationMilliseconds;
PrivateMemoryBytes = privateMemoryBytes;
}
public override string ToString()
{
return $"Duration: {DurationMilliseconds} - Memory: {PrivateMemoryBytes}";
}
}
And:
static void Main(string[] args)
{
Console.WriteLine("Start!");
int loopLength = 10000000;
var collection = new Dictionary<int, Target>();
PerformanceResult result;
using (var codeChecker = new CodeChecker())
{
result = codeChecker.Check((int i) => collection.Add(i, new Target()) , loopLength);
}
Console.WriteLine($"Dict Performance: {result}");
var list = new List<Target>();
using(var codeChecker = new CodeChecker())
{
result = codeChecker.Check((int i) => list.Add(new Target()), loopLength);
}
Console.WriteLine($"List Performance: {result}");
Console.ReadLine();
}
I'm looking for checking performance programmatically and I want to check piece of code specifically not all the application.
Any suggestion to improve aforementioned code?
And I will open to any suggestion for using free tools.

There are lot of factors which may impose a bias into your measurement including CLR and JIT compiler influence, heap state, cold or hot run, overall load in your system, etc. Ideally you need to isolate the pieces of code you'd like to benchmark from each other to exclude mutual impact, benchmark only hot runs, not cold to exclude JIT compilation and other cold run factors and what is most important you need to conduct multiple runs to obtain statistical information as single run can be not representative especially on a system which implies multitasking. Luckily you don't have to do it everything manually - there is great library for bench-marking which does all things mentioned and much more and which is widely used in various .NET projects.

Creating unique callback references

So the problem is as follows: I'm using a third party client class to issue commands to an external system to retrieve data (fairly standard). The problem is that when I issue commands via an instance of this class, it uses a callback reference based on the function name in my code and the line number within that function from which it was called, it then serializes this and other information into JSON and transmits to the external system for processing, with the data return being identified by the reference. This works "well" until we get to iteration, then the callback reference remains the same and I only receive data for one iteration. The third party isn't going to alter their code so I need a way of generating unique references in my code, but I'm unsure of how I can do this within C#. I can't edit their class as it is provided as a DLL, I can only access this system by using it (it is part of their SDK).
Any ideas greatly appreciated!
Example code:
[Note: actual code is part of a Windows Form Application]
The last part client.IsUserInGroup is the problem.
using thirdParty.Client;
class TestProgram
{
static void Main(string[] args)
{
//Area and user objects defined within third party class
List<Area> Areas = new List<Area>();
List<User> myUsers = new List<User>();
int publicAreaID = 0;
bool isConnected=false;
client.Connect("user", "pass",
(connstatus) =>
{
switch (connstatus)
{
case ConnectionStatus.Success:
isConnected = true;
Console.WriteLine("Connected");
break;
case ConnectionStatus.InvalidCredentials:
Console.WriteLine("InvalidCredentials");
break;
case ConnectionStatus.Timeout:
Console.WriteLine("Timeout");
break;
}
});
if (isConnected)
{
client.GetAreas(
(result) =>
{
Areas = result;
});
//Get ID of public area
foreach (Area myArea in Areas)
{
if (myArea.Name.Equals("Public"))
{
publicAreaID = myArea.ID;
}
}
//Get all keyholders in Public area and store in list
client.GetUsersInArea(publicAreaID,
(result) =>
{
myUsers = result;
});
//Iterate over all users in list and verify they are in the everyone group
foreach (User myUser in myUsers)
{
User tempUser = myUser;
client.IsUserInGroup(tempUser.ID, 0,
(result) =>
{
if (result) //this is a bool
{
//This only returns one result..
Console.WriteLine(String.Format("{0} is in Everyone Group and Public Area", tempUser.Name));
}
});
}
client.Disconnect();
}
}
}
UPDATE
I've been doing more testing by removing the foreach loop and just calling client.IsUserInGroup twice to generate alternative callback references; the results are interesting. As expected there are unique references, but there is still only one result displayed, the only way to get both is to create two User objects rather than reuse one. As mentioned above, the "real" code is used in a Windows forms app, could this be something to do with object referencing? Example code below:
new User tempUser1 = myUsers[0];
client.IsUserInGroup(tempUser1.ID, 0,
(result) =>
{
if (result) //this is a bool
{
Console.WriteLine(String.Format("{0} is in Everyone Group and Public Area", tempUser1.Name));
}
});
new User tempUser2 = myUsers[1];
client.IsUserInGroup(tempUser2.ID, 0,
(result) =>
{
if (result) //this is a bool
{
Console.WriteLine(String.Format("{0} is in Everyone Group and Public Area", tempUser2.Name));
}
});

Answer moved from OP's original question:
Ok so I was playing with this a lot over the last few hours and kind of made it work by keeping the iterative loop but doing two things; firstly I assumed that the third party class would synchronize information requests and not allow my code to continue until it had a result returned - this seemingly is not the case as the output from recursion with an extra Console.WriteLine(iterationCount) in it shows the count increasing with no data returned; therefore I am forced to slow down the code by Thread.Sleep (I'm investigating better ways of doing this). Secondly any code within the lambda that could be moved outside, was. Instead a temp bool outside of the lambda was assigned the value of the result bool. The code looks like:
//Iterate over all users in list and verify they are in the everyone group
foreach (User myUser in myUsers)
{
User tempUser = myUser;
bool tempRes = false;
client.IsUserInGroup(tempUser.ID, 0,
(result) =>
{
tempRes = result;
});
if (tempRes)
{
Console.WriteLine(String.Format("{0} is in Everyone Group and Public Area", tempUser.Name));
}
System.Threading.Thread.Sleep(75); //Not a good way of enforcing sync!
}

Multithreading in a Single Transaction

According to this MSDN article, you should be able to multithread a process with each thread enlisted in a single root transaction.
I created a sample based on that article where I expect atransaction to be rolled-back (bool[] results should be all false in the foreach loop). Unfortunately, this is not the case, and the outcome is predictably unpredictable (run the example enough times and you will see any combination of bool values in the array).
In addition, I've tried both DependentCloneOption.BlockCommitUntilComplete and DependentCloneOption.RollbackIfNotComplete neither of which produce the expected result.
Secondly, I think ThreadPool.QueueUserWorkItem is ugly code at best, and it would be nice to see something like this using Parallel.ForEach instead.
And finally, my question :) Why the heck is this not working? What am I doing wrong? Is it just flat-out impossible to wrap multiple threads in a single transaction?
namespace Playing
{
class Program
{
static bool[] results = new bool[] { false, false, false };
static void Main(string[] args)
{
try
{
using (var outer = new TransactionScope(
TransactionScopeOption.Required))
{
for (var i = 0; i < 3; i++ )
{
ThreadPool.QueueUserWorkItem(WorkerItem,
new Tuple<int, object>(
i, Transaction.Current.DependentClone(
DependentCloneOption.BlockCommitUntilComplete)));
}
outer.Complete();
}
}
catch { /* Suppress exceptions */ }
// Expect all to be false
foreach (var r in results)
Console.WriteLine(r);
}
private static void WorkerItem(object state)
{
var tup = (Tuple<int, object>)state;
var i = tup.Item1;
var dependent = (DependentTransaction)tup.Item2;
using (var inner = new TransactionScope(dependent))
{
// Intentionally throw exception to force roll-back
if (i == 2)
throw new Exception();
results[i] = true;
inner.Complete();
}
dependent.Complete();
}
}
}

Yours results[] members that have been set to true won't magically set themselves back to false (sadly). That's what Transaction Managers do. Look at the EnlistXXX methods to get an idea of what's involved.
Basically, you'll need to compensate in the event of a rollback. For example, you could subscribe to the root Transaction's TransactionCompleted event and check if the transaction was rolled back. If it was you'll need to restore the previous values for the child workers that completed.
You can also handle the TransactionAbortedException thrown that you are suppressing, or handle it at the worker level (see an example of catching it on this page: http://msdn.microsoft.com/en-us/library/ms973865.aspx)
Typically, with in-memory "transactions" you are better off using the Task library to have the workers batch up results and then "commit" them in a continuation of a parent Task. It's easier than messing about with Transactions, which you only need to do if you are coordinating between memory and some other Transaction Manager (like SQL Server or other processes).

Unstable unit test result - mock message queue behavior and parallel loop

I am building a class to use parallel loop to access messages from message queue, in order to explain my issue I created a simplified version of code:
public class Worker
{
private IMessageQueue mq;
public Worker(IMessageQueue mq)
{
this.mq = mq;
}
public int Concurrency
{
get
{
return 5;
}
}
public void DoWork()
{
int totalFoundMessage = 0;
do
{
// reset for every loop
totalFoundMessage = 0;
Parallel.For<int>(
0,
this.Concurrency,
() => 0,
(i, loopState, localState) =>
{
Message data = this.mq.GetFromMessageQueue("MessageQueueName");
if (data != null)
{
return localState + 1;
}
else
{
return localState + 0;
}
},
localState =>
{
Interlocked.Add(ref totalFoundMessage, localState);
});
}
while (totalFoundMessage >= this.Concurrency);
}
}
The idea is to set the worker class a concurrency value to control the parallel loop. If after each loop the number of message to retrieve from message queue equals to the concurrency number I assume there are potential more messages in the queue and continue to fetch from queue until the message number is smaller than the concurrency. The TPL code is also inspired by TPL Data Parallelism Issue post.
I have the interface to message queue and message object.
public interface IMessageQueue
{
Message GetFromMessageQueue(string queueName);
}
public class Message
{
}
Thus I created my unit test codes and I used Moq to mock the IMessageQueue interface
[TestMethod()]
public void DoWorkTest()
{
Mock<IMessageQueue> mqMock = new Mock<IMessageQueue>();
Message data = new Message();
Worker w = new Worker(mqMock.Object);
int callCounter = 0;
int messageNumber = 11;
mqMock.Setup(x => x.GetFromMessageQueue("MessageQueueName")).Returns(() =>
{
callCounter++;
if (callCounter < messageNumber)
{
return data;
}
else
{
// simulate MSMQ's behavior last call to empty queue returns null
return (Message)null;
}
}
);
w.DoWork();
int expectedCallTimes = w.Concurrency * (messageNumber / w.Concurrency);
if (messageNumber % w.Concurrency > 0)
{
expectedCallTimes += w.Concurrency;
}
mqMock.Verify(x => x.GetFromMessageQueue("MessageQueueName"), Times.Exactly(expectedCallTimes));
}
I used the idea from Moq to set up a function return based on called times to set up call times based response.
During the unit testing I noticed the testing result is unstable, if you run it multiple times you will see in most cases the test passes, but occasionally the test fails for various reasons.
I have no clue what caused the situation and look for some input from you. Thanks

The problem is that your mocked GetFromMessageQueue() is not thread-safe, but you're calling it from multiple threads at the same time. ++ is inherently thread-unsafe operation.
Instead, you should use locking or Interlocked.Increment().
Also, in your code, you're likely not going to benefit from parallelism, because starting and stopping Parallel.ForEach() has some overhead. A better way would be to have a while (or do-while) inside the Parallel.ForEach(), not the other way around.

My approach would be to restructure. When testing things like timing or concurrency, it is usually prudent to abstract your calls (in this case, use of PLINQ) into a separate class that accepts a number of delegates. You can then test the correct calls are being made to the new class. Then, because the new class is a lot simpler (only a single PLINQ call) and contains no logic, you can leave it untested.
I advocate not testing in this case because unless you are working on something super-critical (life support systems, airplanes, etc), it becomes more trouble than it's worth to test. Trust the framework will execute the PLINQ query as expected. You should only be testing those things which make sense to test, and that provide value to your project or client.