Develop Reference

Scaling Akka.Net

We use the Akka.net Framework for highly scaling applications in the energy sector.
We use Akka.net for various tasks, mostly in the following form:
var system=ActorSystem.Create("actorSystem");
var props=Props.Create<UpdateActor>();
.WithRouter(new SmallesMailboxPool(100));
var actorRef=system.ActorOf(props,"UpdateActor");
foreach(var timerow in timeRowList)
actorRef.Tell(timerow)
Unfortunately the Akka.net framework scales very badly in many cases.
The CPU load is only 12%.
Obviously only one thread or a few threads are used.
How can you configure Akka.Net to use multiple threads for processing the actors?

This is an educated guess, but if you're using SmallestMailboxPool, keep in mind that it works pretty badly with non-blocking I/O and terribly with stashing.
First thing is usually to check, if there are no blocking operations (like synchronous I/O, calling AsyncMethod().Result or Thread.Sleep), which will block current thread, effectively preventing it from being used by other actors.
Another issue is very specific to smallest mailbox router, and it's related to stashing and persistent actors.
Stashing
Stashing is one of the popular ways to work with multi-step operations. This pattern can be represented as bellow.
public class MyActor : ActorBase, IWithUnboundedStash
{
public IStash Stash { get; set; }
public Receive Active(State workUnit) => message =>
{
switch(message)
{
case DoWork:
// stash all messages not related to current work
Stash.Stash(message);
return true;
case WorkDone done:
// when current unit of work is done, unstash pending messages
Stash.UnstashAll();
Become(Idle);
return true;
}
};
public bool Idle(object message)
{
switch(message)
{
case DoWork work:
StartWork(work.State);
Become(Active(work.State)); //continue work in new behavior
return true;
default:
return false;
}
}
public bool Receive(object message) => Idle(message);
}
This case is pretty common i.e. persistent actors use it during their recovery process. Problem is that, it's cleaning up the mailbox, which gives SmallestMailbox routers a false sense that this actor's mailbox is empty, while in practice it's just stashing all incoming message.
This is also a reason why peristent actors should not be routed using SmallestMailbox routers! Tbh. I cannot think of any scenario where putting persistent actors behind router of any kind is a valid option.

I think what you need to do is create an an actor coordinator class, and inside create a list/dictionary of actors. then (for what i understand) they should be working in parallel after you tell the coordinator about new updates.
public class UpdateCoordinator : ReceiveActor
{
//all your update referenced
private readonly Dictionary<int, IActorRef> _updates;
public UpdateCoordinator()
{
_updates = new Dictionary<int, IActorRef>();
//create the update reference
Receive<UpdateMessage>(updateMessage =>
{
//add to the list of actors
CreateUpdateReferenceIfNotExists(updateMessage.Identifier);
IActorRef childUpdateRef = _updates[updateMessage.Identifier];
//start your update actor
childUpdateRef.Tell(updateMessage);
});
}
private void CreateUpdateReferenceIfNotExists(int identifier)
{
if (!_updates.ContainsKey(identifier))
{
IActorRef newChildUpdateRef = Context.ActorOf(Props.Create(()=> new UpdateActor(identifier)), $"Update_{identifier}");
_updates.Add(identifier, newChildUpdateRef);
}
}
}

c# How to load test a webservice

I need to test if there's any memory leak in our application and monitor to see if memory usage increases too much while processing the requests.
I'm trying to develop some code to make multiple simultaneous calls to our api/webservice method. This api method is not asynchronous and takes some time to complete its operation.
I've made a lot of research about Tasks, Threads and Parallelism, but so far I had no luck. The problem is, even after trying all the below solutions, the result is always the same, it appears to be processing only two requests at the time.
Tried:
-> Creating tasks inside a simple for loop and starting them with and without setting them with TaskCreationOptions.LongRunning
-> Creating threads inside a simple for loop and starting them with and without high priority
-> Creating a list of actions on a simple for loop and starting them using
Parallel.Foreach(list, options, item => item.Invoke)
-> Running directly inside a Parallel.For loop (below)
-> Running TPL methods with and without Options and TaskScheduler
-> Tried with different values for MaxParallelism and maximum threads
-> Checked this post too, but it didn't help either. (Could I be missing something?)
-> Checked some other posts here in Stackoverflow, but with F# solutions that I don't know how to properly translate them to C#. (I never used F#...)
(Task Scheduler class taken from msdn)
Here's the basic structure that I have:
public class Test
{
Data _data;
String _url;
public Test(Data data, string url)
{
_data = data;
_url = url;
}
public ReturnData Execute()
{
ReturnData returnData;
using(var ws = new WebService())
{
ws.Url = _url;
ws.Timeout = 600000;
var wsReturn = ws.LongRunningMethod(data);
// Basically convert wsReturn to my method return, with some logic if/else etc
}
return returnData;
}
}
sealed class ThreadTaskScheduler : TaskScheduler, IDisposable
{
// The runtime decides how many tasks to create for the given set of iterations, loop options, and scheduler's max concurrency level.
// Tasks will be queued in this collection
private BlockingCollection<Task> _tasks = new BlockingCollection<Task>();
// Maintain an array of threads. (Feel free to bump up _n.)
private readonly int _n = 100;
private Thread[] _threads;
public TwoThreadTaskScheduler()
{
_threads = new Thread[_n];
// Create unstarted threads based on the same inline delegate
for (int i = 0; i < _n; i++)
{
_threads[i] = new Thread(() =>
{
// The following loop blocks until items become available in the blocking collection.
// Then one thread is unblocked to consume that item.
foreach (var task in _tasks.GetConsumingEnumerable())
{
TryExecuteTask(task);
}
});
// Start each thread
_threads[i].IsBackground = true;
_threads[i].Start();
}
}
// This method is invoked by the runtime to schedule a task
protected override void QueueTask(Task task)
{
_tasks.Add(task);
}
// The runtime will probe if a task can be executed in the current thread.
// By returning false, we direct all tasks to be queued up.
protected override bool TryExecuteTaskInline(Task task, bool taskWasPreviouslyQueued)
{
return false;
}
public override int MaximumConcurrencyLevel { get { return _n; } }
protected override IEnumerable<Task> GetScheduledTasks()
{
return _tasks.ToArray();
}
// Dispose is not thread-safe with other members.
// It may only be used when no more tasks will be queued
// to the scheduler. This implementation will block
// until all previously queued tasks have completed.
public void Dispose()
{
if (_threads != null)
{
_tasks.CompleteAdding();
for (int i = 0; i < _n; i++)
{
_threads[i].Join();
_threads[i] = null;
}
_threads = null;
_tasks.Dispose();
_tasks = null;
}
}
}
And the test code itself:
private void button2_Click(object sender, EventArgs e)
{
var maximum = 100;
var options = new ParallelOptions
{
MaxDegreeOfParallelism = 100,
TaskScheduler = new ThreadTaskScheduler()
};
// To prevent UI blocking
Task.Factory.StartNew(() =>
{
Parallel.For(0, maximum, options, i =>
{
var data = new Data();
// Fill data
var test = new Test(data, _url); //_url is pre-defined
var ret = test.Execute();
// Check return and display on screen
var now = DateTime.Now.ToString("HH:mm:ss");
var newText = $"{Environment.NewLine}[{now}] - {ret.ReturnId}) {ret.ReturnDescription}";
AppendTextBox(newText, ref resultTextBox);
}
}
public void AppendTextBox(string value, ref TextBox textBox)
{
if (InvokeRequired)
{
this.Invoke(new ActionRef<string, TextBox>(AppendTextBox), value, textBox);
return;
}
textBox.Text += value;
}
And the result that I get is basically this:
[10:08:56] - (0) OK
[10:08:56] - (0) OK
[10:09:23] - (0) OK
[10:09:23] - (0) OK
[10:09:49] - (0) OK
[10:09:50] - (0) OK
[10:10:15] - (0) OK
[10:10:16] - (0) OK
etc
As far as I know there's no limitation on the server side. I'm relatively new to the Parallel/Multitasking world. Is there any other way to do this? Am I missing something?
(I simplified all the code for clearness and I believe that the provided code is enough to picture the mentioned scenarios. I also didn't post the application code, but it's a simple WinForms screen just to call and show results. If any code is somehow relevant, please let me know, I can edit and post it too.)
Thanks in advance!
EDIT1: I checked on the server logs that it's receiving the requests two by two, so it's indeed something related to sending them, not receiving.
Could it be a network problem/limitation related to how the framework manages the requests/connections? Or something with the network at all (unrelated to .net)?
EDIT2: Forgot to mention, it's a SOAP webservice.
EDIT3: One of the properties that I send (inside data) needs to change for each request.
EDIT4: I noticed that there's always an interval of ~25 secs between each pair of request, if it's relevant.

I would recommend not to reinvent the wheel and just use one of the existing solutions:
Most obvious choice: if your Visual Studio license allows you can use MS Load Testing Framework, most likely you won't even have to write a single line of code: How to: Create a Web Service Test
SoapUI is a free and open source web services testing tool, it has some limited load testing capabilities
If for some reasons SoapUI is not suitable (i.e. you need to run load tests in clustered mode from several hosts or you need more enhanced reporting) you can use Apache JMeter - free and open source multiprotocol load testing tool which supports web services load testing as well.

A good solution to create load tests without write a own project is use this service https://loader.io/targets
It is free for small tests, you can POST Parameters, Header,... and you have a nice reporting.

Isnt the "two requests at a time" the result of the default maxconnection=2 limit on connectionManagement?
<configuration>
<system.net>
<connectionManagement>
<add address = "http://www.contoso.com" maxconnection = "4" />
<add address = "*" maxconnection = "2" />
</connectionManagement>
</system.net>
</configuration>

My favorite load testing library is NBomber. It has an easy and powerful API, realistic user simulations, and provides you with nice HTML reports about latency and requests per second.
I used it to test my API and wrote an article about how I did it.

Creating a buffer for Consumer and Producer threads using Queue c# .NET

I am writing a windows service application that is capable of collecting data from sensors like temperature, pressure volume etc...
The frequency at which the data is read is pretty high, there could be a hundred sensors and the data being received could be at a frequency could be one per second per sensor..
I need to store this data to an oracle database, for obvious reasons i dont want to hit the database at such a high rate.
Hence i want to create a Buffer.
My plan is to create a Buffer using the standard .NET Queue, a few threads keep Enqueue data into the queue and another timer driven thread can keep writing into the database at regular intervals.
What i want to know is..?? Is This thread safe
If this is not, what is the best way of creating a in memory buffer

To answer your question, as long as you lock accesses, you can have multiple threads access a regular queue.
For me though, I didn't use that and wanted to use queues with locks to keep them thread safe. I have been doing this in c# for one of my programs. I just use a regular queue, and then put a locker on accesses to it (enqueue, dequeue, count). It is completely thread safe if you just lock the accesses.
My setup comes from the tutorial/example here: http://www.albahari.com/threading/part2.aspx#_ProducerConsumerQWaitHandle
My situation is a little different than yours, but pretty similar. For me, my data can come in very fast, and if I don't queue it I lose the data if multiple come in at the same time. Then I have a thread running that slowly takes items off the queue and processes them. This hand-off uses an AutoResetEvent to hold my working-thread until data is ready to be processed. In your case you would use a timer or something that happens regularly.
I copy/pasted my code and tried to change the names. Hopefully I didn't completely break it by missing some name changes, but you should be able to get the gist.
public class MyClass : IDisposable
{
private Thread sensorProcessingThread = null;
private Queue<SensorData> sensorQueue = new Queue<SensorData>();
private readonly object _sensorQueueLocker = new object();
private EventWaitHandle _whSensorEvent = new AutoResetEvent(false);
public MyClass () {
sensorProcessingThread = new Thread(sensorProcessingThread_DoWork);
sensorProcessingThread.Start();
}
public void Dispose()
{
// Signal the end by sending 'null'
EnqueueSensorEvent(null);
sensorProcessingThread.Join();
_whSensorEvent.Close();
}
// The fast sensor data comes in, locks queue, and then
// enqueues the data, and releases the EventWaitHandle
private void EnqueueSensorEvent( SensorData wd )
{
lock ( _sensorQueueLocker )
{
sensorQueue.Enqueue(wd);
_whSensorEvent.Set();
}
}
// When asynchronous events come in, I just throw them into queue
private void OnSensorEvent( object sender, MySensorArgs e )
{
EnqueueSensorEvent(new SensorData(sender, e));
}
// I have several types of events that can come in,
// they just get packaged up into the same "SensorData"
// struct, and I worry about the contents later
private void FileSystem_Changed( object sender, System.IO.FileSystemEventArgs e )
{
EnqueueSensorEvent(new SensorData(sender, e));
}
// This is the slower process that waits for new SensorData,
// and processes it. Note, if it sees 'null' as data,
// then it knows it should quit the while(true) loop.
private void sensorProcessingThread_DoWork( object obj )
{
while ( true )
{
SensorData wd = null;
lock ( _sensorQueueLocker )
{
if ( sensorQueue.Count > 0 )
{
wd = sensorQueue.Dequeue();
if ( wd == null )
{
// Quit the loop, thread finishes
return;
}
}
}
if ( wd != null )
{
try
{
// Call specific handlers for the type of SensorData that was received
if ( wd.isSensorDataType1 )
{
SensorDataType1_handler(wd.sender, wd.SensorDataType1Content);
}
else
{
FileSystemChanged_handler(wd.sender, wd.FileSystemChangedContent);
}
}
catch ( Exception exc )
{
// My sensor processing also has a chance of failing to process completely, so I have a retry
// methodology that gives up after 5 attempts
if ( wd.NumFailedUpdateAttempts < 5 )
{
wd.NumFailedUpdateAttempts++;
lock ( _sensorQueueLocker )
{
sensorQueue.Enqueue(wd);
}
}
else
{
log.Fatal("Can no longer try processing data", exc);
}
}
}
else
_whWatchEvent.WaitOne(); // No more tasks, wait for a signal
}
}
Something you could maybe look at is Reactive (Rx) for .net from Microsoft. Check out: https://msdn.microsoft.com/en-us/data/gg577611.aspx and at the bottom of page is a pdf tutorial "Curing the asynchronous blues": http://go.microsoft.com/fwlink/?LinkId=208528 This is something very different but maybe you will see something you like.

Missed Garbage Collection Notifications

We are running a web farm using .NET. Each web server holds a considerable amount of static objects in it's memory. A Gen 2 garbage collection (GC) takes 10-20 seconds and it runs every 5 minutes.
We ran more or less into the same problems that StackOverflow ran into: http://samsaffron.com/archive/2011/10/28/in-managed-code-we-trust-our-recent-battles-with-the-net-garbage-collector
At the moment, we are reducing the number of objects in the cache. However, this takes time.
At the same time, we implemented the methods documented here to get notifications in .NET about approaching GCs.
The goal is to take a web server out of the farm when a GC is approaching and include it into the farm after the GC is over.
However, we only get a notification for 0.7% of all GCs.
We are using a maxGenerationThreshold and largeObjectHeapThreshold of 8. We tried other thresholds, but the amount of missed GCs did not change.
We are using concurrent server garbage collection (http://msdn.microsoft.com/en-us/library/ms229357.aspx). The GCLatencyMode is Interactive (see http://msdn.microsoft.com/en-us/library/system.runtime.gclatencymode.aspx). Here again, we tried to use other GC modes (Workstation mode, Batch, etc.). And again we did not get a notification for most of the GCs.
Are we doing something wrong, or is it impossible to get a notification for every GC that occurs?
How can we increase the number of notifications?
According to http://assets.red-gate.com/community/books/assets/Under_the_Hood_of_.NET_Management.pdf, at the beginning a GC is triggered when Gen2 hits ~10 MB. We have a lot of RAM, so if we could set this threshold manually to a higher level, it would take more time to reach this threshold and in my understanding the probability would increase to get a notification. Is there a way to modify this threshold?
This is the code that registers and listens to the notifications:
GC.RegisterForFullGCNotification(gcThreshold, gcThreshold);
// Start a thread using WaitForFullGCProc.
thWaitForFullGC = new Thread(WaitForFullGCProc);
thWaitForFullGC.Name = "HealthTestGCNotificationListenerThread (Threshold=" + gcThreshold + ")";
thWaitForFullGC.IsBackground = true;
WaitForFullGCProc():
private void WaitForFullGCProc()
{
try
{
while (!gcAbort)
{
// Check for a notification of an approaching collection.
GCNotificationStatus s;
do
{
int timeOut = CheckForMissedGc() > 0 ? 5000 : (10 * 60 * 1000);
s = GC.WaitForFullGCApproach(timeOut);
if (this.GcState == GCState.InducedUnnotified)
{
// Set the GcState back to okay to prevent the message from staying in the ApplicationMonitoring.
this.GcState = GCState.Okay;
}
} while (s == GCNotificationStatus.Timeout);
if (s == GCNotificationStatus.Succeeded)
{
SetGcState(GCState.Approaching, "GC is approaching..");
gcApproachNotificationCount++;
}
else
{
...
}
Stopwatch stopwatch = Stopwatch.StartNew();
s = GC.WaitForFullGCComplete((int)PrewarnTime.TotalMilliseconds);
long elapsed = stopwatch.ElapsedMilliseconds;
if (s == GCNotificationStatus.Timeout)
{
if (this.ForceGCWhenApproaching && !this.IsInGc && !this.IsPeriodicGcApproaching)
{
this.IsInGc = true;
GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced, blocking: true);
GC.WaitForPendingFinalizers();
elapsed = stopwatch.ElapsedMilliseconds;
this.IsInGc = false;
}
}
}
gcAbort = false;
}
catch (Exception e)
{
}
}

Note: This is more of a comment but includes a large code sample.
Have you considered trying to get your GC notifications through another way? Jeffrey Richter ( CLR via C#) explains a good way to get notifications,it uses an object and checks its finalizer method in what generation it is.
This is the class: It uses internal objects which are either collected if the supplied generation matches ( See new GenObject(0); for example. ) or resurrected for the next higher generation.
And you just subscribe to it with GCNotification.GCDone += GCDoneHandler;
public static class GCNotification
{
private static Action<Int32> s_gcDone = null; // The event's field
public static event Action<Int32> GCDone
{
add
{
// If there were no registered delegates before, start reporting notifications now
if (s_gcDone == null) { new GenObject(0); new GenObject(1); new GenObject(2); }
s_gcDone += value;
}
remove { s_gcDone -= value; }
}
private sealed class GenObject
{
private Int32 m_generation;
public GenObject(Int32 generation) { m_generation = generation; }
~GenObject()
{ // This is the Finalize method
// If this object is in the generation we want (or higher),
// notify the delegates that a GC just completed
if (GC.GetGeneration(this) >= m_generation)
{
Action<Int32> temp = Volatile.Read(ref s_gcDone);
if (temp != null) temp(m_generation);
}
// Keep reporting notifications if there is at least one delegate registered,
// the AppDomain isn't unloading, and the process isn’t shutting down
if ((s_gcDone != null)
&& !AppDomain.CurrentDomain.IsFinalizingForUnload()
&& !Environment.HasShutdownStarted)
{
// For Gen 0, create a new object; for Gen 2, resurrect the object
// & let the GC call Finalize again the next time Gen 2 is GC'd
if (m_generation == 0) new GenObject(0);
else GC.ReRegisterForFinalize(this);
}
else { /* Let the objects go away */ }
}
}
}

Unstable unit test result - mock message queue behavior and parallel loop

I am building a class to use parallel loop to access messages from message queue, in order to explain my issue I created a simplified version of code:
public class Worker
{
private IMessageQueue mq;
public Worker(IMessageQueue mq)
{
this.mq = mq;
}
public int Concurrency
{
get
{
return 5;
}
}
public void DoWork()
{
int totalFoundMessage = 0;
do
{
// reset for every loop
totalFoundMessage = 0;
Parallel.For<int>(
0,
this.Concurrency,
() => 0,
(i, loopState, localState) =>
{
Message data = this.mq.GetFromMessageQueue("MessageQueueName");
if (data != null)
{
return localState + 1;
}
else
{
return localState + 0;
}
},
localState =>
{
Interlocked.Add(ref totalFoundMessage, localState);
});
}
while (totalFoundMessage >= this.Concurrency);
}
}
The idea is to set the worker class a concurrency value to control the parallel loop. If after each loop the number of message to retrieve from message queue equals to the concurrency number I assume there are potential more messages in the queue and continue to fetch from queue until the message number is smaller than the concurrency. The TPL code is also inspired by TPL Data Parallelism Issue post.
I have the interface to message queue and message object.
public interface IMessageQueue
{
Message GetFromMessageQueue(string queueName);
}
public class Message
{
}
Thus I created my unit test codes and I used Moq to mock the IMessageQueue interface
[TestMethod()]
public void DoWorkTest()
{
Mock<IMessageQueue> mqMock = new Mock<IMessageQueue>();
Message data = new Message();
Worker w = new Worker(mqMock.Object);
int callCounter = 0;
int messageNumber = 11;
mqMock.Setup(x => x.GetFromMessageQueue("MessageQueueName")).Returns(() =>
{
callCounter++;
if (callCounter < messageNumber)
{
return data;
}
else
{
// simulate MSMQ's behavior last call to empty queue returns null
return (Message)null;
}
}
);
w.DoWork();
int expectedCallTimes = w.Concurrency * (messageNumber / w.Concurrency);
if (messageNumber % w.Concurrency > 0)
{
expectedCallTimes += w.Concurrency;
}
mqMock.Verify(x => x.GetFromMessageQueue("MessageQueueName"), Times.Exactly(expectedCallTimes));
}
I used the idea from Moq to set up a function return based on called times to set up call times based response.
During the unit testing I noticed the testing result is unstable, if you run it multiple times you will see in most cases the test passes, but occasionally the test fails for various reasons.
I have no clue what caused the situation and look for some input from you. Thanks

The problem is that your mocked GetFromMessageQueue() is not thread-safe, but you're calling it from multiple threads at the same time. ++ is inherently thread-unsafe operation.
Instead, you should use locking or Interlocked.Increment().
Also, in your code, you're likely not going to benefit from parallelism, because starting and stopping Parallel.ForEach() has some overhead. A better way would be to have a while (or do-while) inside the Parallel.ForEach(), not the other way around.

My approach would be to restructure. When testing things like timing or concurrency, it is usually prudent to abstract your calls (in this case, use of PLINQ) into a separate class that accepts a number of delegates. You can then test the correct calls are being made to the new class. Then, because the new class is a lot simpler (only a single PLINQ call) and contains no logic, you can leave it untested.
I advocate not testing in this case because unless you are working on something super-critical (life support systems, airplanes, etc), it becomes more trouble than it's worth to test. Trust the framework will execute the PLINQ query as expected. You should only be testing those things which make sense to test, and that provide value to your project or client.