Develop Reference

How to implement a sorted buffer?

I need to traverse a collection of disjoint folders; each folder is associated to a visited time configurated somewhere in the folder.
I then sort the folders, and process the one with the earliest visited time first. Note the processing is generally slower than the traversing.
My code targets Framework4.8.1; Currently my implementation is as follows:
public class BySeparateThread
{
ConcurrentDictionary<string, DateTime?> _dict = new ConcurrentDictionary<string, DateTime?>();
private object _lock;
/// <summary>
/// this will be called by producer thread;
/// </summary>
/// <param name="address"></param>
/// <param name="time"></param>
public void add(string address,DateTime? time) {
_dict.TryAdd(address, time);
}
/// <summary>
/// called by subscriber thread;
/// </summary>
/// <returns></returns>
public string? next() {
lock (_lock) {
var r = _dict.FirstOrDefault();
//return sortedList.FirstOrDefault().Value;
if (r.Key is null)
{
return r.Key;
}
if (r.Value is null)
{
_dict.TryRemove(r.Key, out var _);
return r.Key;
}
var key = r.Key;
foreach (var item in _dict.Skip(1) )
{
if (item.Value is null)
{
_dict.TryRemove(item.Key, out var _);
return item.Key;
}
if (item.Value< r.Value)
{
r=item;
}
}
_dict.TryRemove(key, out var _);
return key;
}
}
/// <summary>
/// this will be assigned of false by producer thread;
/// </summary>
public bool _notComplete = true;
/// <summary>
/// shared configuration for subscribers;
/// </summary>
fs.addresses_.disjoint.deV_._bak.Io io; //.io_._CfgX.Create(cancel, git)
/// <summary>
/// run this in a separate thread other than <see cref="add(string, DateTime?)"/>
/// </summary>
/// <param name="sln"></param>
/// <returns></returns>
public async Task _asyn_ofAddress(string sln)
{
while (_notComplete)
{
var f = next();
if (f is null )
{
await Task.Delay(30*1000);
//await Task.Yield();
continue;
}
/// degree of concurrency is controlled by a semophore; for instance, at most 4 are tackled:
new dev.srcs.each.sln_.delvable.Bak_srcsInAddresses(io)._startTask_ofAddress(sln);
}
}
}
For the above, I'm concerned about the while(_notComplete) part, as it looks like there would be many loops doing nothing there. I think there should be better ways to remove the while by utilizing the fact that the collection can notify whether it's empty or not at some/various stages such as when we add.
There would be better implementation which can be based on some mature framework such as those being considered by me these days but I often stopped wondering at some implementation details:
BlockingCollection
for this one, I don't know how to make the collection added and sorted dynamically while producer and subscriber are on the run;
Channel
Again, I could not come up with one fitting my need after I read its examples;
Pipeline
I havenot fully understood it;
Rx
I tried to implement an observable and an observer. It only gives me a macroscope framework, but when I get into the details, I ended with what I'm currently doing and I begin to wonder: with what I'm doing, I don't need Rx here.
Dataflow
Shall I implement my own BufferBlock or ActionBlock? It seems the built-in bufferBlock cannot be customized to sort things before releasing them to the next block.
Sorting buffered Observables seems similar to my problem; but it ends with a solution similar to the one I currently have but am not satisfied with, as stated in the above.
Could some one give me a sample code? Please give as concrete code as you can; As you can see, I have researched some general ideas/paths and finally what stops me short is the details, which are often glossed over in some docs.

I just found one solution which is better than my current one. I believe there are some even better ones, so please do post your answers if you find some; my current one is just what I can hack for what I know so far.
I found Prioritized queues in Task Parallel Library, and I write a similar one for my case:
using System;
using System.Collections;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Reactive.Subjects;
using System.Threading;
using System.Threading.Tasks;
namespace nilnul.dev.srcs.every.slns._bak
{
public class BySortedSet : IProducerConsumerCollection<(string, DateTime)>
{
private class _Comparer : IComparer<(string, DateTime)>
{
public int Compare((string, DateTime) first, (string, DateTime) second)
{
var returnValue = first.Item2.CompareTo(second.Item2);
if (returnValue == 0)
returnValue = first.Item1.CompareTo(second.Item1);
return returnValue;
}
static public _Comparer Singleton
{
get
{
return nilnul._obj.typ_.nilable_.unprimable_.Singleton<_Comparer>.Instance;// just some magic to get an instance
}
}
}
SortedSet<(string, DateTime)> _dict = new SortedSet<(string, DateTime)>(
_Comparer.Singleton
);
private object _lock=new object();
public int Count
{
get
{
lock(_lock){
return _dict.Count;
}
}
}
public object SyncRoot => _lock;
public bool IsSynchronized => true;
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
//throw new NotImplementedException();
}
public void CopyTo((string, DateTime)[] array, int index)
{
lock (_lock)
{
foreach (var item in _dict)
{
array[index++] = item;
}
}
}
public void CopyTo(Array array, int index)
{
lock (_lock)
{
foreach (var item in _dict)
{
array.SetValue(item, index++);
}
}
}
public bool TryAdd((string, DateTime) item)
{
lock (_lock)
{
return _dict.Add(item);
}
}
public bool TryTake(out (string, DateTime) item)
{
lock (_lock)
{
item = _dict.Min;
if (item==default)
{
return false;
}
return _dict.Remove(item);
}
}
public (string, DateTime)[] ToArray()
{
lock (_lock)
{
return this._dict.ToArray();
}
}
public IEnumerator<(string, DateTime)> GetEnumerator()
{
return ToArray().AsEnumerable().GetEnumerator();
}
/// <summary>
/// </summary>
/// <returns></returns>
public BlockingCollection<(string, DateTime)> asBlockingCollection() {
return new BlockingCollection<(string, DateTime)>(
this
);
}
}
}
Then I can use that like:
static public void ExampleUse(CancellationToken cancellationToken) {
var s = new BySortedSet().asBlockingCollection();
/// traversal thread:
s.Add(("", DateTime.MinValue));
//...
s.CompleteAdding();
/// tackler thread:
///
foreach (var item in s.GetConsumingEnumerable(cancellationToken))
{
/// process the item;
/// todo: degree of parallelism is controlled by the tackler, or is there a better way like in dataflow or Rx or sth else?
}
}
Thanks!

FluentValidation preload data

I want validate a request model with some ids. I try to preload all required data with a bulk request.
The problem is the RuleForEach inside my WhereAsync is called before the LoadUserGroupsAsync is done or started. I start the validation with TestValidateAsync(request).
Is there a better solution for this I have unfortunately not found any solutions for it. Also I have no access to the model from outside a RuleFor, RuleForEach, Where, ...
private readonly List<UserGroup> _userGroups;
WhenAsync(async (request, cancellationToken) => await this.LoadUserGroupsAsync(request.Items, cancellationToken), () =>
{
RuleForEach(o => o.Items).SetValidator(new UserUpdateValidator(this._userGroups));
});
private async Task<bool> LoadUserGroupsAsync(UserUpdateDto[] userUpdates, CancellationToken cancellationToken)
{
var ids = userUpdates.Select(o => o.userGroupId);
this._userGroups = await this._userGroupService.GetByIdsAsync(ids, cancellationToken);
return true;
}
public class UserUpdateValidator : AbstractValidator<UserUpdateDto>
{
public UserUpdateValidator(
UserGroup[] groups)
{
RuleFor(item => item.UserGroupId).Must(userGroupId =>
{
var group = groups.SingleOrDefault(o => o.Id == userGroupId);
if (group == null)
{
return false;
}
return true;
}).WithMessage("Group is invalid");
RuleFor(item => item.UserGroupId).Must(userGroupId =>
{
var group = groups.SingleOrDefault(o => o.Id == userGroupId);
return group.Active;
}).WithMessage("Group is inactive");
RuleFor(item => item.Password).Must((context, password) =>
{
var group = groups.SingleOrDefault(o => o.Id == context.UserGroupId);
if (group.Permissions.Contains("AllowPasswordChange"))
{
return true;
}
return false;
}).WithMessage("It is now allowed to change the password for your user");
}
}
Update 2021-04-28 - Add more Informations to example

You can use lazy loading and a wrapper object for the users. This would require calling sync methods instead of async methods to load the users, however.
You could use a Lazy<IEnumerable<User>> object, but that would probably require refactoring your child validator. I like creating a wrapper class for Lazy<IEnumerable<User>> just to make the code backwards-compatible for any other code accepting IEnumerable<T> objects:
/// <summary>
/// Represents a lazy loaded enumerable of type T
/// </summary>
public class LazyEnumerable<T> : IEnumerable<T>
{
private readonly Lazy<IEnumerable<T>> items;
/// <summary>
/// Initializes a new lazy loaded enumerable.
/// </summary>
/// <param name="itemFactory">A lambda expression that returns the items to be iterated over.</param>
public LazyEnumerable(Func<IEnumerable<T> itemFactory)
{
items = new Lazy<T>>(itemFactory);
}
/// <summary>
/// Initializes a new lazy loaded enumerable.
/// </summary>
/// <param name="itemFactory">The Lazy<T> object used to lazily retrieve the items to be iterated over.</param>
public LazyEnumerable(Lazy<IEnumerable<T>> items)
{
this.items = items;
}
public IEnumerator<T> GetEnumerator()
{
return items.Value.GetEnumerator();
}
IEnumerator IEnumerable.GetEnumerator()
{
return items.Value.GetEnumerator();
}
}
This is actually a general purpose class that can be used for any type. I wish .NET had this class in its base library, to be honest. I tend to copy and paste this in to every project, because it is so easy to use. In any event...
Then modify your validator class. Since you did not post enough code for your validator, I took a few guesses about the names of things and structure of your code:
public class YourValidator : AbstractValidator<X>
{
private UserService _userService;
private IEnumerable<UserGroup> _userGroups;
public YourValidator(UserService userService)
{
_userService = userService;
When((request, cancellationToken) => PreloadUserGroups(request.Items, cancellationToken), () =>
{
RuleForEach(o => o.Items).SetValidator(new UserUpdateValidator(_userGroups));
});
}
private bool PreloadUserGroups(UserUpdateDto[] userUpdates, CancellationToken cancellationToken)
{
var ids = userUpdates.Select(o => o.userGroupId);
_userGroups = new LazyEnumerable<UserGroup>(() => _userService.GetByIds(ids, cancellationToken));
return true;
}
}
This will lazy load the users, and since you pass the same object to all child validators it will load the users only once, regardless of how many times the collection is iterated.
Lastly, modify your child validator class to accept an IEnumerable<UserGroup> object instead of an array:
public class UserUpdateValidator : AbstractValidator<UserUpdateDto>
{
public UserUpdateValidator(IEnumerable<UserGroup> groups)

Reactive services. Grouping and caching streams

New: Entire source code with tests is now at https://github.com/bboyle1234/ReactiveTest
Let's imagine we have a view state object that is able to be updated by small partial view change events. Here are some example models of the total view, the incremental view update events and the accumulator function Update that builds the total view:
interface IDeviceView : ICloneable {
Guid DeviceId { get; }
}
class DeviceTotalView : IDeviceView {
public Guid DeviceId { get; set; }
public int Voltage { get; set; }
public int Currents { get; set; }
public object Clone() => this.MemberwiseClone();
}
class DeviceVoltagesUpdateView : IDeviceView {
public Guid DeviceId { get; set; }
public int Voltage { get; set; }
public object Clone() => this.MemberwiseClone();
}
class DeviceCurrentsUpdateView : IDeviceView {
public Guid DeviceId { get; set; }
public int Current { get; set; }
public object Clone() => this.MemberwiseClone();
}
class DeviceUpdateEvent {
public DeviceTotalView View;
public IDeviceView LastUpdate;
}
static DeviceUpdateEvent Update(DeviceUpdateEvent previousUpdate, IDeviceView update) {
if (update.DeviceId != previousUpdate.View.DeviceId) throw new InvalidOperationException("Device ids do not match (numskull exception).");
var view = (DeviceTotalView)previousUpdate.View.Clone();
switch (update) {
case DeviceVoltagesUpdateView x: {
view.Voltage = x.Voltage;
break;
}
case DeviceCurrentsUpdateView x: {
view.Currents = x.Current;
break;
}
}
return new DeviceUpdateEvent { View = view, LastUpdate = update };
}
Next, let's imagine we already have an injectable service that is able to produce an observable stream of the small update events for all devices, and that we want to create a service that can produce an aggregated view stream for individual devices.
Here is the interface of the service we want to create:
interface IDeviceService {
/// <summary>
/// Gets an observable that produces aggregated update events for the device with the given deviceId.
/// On subscription, the most recent event is immediately pushed to the subscriber.
/// There can be multiple subscribers.
/// </summary>
IObservable<DeviceUpdateEvent> GetDeviceStream(Guid deviceId);
}
How can I implement this interface and its requirements using the reactive extensions in the System.Reactive v4 library, targeting .netstandard2.0? Here's my boiler code with comments and that's as far as I've been able to get.
class DeviceService : IDeviceService {
readonly IObservable<IDeviceView> Source;
public DeviceService(IObservable<IDeviceView> source) { // injected parameter
/// When injected here, "source" is cold in the sense that it won't produce events until the first time it is subscribed.
/// "source" will throw an exception if its "Subscribe" method is called more than once as it is intended to have only one observer and
/// be read all the way from the beginning.
Source = source;
/// Callers of the "Subscribe" method below will expect data to be preloaded and will expect to be immediately delivered the most
/// recent event. So we need to immediately subscribe to "source" and start preloading the aggregate streams.
/// I'm assuming there is going to need to be a groupby to split the stream by device id.
var groups = source.GroupBy(x => x.DeviceId);
/// Now somehow we need to perform the aggregrate function on each grouping.
/// And create an observable that immediately delivers the most recent aggregated event when "Subscribe" is called below.
}
public IObservable<DeviceUpdateEvent> GetDeviceStream(Guid deviceId) {
/// How do we implement this? The observable that we return must be pre-loaded with the latest update
throw new NotImplementedException();
}
}

You have some weird code in that gist. Here's what I got working:
public class DeviceService : IDeviceService, IDisposable
{
readonly IObservable<IDeviceView> Source;
private readonly Dictionary<Guid, IObservable<DeviceUpdateEvent>> _updateStreams = new Dictionary<Guid, IObservable<DeviceUpdateEvent>>();
private readonly IObservable<(Guid, IObservable<DeviceUpdateEvent>)> _groupedStream;
private readonly CompositeDisposable _disposable = new CompositeDisposable();
public DeviceService(IObservable<IDeviceView> source)
{
Source = source;
_groupedStream = source
.GroupBy(v => v.DeviceId)
.Select(o => (o.Key, o
.Scan(new DeviceUpdateEvent { View = DeviceTotalView.GetInitialView(o.Key), LastUpdate = null }, (lastTotalView, newView) => lastTotalView.Update(newView))
.Replay(1)
.RefCount()
));
var groupSubscription = _groupedStream.Subscribe(t =>
{
_updateStreams[t.Item1] = t.Item2;
_disposable.Add(t.Item2.Subscribe());
});
_disposable.Add(groupSubscription);
}
public void Dispose()
{
_disposable.Dispose();
}
public IObservable<DeviceUpdateEvent> GetDeviceStream(Guid deviceId)
{
/// How do we implement this? The observable that we return must be pre-loaded with the latest update
if(this._updateStreams.ContainsKey(deviceId))
return this._updateStreams[deviceId];
return _groupedStream
.Where(t => t.Item1 == deviceId)
.Select(t => t.Item2)
.Switch();
}
}
The meat here is the _groupedStream piece. You group by DeviceId, as you said, then you use Scan to update state. I also moved Update to a static class and made it an extension method. You'll need an initial state, so I modified your DeviceTotalView class to get that. Modify accordingly:
public class DeviceTotalView : IDeviceView
{
public Guid DeviceId { get; set; }
public int Voltage { get; set; }
public int Currents { get; set; }
public object Clone() => this.MemberwiseClone();
public static DeviceTotalView GetInitialView(Guid deviceId)
{
return new DeviceTotalView
{
DeviceId = deviceId,
Voltage = 0,
Currents = 0
};
}
}
Next, the .Replay(1).Refcount() serves to remember the most recent update then provide that on subscription. We then stuff all of these child observables into a dictionary for easy retrieval on the method call. The dummy subscriptions (_disposable.Add(t.Item2.Subscribe())) are necessary for Replay to work.
In the event that there's an early request for a DeviceId that doesn't yet have an update, we subscribe to the _groupedStream which will wait for the first update, producing that Id's observable, then .Switch subscribes to that child observable.
However, all of this failed against your test code, I'm guessing because of the ConnectableObservableForAsyncProducerConsumerQueue class. I didn't want to debug that, because I wouldn't recommend doing something like that. In general it's not recommended to mix TPL and Rx code. They problems they solve largely overlap and they get in each other's way. So I modified your test code replacing that connectable observable queue thing with a Replay subject.
I also added the test-case for an early request (before an updates for that Device have arrived):
DeviceUpdateEvent deviceView1 = null;
DeviceUpdateEvent deviceView2 = null;
DeviceUpdateEvent deviceView3 = null;
var subject = new ReplaySubject<IDeviceView>();
var id1 = Guid.NewGuid();
var id2 = Guid.NewGuid();
var id3 = Guid.NewGuid();
subject.OnNext(new DeviceVoltagesUpdateView { DeviceId = id1, Voltage = 1 });
subject.OnNext(new DeviceVoltagesUpdateView { DeviceId = id1, Voltage = 2 });
subject.OnNext(new DeviceVoltagesUpdateView { DeviceId = id2, Voltage = 100 });
var service = new DeviceService(subject);
service.GetDeviceStream(id1).Subscribe(x => deviceView1 = x);
service.GetDeviceStream(id2).Subscribe(x => deviceView2 = x);
service.GetDeviceStream(id3).Subscribe(x => deviceView3 = x);
/// I believe there is no need to pause here because the Subscribe method calls above
/// block until the events have all been pushed into the subscribers above.
Assert.AreEqual(deviceView1.View.DeviceId, id1);
Assert.AreEqual(deviceView2.View.DeviceId, id2);
Assert.AreEqual(deviceView1.View.Voltage, 2);
Assert.AreEqual(deviceView2.View.Voltage, 100);
Assert.IsNull(deviceView3);
subject.OnNext(new DeviceVoltagesUpdateView { DeviceId = id2, Voltage = 101 });
Assert.AreEqual(deviceView2.View.Voltage, 101);
subject.OnNext(new DeviceVoltagesUpdateView { DeviceId = id3, Voltage = 101 });
Assert.AreEqual(deviceView3.View.DeviceId, id3);
Assert.AreEqual(deviceView3.View.Voltage, 101);
That passes fine and can be run without async.
Also, as a general tip, I would recommend doing unit tests for Rx code with the Microsoft.Reactive.Testing package, rather than time-gapping things.

A huge thanks to #Shlomo for the answer above.
The implementation given in the accepted answer, whilst a magical education for me, had a couple of issues that also needed to be solved in turn. The first was a threadrace problem, and the second was performance when a large number of devices were in the system. I ended up solving the threadrace AND dramatically improving performance with this modified implementation:
In the constructor, the grouped and scanned device stream is subscribed directly to a BehaviorSubject, which implements the Replay(1).RefCount() functionality required to immediately notify new subscribers of the latest value in the stream.
In the GetDeviceStream method, we continue to use a dictionary lookup to find the device stream, creating a preloaded BehaviorSubject if it doesn't already exist in the dictionary. We have removed the Where search that existed in the previous implementation in the question above. Using the where search caused a threadrace problem that was solved by making the grouped stream replayable. That caused an expontial performance issue. Replacing it with FirstOrDefault reduced the time take by half, and then removing it completely in favor of the GetCreate dictionary technique gave perfect perfomance O(1) instead of O(n2).
GetCreateSubject uses the Lazy proxy object as the dictionary value because the ConcurrentDictionary can sometimes call the Create method more than once for a single key. Supplying a Lazy to the dictionary ensures that the Value property is only called on one of the lazies, and therefore only one BehaviorSubject is created per device.
class DeviceService : IDeviceService, IDisposable {
readonly CompositeDisposable _disposable = new CompositeDisposable();
readonly ConcurrentDictionary<Guid, Lazy<BehaviorSubject<DeviceUpdateEvent>>> _streams = new ConcurrentDictionary<Guid, Lazy<BehaviorSubject<DeviceUpdateEvent>>>();
BehaviorSubject<DeviceUpdateEvent> GetCreateSubject(Guid deviceId) {
return _streams.GetOrAdd(deviceId, Create).Value;
Lazy<BehaviorSubject<DeviceUpdateEvent>> Create(Guid id) {
return new Lazy<BehaviorSubject<DeviceUpdateEvent>>(() => {
var subject = new BehaviorSubject<DeviceUpdateEvent>(DeviceUpdateEvent.GetInitialView(deviceId));
_disposable.Add(subject);
return subject;
});
}
}
public DeviceService(IConnectableObservable<IDeviceView> source) {
_disposable.Add(source
.GroupBy(x => x.DeviceId)
.Subscribe(deviceStream => {
_disposable.Add(deviceStream
.Scan(DeviceUpdateEvent.GetInitialView(deviceStream.Key), DeviceUtils.Update)
.Subscribe(GetCreateSubject(deviceStream.Key)));
}));
_disposable.Add(source.Connect());
}
public void Dispose() {
_disposable.Dispose();
}
public IObservable<DeviceUpdateEvent> GetDeviceStream(Guid deviceId) {
return GetCreateSubject(deviceId).AsObservable();
}
}
[TestMethod]
public async Task Test2() {
var input = new AsyncProducerConsumerQueue<IDeviceView>();
var source = new ConnectableObservableForAsyncProducerConsumerQueue<IDeviceView>(input);
var service = new DeviceService(source);
var ids = Enumerable.Range(0, 100000).Select(i => Guid.NewGuid()).ToArray();
var idsRemaining = ids.ToHashSet();
var t1 = Task.Run(async () => {
foreach (var id in ids) {
await input.EnqueueAsync(new DeviceVoltagesUpdateView { DeviceId = id, Voltage = 1 });
}
});
var t2 = Task.Run(() => {
foreach (var id in ids) {
service.GetDeviceStream(id).Subscribe(x => idsRemaining.Remove(x.View.DeviceId));
}
});
await Task.WhenAll(t1, t2);
var sw = Stopwatch.StartNew();
while (idsRemaining.Count > 0) {
if (sw.Elapsed.TotalSeconds > 600) throw new Exception("Failed");
await Task.Delay(100);
}
}
See entire problem source code and test code here: https://github.com/bboyle1234/ReactiveTest

How to clear MemoryCache?

I have created a cache using the MemoryCache class. I add some items to it but when I need to reload the cache I want to clear it first. What is the quickest way to do this? Should I loop through all the items and remove them one at a time or is there a better way?

Dispose the existing MemoryCache and create a new MemoryCache object.

The problem with enumeration
The MemoryCache.GetEnumerator() Remarks section warns: "Retrieving an enumerator for a MemoryCache instance is a resource-intensive and blocking operation. Therefore, the enumerator should not be used in production applications."
Here's why, explained in pseudocode of the GetEnumerator() implementation:
Create a new Dictionary object (let's call it AllCache)
For Each per-processor segment in the cache (one Dictionary object per processor)
{
Lock the segment/Dictionary (using lock construct)
Iterate through the segment/Dictionary and add each name/value pair one-by-one
to the AllCache Dictionary (using references to the original MemoryCacheKey
and MemoryCacheEntry objects)
}
Create and return an enumerator on the AllCache Dictionary
Since the implementation splits the cache across multiple Dictionary objects, it must bring everything together into a single collection in order to hand back an enumerator. Every call to GetEnumerator executes the full copy process detailed above. The newly created Dictionary contains references to the original internal key and value objects, so your actual cached data values are not duplicated.
The warning in the documentation is correct. Avoid GetEnumerator() -- including all of the answers above that use LINQ queries.
A better and more flexible solution
Here's an efficient way of clearing the cache that simply builds on the existing change monitoring infrastructure. It also provides the flexibility to clear either the entire cache or just a named subset and has none of the problems discussed above.
// By Thomas F. Abraham (http://www.tfabraham.com)
namespace CacheTest
{
using System;
using System.Diagnostics;
using System.Globalization;
using System.Runtime.Caching;
public class SignaledChangeEventArgs : EventArgs
{
public string Name { get; private set; }
public SignaledChangeEventArgs(string name = null) { this.Name = name; }
}
/// <summary>
/// Cache change monitor that allows an app to fire a change notification
/// to all associated cache items.
/// </summary>
public class SignaledChangeMonitor : ChangeMonitor
{
// Shared across all SignaledChangeMonitors in the AppDomain
private static event EventHandler<SignaledChangeEventArgs> Signaled;
private string _name;
private string _uniqueId = Guid.NewGuid().ToString("N", CultureInfo.InvariantCulture);
public override string UniqueId
{
get { return _uniqueId; }
}
public SignaledChangeMonitor(string name = null)
{
_name = name;
// Register instance with the shared event
SignaledChangeMonitor.Signaled += OnSignalRaised;
base.InitializationComplete();
}
public static void Signal(string name = null)
{
if (Signaled != null)
{
// Raise shared event to notify all subscribers
Signaled(null, new SignaledChangeEventArgs(name));
}
}
protected override void Dispose(bool disposing)
{
SignaledChangeMonitor.Signaled -= OnSignalRaised;
}
private void OnSignalRaised(object sender, SignaledChangeEventArgs e)
{
if (string.IsNullOrWhiteSpace(e.Name) || string.Compare(e.Name, _name, true) == 0)
{
Debug.WriteLine(
_uniqueId + " notifying cache of change.", "SignaledChangeMonitor");
// Cache objects are obligated to remove entry upon change notification.
base.OnChanged(null);
}
}
}
public static class CacheTester
{
public static void TestCache()
{
MemoryCache cache = MemoryCache.Default;
// Add data to cache
for (int idx = 0; idx < 50; idx++)
{
cache.Add("Key" + idx.ToString(), "Value" + idx.ToString(), GetPolicy(idx));
}
// Flush cached items associated with "NamedData" change monitors
SignaledChangeMonitor.Signal("NamedData");
// Flush all cached items
SignaledChangeMonitor.Signal();
}
private static CacheItemPolicy GetPolicy(int idx)
{
string name = (idx % 2 == 0) ? null : "NamedData";
CacheItemPolicy cip = new CacheItemPolicy();
cip.AbsoluteExpiration = System.DateTimeOffset.UtcNow.AddHours(1);
cip.ChangeMonitors.Add(new SignaledChangeMonitor(name));
return cip;
}
}
}

From http://connect.microsoft.com/VisualStudio/feedback/details/723620/memorycache-class-needs-a-clear-method
The workaround is:
List<string> cacheKeys = MemoryCache.Default.Select(kvp => kvp.Key).ToList();
foreach (string cacheKey in cacheKeys)
{
MemoryCache.Default.Remove(cacheKey);
}

var cacheItems = cache.ToList();
foreach (KeyValuePair<String, Object> a in cacheItems)
{
cache.Remove(a.Key);
}

If performance isn't an issue then this nice one-liner will do the trick:
cache.ToList().ForEach(a => cache.Remove(a.Key));

It seems that there is a Trim method.
So to clear all contents you'd just do
cache.Trim(100)
EDIT:
after digging some more, it seems that looking into Trim is not worth your time
https://connect.microsoft.com/VisualStudio/feedback/details/831755/memorycache-trim-method-doesnt-evict-100-of-the-items
How do I clear a System.Runtime.Caching.MemoryCache

Ran across this, and based on it, wrote a slightly more effective, parallel clear method:
public void ClearAll()
{
var allKeys = _cache.Select(o => o.Key);
Parallel.ForEach(allKeys, key => _cache.Remove(key));
}

You could also do something like this:
Dim _Qry = (From n In CacheObject.AsParallel()
Select n).ToList()
For Each i In _Qry
CacheObject.Remove(i.Key)
Next

You can dispose the MemoryCache.Default cache and then re-set the private field singleton to null, to make it recreate the MemoryCache.Default.
var field = typeof(MemoryCache).GetField("s_defaultCache",
BindingFlags.Static |
BindingFlags.NonPublic);
field.SetValue(null, null);

I was only interested in clearing the cache and found this as an option, when using the c# GlobalCachingProvider
var cache = GlobalCachingProvider.Instance.GetAllItems();
if (dbOperation.SuccessLoadingAllCacheToDB(cache))
{
cache.Clear();
}

a bit improved version of magritte answer.
var cacheKeys = MemoryCache.Default.Where(kvp.Value is MyType).Select(kvp => kvp.Key).ToList();
foreach (string cacheKey in cacheKeys)
{
MemoryCache.Default.Remove(cacheKey);
}

This discussion is also being done here:
https://learn.microsoft.com/en-us/answers/answers/983399/view.html
I wrote an answer there and I'll transcribe it here:
using System.Collections.Generic;
using Microsoft.Extensions.Caching.Memory;
using ServiceStack;
public static class IMemoryCacheExtensions
{
static readonly List<object> entries = new();
/// <summary>
/// Removes all entries, added via the "TryGetValueExtension()" method
/// </summary>
/// <param name="cache"></param>
public static void Clear(this IMemoryCache cache)
{
for (int i = 0; i < entries.Count; i++)
{
cache.Remove(entries[i]);
}
entries.Clear();
}
/// <summary>
/// Use this extension method, to be able to remove all your entries later using "Clear()" method
/// </summary>
/// <typeparam name="TItem"></typeparam>
/// <param name="cache"></param>
/// <param name="key"></param>
/// <param name="value"></param>
/// <returns></returns>
public static bool TryGetValueExtension<TItem>(this IMemoryCache cache, object key, out TItem value)
{
entries.AddIfNotExists(key);
if (cache.TryGetValue(key, out object result))
{
if (result == null)
{
value = default;
return true;
}
if (result is TItem item)
{
value = item;
return true;
}
}
value = default;
return false;
}
}

C# Priority Queue

I'm looking for a priority queue with an interface like this:
class PriorityQueue<T>
{
public void Enqueue(T item, int priority)
{
}
public T Dequeue()
{
}
}
All the implementations I've seen assume that item is an IComparable but I don't like this approach; I want to specify the priority when I'm pushing it onto the queue.
If a ready-made implementation doesn't exist, what's the best way to go about doing this myself? What underlying data structure should I use? Some sort of self-balancing tree, or what? A standard C#.net structure would be nice.

If you have an existing priority queue implementation based on IComparable, you can easily use that to build the structure you need:
public class CustomPriorityQueue<T> // where T need NOT be IComparable
{
private class PriorityQueueItem : IComparable<PriorityQueueItem>
{
private readonly T _item;
private readonly int _priority:
// obvious constructor, CompareTo implementation and Item accessor
}
// the existing PQ implementation where the item *does* need to be IComparable
private readonly PriorityQueue<PriorityQueueItem> _inner = new PriorityQueue<PriorityQueueItem>();
public void Enqueue(T item, int priority)
{
_inner.Enqueue(new PriorityQueueItem(item, priority));
}
public T Dequeue()
{
return _inner.Dequeue().Item;
}
}

You can add safety checks and what not, but here is a very simple implementation using SortedList:
class PriorityQueue<T> {
SortedList<Pair<int>, T> _list;
int count;
public PriorityQueue() {
_list = new SortedList<Pair<int>, T>(new PairComparer<int>());
}
public void Enqueue(T item, int priority) {
_list.Add(new Pair<int>(priority, count), item);
count++;
}
public T Dequeue() {
T item = _list[_list.Keys[0]];
_list.RemoveAt(0);
return item;
}
}
I'm assuming that smaller values of priority correspond to higher priority items (this is easy to modify).
If multiple threads will be accessing the queue you will need to add a locking mechanism too. This is easy, but let me know if you need guidance here.
SortedList is implemented internally as a binary tree.
The above implementation needs the following helper classes. This address Lasse V. Karlsen's comment that items with the same priority can not be added using the naive implementation using a SortedList.
class Pair<T> {
public T First { get; private set; }
public T Second { get; private set; }
public Pair(T first, T second) {
First = first;
Second = second;
}
public override int GetHashCode() {
return First.GetHashCode() ^ Second.GetHashCode();
}
public override bool Equals(object other) {
Pair<T> pair = other as Pair<T>;
if (pair == null) {
return false;
}
return (this.First.Equals(pair.First) && this.Second.Equals(pair.Second));
}
}
class PairComparer<T> : IComparer<Pair<T>> where T : IComparable {
public int Compare(Pair<T> x, Pair<T> y) {
if (x.First.CompareTo(y.First) < 0) {
return -1;
}
else if (x.First.CompareTo(y.First) > 0) {
return 1;
}
else {
return x.Second.CompareTo(y.Second);
}
}
}

You could write a wrapper around one of the existing implementations that modifies the interface to your preference:
using System;
class PriorityQueueThatYouDontLike<T> where T: IComparable<T>
{
public void Enqueue(T item) { throw new NotImplementedException(); }
public T Dequeue() { throw new NotImplementedException(); }
}
class PriorityQueue<T>
{
class ItemWithPriority : IComparable<ItemWithPriority>
{
public ItemWithPriority(T t, int priority)
{
Item = t;
Priority = priority;
}
public T Item {get; private set;}
public int Priority {get; private set;}
public int CompareTo(ItemWithPriority other)
{
return Priority.CompareTo(other.Priority);
}
}
PriorityQueueThatYouDontLike<ItemWithPriority> q = new PriorityQueueThatYouDontLike<ItemWithPriority>();
public void Enqueue(T item, int priority)
{
q.Enqueue(new ItemWithPriority(item, priority));
}
public T Dequeue()
{
return q.Dequeue().Item;
}
}
This is the same as itowlson's suggestion. I just took longer to write mine because I filled out more of the methods. :-s

Here's a very simple lightweight implementation that has O(log(n)) performance for both push and pop. It uses a heap data structure built on top of a List<T>.
/// <summary>Implements a priority queue of T, where T has an ordering.</summary>
/// Elements may be added to the queue in any order, but when we pull
/// elements out of the queue, they will be returned in 'ascending' order.
/// Adding new elements into the queue may be done at any time, so this is
/// useful to implement a dynamically growing and shrinking queue. Both adding
/// an element and removing the first element are log(N) operations.
///
/// The queue is implemented using a priority-heap data structure. For more
/// details on this elegant and simple data structure see "Programming Pearls"
/// in our library. The tree is implemented atop a list, where 2N and 2N+1 are
/// the child nodes of node N. The tree is balanced and left-aligned so there
/// are no 'holes' in this list.
/// <typeparam name="T">Type T, should implement IComparable[T];</typeparam>
public class PriorityQueue<T> where T : IComparable<T> {
/// <summary>Clear all the elements from the priority queue</summary>
public void Clear () {
mA.Clear ();
}
/// <summary>Add an element to the priority queue - O(log(n)) time operation.</summary>
/// <param name="item">The item to be added to the queue</param>
public void Add (T item) {
// We add the item to the end of the list (at the bottom of the
// tree). Then, the heap-property could be violated between this element
// and it's parent. If this is the case, we swap this element with the
// parent (a safe operation to do since the element is known to be less
// than it's parent). Now the element move one level up the tree. We repeat
// this test with the element and it's new parent. The element, if lesser
// than everybody else in the tree will eventually bubble all the way up
// to the root of the tree (or the head of the list). It is easy to see
// this will take log(N) time, since we are working with a balanced binary
// tree.
int n = mA.Count; mA.Add (item);
while (n != 0) {
int p = n / 2; // This is the 'parent' of this item
if (mA[n].CompareTo (mA[p]) >= 0) break; // Item >= parent
T tmp = mA[n]; mA[n] = mA[p]; mA[p] = tmp; // Swap item and parent
n = p; // And continue
}
}
/// <summary>Returns the number of elements in the queue.</summary>
public int Count {
get { return mA.Count; }
}
/// <summary>Returns true if the queue is empty.</summary>
/// Trying to call Peek() or Next() on an empty queue will throw an exception.
/// Check using Empty first before calling these methods.
public bool Empty {
get { return mA.Count == 0; }
}
/// <summary>Allows you to look at the first element waiting in the queue, without removing it.</summary>
/// This element will be the one that will be returned if you subsequently call Next().
public T Peek () {
return mA[0];
}
/// <summary>Removes and returns the first element from the queue (least element)</summary>
/// <returns>The first element in the queue, in ascending order.</returns>
public T Next () {
// The element to return is of course the first element in the array,
// or the root of the tree. However, this will leave a 'hole' there. We
// fill up this hole with the last element from the array. This will
// break the heap property. So we bubble the element downwards by swapping
// it with it's lower child until it reaches it's correct level. The lower
// child (one of the orignal elements with index 1 or 2) will now be at the
// head of the queue (root of the tree).
T val = mA[0];
int nMax = mA.Count - 1;
mA[0] = mA[nMax]; mA.RemoveAt (nMax); // Move the last element to the top
int p = 0;
while (true) {
// c is the child we want to swap with. If there
// is no child at all, then the heap is balanced
int c = p * 2; if (c >= nMax) break;
// If the second child is smaller than the first, that's the one
// we want to swap with this parent.
if (c + 1 < nMax && mA[c + 1].CompareTo (mA[c]) < 0) c++;
// If the parent is already smaller than this smaller child, then
// we are done
if (mA[p].CompareTo (mA[c]) <= 0) break;
// Othewise, swap parent and child, and follow down the parent
T tmp = mA[p]; mA[p] = mA[c]; mA[c] = tmp;
p = c;
}
return val;
}
/// <summary>The List we use for implementation.</summary>
List<T> mA = new List<T> ();
}

That is the exact interface used by my highly optimized C# priority-queue.
It was developed specifically for pathfinding applications (A*, etc.), but should work perfectly for any other application as well.
public class User
{
public string Name { get; private set; }
public User(string name)
{
Name = name;
}
}
...
var priorityQueue = new SimplePriorityQueue<User>();
priorityQueue.Enqueue(new User("Jason"), 1);
priorityQueue.Enqueue(new User("Joseph"), 10);
//Because it's a min-priority queue, the following line will return "Jason"
User user = priorityQueue.Dequeue();

What would be so terrible about something like this?
class PriorityQueue<TItem, TPriority> where TPriority : IComparable
{
private SortedList<TPriority, Queue<TItem>> pq = new SortedList<TPriority, Queue<TItem>>();
public int Count { get; private set; }
public void Enqueue(TItem item, TPriority priority)
{
++Count;
if (!pq.ContainsKey(priority)) pq[priority] = new Queue<TItem>();
pq[priority].Enqueue(item);
}
public TItem Dequeue()
{
--Count;
var queue = pq.ElementAt(0).Value;
if (queue.Count == 1) pq.RemoveAt(0);
return queue.Dequeue();
}
}
class PriorityQueue<TItem> : PriorityQueue<TItem, int> { }

I realise that your question specifically asks for a non-IComparable-based implementation, but I want to point out a recent article from Visual Studio Magazine.
http://visualstudiomagazine.com/articles/2012/11/01/priority-queues-with-c.aspx
This article with #itowlson's can give a complete answer.

A little late but I'll add it here for reference
https://github.com/ERufian/Algs4-CSharp
Key-value-pair priority queues are implemented in Algs4/IndexMaxPQ.cs, Algs4/IndexMinPQ.cs and Algs4/IndexPQDictionary.cs
Notes:
If the Priorities are not IComparable's, an IComparer can be specified in the constructor
Instead of enqueueing the object and its priority, what is enqueued is an index and its priority (and, for the original question, a separate List or T[] would be needed to convert that index to the expected result)

.NET6 finally offers an API for PriorityQueue
See here

Seems like you could roll your own with a seriews of Queues, one for each priority. Dictionary and just add it to the appropriate one.