Develop Reference

SlidingExpiration and MemoryCache

Looking at the documentation for MemoryCache I expected that if an object was accessed within the Expiration period the period would be refreshed. To be honest I think I inferred from the name 'Sliding' as much as anything.
However, it appears from this test
[Test]
public void SlidingExpiryNotRefreshedOnTouch()
{
var memoryCache = new MemoryCache("donkey")
{
{
"1",
"jane",
new CacheItemPolicy {SlidingExpiration = TimeSpan.FromSeconds(1) }
}
};
var enumerable = Enumerable.Repeat("1", 100)
.TakeWhile((id, index) =>
{
Thread.Sleep(100);
return memoryCache.Get(id) != null; // i.e. it still exists
})
.Select((id, index) => (index+2)*100.0/1000); // return the elapsed time
var expires = enumerable.Last(); // gets the last existing entry
expires.Should().BeGreaterThan(1.0);
}
It fails and exhibits the behavior that the object is ejected once the TimeSpan is complete whether or not the object has been accessed. The Linq query is executed at the enumerable.Last(); statement, at which point it will only take while the cache has not expired. As soon as it stops the last item in the list will indicate how long the item lived in the cache for.
For Clarity This question is about the behaviour of MemoryCache. Not the linq query.
Is this anyone else's expectation (i.e. that the expiration does not slide with each touch)?
Is there a mode that extends the lifetime of objects that are 'touched'?
Update I found even if I wrote a wrapper around the cache and re-added the object back to the cache every time I retrieved it, with another SlidingExpiration its still only honored the initial setting. To get it to work the way I desired I had to physically remove it from the cache before re-adding it! This could cause undesirable race conditions in a multi-threaded environment.

... new CacheItemPolicy {SlidingExpiration = TimeSpan.FromSeconds(1) }
This is not adequately documented in MSDN. You were a bit unlucky, 1 second is not enough. By a hair, use 2 seconds and you'll see it works just like you hoped it would. Tinker some more with FromMilliseconds() and you'll see that ~1.2 seconds is the happy minimum in this program.
Explaining this is rather convoluted, I have to talk about how MemoryCache avoids having to update the sliding timer every single time you access the cache. Which is relatively expensive, as you might imagine. Let's take a shortcut and take you to the relevant Reference Source code. Small enough to paste here:
internal void UpdateSlidingExp(DateTime utcNow, CacheExpires expires) {
if (_slidingExp > TimeSpan.Zero) {
DateTime utcNewExpires = utcNow + _slidingExp;
if (utcNewExpires - _utcAbsExp >= CacheExpires.MIN_UPDATE_DELTA || utcNewExpires < _utcAbsExp) {
expires.UtcUpdate(this, utcNewExpires);
}
}
}
CacheExpires.MIN_UPDATE_DELTA is the crux, it prevents UtcUpdate() from being called. Or to put it another way, at least MIN_UPDATE_DELTA worth of time has to pass before it will update the sliding timer. The CacheExpired class is not indexed by the Reference Source, a hint that they are not entirely happy about the way it works :) But a decent decompiler can show you:
static CacheExpires()
{
MIN_UPDATE_DELTA = new TimeSpan(0, 0, 1);
MIN_FLUSH_INTERVAL = new TimeSpan(0, 0, 1);
// etc...
}
In other words, hard-coded to 1 second. With no way to change it right now, that's pretty ugly. It takes ~1.2 seconds for the SlidingExpiration value in this test program because Thread.Sleep(100) does not actually sleep for 100 milliseconds, it takes a bit more. Or to put it another way, it will be the 11th Get() call that gets the sliding timer to slide in this test program. You didn't get that far.
Well, this ought to be documented but I'd guess this is subject to change. For now, you'll need to assume that a practical sliding expiration time should be at least 2 seconds.

How to get precise DateTime in C#

I know that DateTime.UtcNow does not have a relatively high degree of precision (10-15 ms resolution).
This is a problem for me because in my log files, I want better resolution for the purposes of resolving race conditions.
The question is: how can I get the current date/time with a better precision than what DateTime.UtcNow offers?
The only solution which I've seen up till now has been this: https://stackoverflow.com/a/15008836/270348. Is that a suitable way to solve the problem?

You can use Stopwatch for more precise measurements of time. You can then have each log entry record the time from the start of the first operation. If it's important, you can record the DateTime of the first operation and therefore calculate the times of the rest, but it sounds like just having the ticks/nanoseconds since the start of the first operation is good enough for your purposes.

If you want to add tracing you could write your own ETW Trace Provider like here. Then you do not need to take care how the timing is fetched. The Windows Kernel will take care that the timing is accurate and the order of the events is correct.
If you do not want to write your own ETW Provider you can use EventSource which is available as Nuget Package which allows you to use it from .NET 4.0 as well. More infos has Vance Morrison on his blog.
When use ETW events then you get a powerful trace analyzer for free with the Windows Performance Toolkit. As added bonus you can do system wide profiling in all processes with minimal impact as well.
This allows you to selectively add call stacks for every event you write which can be invaluable when you want to track down who did call your Dispose method two times.

Do you really need a precise timer or what you really need is something that give a sort? For example:
static class AutoIncrement
{
private static long num;
public static long Current
{
get
{
return Interlocked.Increment(ref num);
}
}
}
Console.WriteLine(AutoIncrement.Current);
Console.WriteLine(AutoIncrement.Current);
Console.WriteLine(AutoIncrement.Current);
Console.WriteLine(AutoIncrement.Current);
Console.WriteLine(AutoIncrement.Current);
This is guaranteed to be unique, and to change every time, and to be sorted. Its precision is better than microsecond. Its precision is absolute on a single PC.
or if you really want Timestamps...
static class AutoIncrement
{
private static long lastDate;
public static DateTime Current
{
get
{
long lastDateOrig, lastDateNew;
do
{
lastDateOrig = lastDate;
lastDateNew = lastDateOrig + 1;
lastDateNew = Math.Max(DateTime.UtcNow.Ticks, lastDateNew);
}
while (Interlocked.CompareExchange(ref lastDate, lastDateNew, lastDateOrig) != lastDateOrig);
return new DateTime(lastDateNew, DateTimeKind.Utc);
}
}
}
DateTime ac = AutoIncrement.Current;
Console.WriteLine(CultureInfo.InvariantCulture, "{0} {1:yyyy/MM/dd HH:mm:ss.fffffff}", ac.Ticks, ac);
This last variant uses as a base DateTime.UtcNow, but each call if it isn't changed, it increments it by one.

ensure that utc datetime is persisted

Let us say I have a situation where I set the creation UTC date time just before I persist the entity. Usually, I would do it like this:
var something = new Something;
something.CreationDT = DateTime.UtcNow;
repo.Save(something);
What is the best way to ensure that this always happens. I know how to create a DateTime wrapper for testing purposes but this would not necessary help me here as programmers could still use:
something.CreationDT = DateTime.Now;
Is there a pattern to ensure that every programmer uses UTC?

If you always want to force the UTC date on the object and this is the exact code, then just do it as part of the constructor e.g.
public Something()
{
CreatedDT = DateTime.UtcNow;
}
public DateTime CreatedDT { get; private set; }
...
Save(new Something());
Obviously this doesn't cater for scenarios where you create the object earlier in the application and persist later, however, that doesn't appear to be the case here.
I would definitely avoid allowing users to set Now whilst internally returning UtcNow, that's just going to confuse people. If anything use UtcNow is better as it's more readable for the developer - it's clear to see that the value has to be UTC.
For this sort of thing, I generally try to let the DB do as much work as possible so I would have a default value set to the current UTC date/time so I don't need to worry about sending it from my client at all.

Directory with datetime+string key, and automatic removal of old entries

I have an application that receives certain "events", uniquely identified by a 12 chars string and a DateTime. At each event is associated a result that is a string.
I need to keep these events in memory (for a maximum of for example 8 hours) and be able, in case I receive a second time the same event, being able to know I've already received it (in the last 8 hours).
Events to store will be less than 1000.
I can't use an external storage, it has to be done in memory.
My idea is to use a Dictionary where the key is a class composed of a string and a datetime, the value is the result.
EDIT: the string itself (actually a MAC address) does not identify uniquely the event, it's the MAC AND the DateTime, those two combined are unique, that's why the key must be formed by both.
The application is a server that receives a certain event from a client: the event is marked on the client by client MAC and by the client datetime (can't use a guid).
It may happen that the client retransmits the same data, and by checking the dictionary for that MAC/Datetime key I would know that I have already received that data.
Then, every hour (for example), I can foreach through the whole collection and remove all the keys where datetime is older than 8 hours.
Can you suggest a better approach to the problem or to the data formats I have chosen? In terms of performance and cleaniness of the code.
Or a better way to delete old data, with LINQ for example.
Thanks,
Mattia

The event time has to not be part of the key -- if it is, how are you going to be able to tell that you have already received this event? So you should move to a dictionary where the keys are event names and the values are tuples of date and result.
Once in a while you can trim old data from the dictionary easily with LINQ:
dictionary = dictionary
.Where(p => p.Value.DateOfEvent >= DateTime.Now.AddHours(-8))
.ToDictionary();

If requirements state that updating once per hour is good enough, and you're never having more than 1000 items in the dictionary, your solution should be perfectly adequate and probably the most easily understood by anyone else looking at your code. I'd probably recommend immutable structs for the key instead of classes, but that's it.
If there's a benefit to removing them immediately rather than once per hour, you could do something where you also add a Timer that removes it after exactly 8 hours, but then you've got to deal with thread safety and cleaning up all the timers and such. Likely not worth it.
I'd avoid the OrderedDictionary approach since it's more code, and may be slower since it has to reorder with every insert.
It's a common mantra these days to focus first on keeping code simple, only optimize when necessary. Until you have a known bottleneck and have profiled it, you never know if you're even optimizing the right thing. (And from your description, there's no telling which part will be slowest without profiling it).

I would go for a Dictionary.
This way you can searh very fast for the string (O(1)-operation).
Other collections are slower:
OrderedDictionary: is slow because it needs boxing and unboxing.
SortedDictionary: performs an O(log n) operation.
All normal arrays and lists: use an O(n/2) operation.
An example:
public class Event
{
public Event(string macAddress, DateTime time, string data)
{
MacAddress = macAddress;
Time = time;
Data = data;
}
public string MacAddress { get; set; }
public DateTime Time { get; set; }
public string Data { get; set; }
}
public class EventCollection
{
private readonly Dictionary<Tuple<string, DateTime>, Event> _Events = new Dictionary<Tuple<string, DateTime>, Event>();
public void Add(Event e)
{
_Events.Add(new Tuple<string, DateTime>(e.MacAddress, e.Time), e);
}
public IList<Event> GetOldEvents(bool autoRemove)
{
DateTime old = DateTime.Now - TimeSpan.FromHours(8);
List<Event> results = new List<Event>();
foreach(Event e in _Events.Values)
if (e.Time < old)
results.Add(e);
// Clean up
if (autoRemove)
foreach(Event e in results)
_Events.Remove(new Tuple<string, DateTime>(e.MacAddress, e.Time));
return results;
}
}

I would use an OrderedDictionary where the key is the 12 charactor identifier and the result and datetime are part of the value. Sadly OrderedDictionary is not generic (key and value are objects), so you would need to do the casting and type checking yourself. When you need to remove the old events, you can foreach through the OrderedDictionary and stop when you get to a time new enough to keep. This assumes the datetimes you use are in order when you add them to the dictionary.

Difficulty comparing two DateTime instances

I'm writing a Unit Test class in C# (.NET 4.5). In one of the tests I'm checking the values of various properties after an instance of our class FeedbackDao is constructed. On construction, the FeedbackDate property of FeedbackDao is set to DateTime.Now.
FeedbackDao feedbackDao = new FeedbackDao();
// a couple of lines go here then I set up this test:
Assert.IsTrue(feedbackDao.FeedbackDate.CompareTo(DateTime.Now) < 0);
My assumption is that feedbackDao.FeedbackDate should always be just a little earlier than the current time returned by DateTime.Now, even if it's only by a millisecond, and my IsTrue test should always pass, but sometimes it passes and sometimes it fails. When I add a message like this:
Assert.IsTrue(feedbackDao.FeedbackDate.CompareTo(DateTime.Now) < 0,
feedbackDao.FeedbackDate.CompareTo(DateTime.Now).ToString());
the message sometimes reads -1 (meaning that FeedbackDate is earlier than Now) and sometimes reads 0 (meaning that the DateTime instances are equal).
Why is FeedbackDate not always earlier than Now? And, if I can't trust that comparison, how can I write a rigorous test to check the value of FeedbackDate when FeedbackDao is constructed?

My assumption is that feedbackDao.FeebackDate should always be just a little earlier than the current time returned by DateTime.Now, even if it's only by a millisecond.
What makes you think that? That would suggest that 1000 calls would have to take at least 1 second which seems unlikely.
Add to that the fact that DateTime.Now only has a practical granularity of about 10-15ms IIRC, and very often if you call DateTime.Now twice in quick succession you'll get the same value twice.
For the purpose of testability - and clean expression of dependencies - I like to use a "clock" interface (IClock) which is always used to extract the current system time. You can then write a fake implementation to control time however you see fit.
Additionally, this assertion is flawed:
Assert.IsTrue(feedbackDao.FeebackDate.CompareTo(DateTime.Now) < 0,
feedbackDao.FeebackDate.CompareTo(DateTime.Now).ToString());
It's flawed because it evaluates DateTime.Now twice... so the value that it reports isn't necessarily the same one that it checks. It would be better as:
DateTime now = DateTime.Now;
Assert.IsTrue(feedbackDao.FeebackDate.CompareTo(now) < 0,
feedbackDao.FeebackDate.CompareTo(now).ToString());
Or even better:
DateTime now = DateTime.Now;
DateTime feedbackDate = feedbackDao.FeebackDate;
Assert.IsTrue(now < feedbackDate,
feedbackDate + " should be earlier than " + now);

Your test is not that useful as it is, you're asserting that the value is less than DateTime.Now but that does not mean it was correctly set to the expected value. If the date time is not initialized it will have the DateTime.MinValue and that value will always pass the test.
This test is as valid as testing for feedbackDao.FeebackDate.CompareTo(DateTime.Now) <= 0 and with that you would not have the problem that motivated you to write this question.
You need to extract the dependency on DateTime.Now or use a mocking framework that supports mocking DateTime.Now and assert that the value is initialized to the correct one. You can check Microsoft Moles, now renamed to Fakes in VS 2012, which is the only mocking framework that I know that is free (kind of for the latest version, since it ships with VS and don't know if it is available on the express editions) and that will let you replace a call to DateTime.Now.
Update:
Without resorting to a mocking framework you could improve your test by doing something like this:
var lowerBoundary = DateTime.Now;
var dao = new FeedbackDao();
var upperBoundary = DateTime.Now;
Assert.IsTrue(dao.Date >= lowerBoundary && dao.Date <= upperBoundary);

When unit testing, I consider DateTime.Now to be an external dependency, and thus something needing to be mocked. What I've done in the past when testing scenarios involving DateTime.Now, I've just passed a Func<DateTime> in via the constructor of the class, which allows me to mock DateTime.Now during testing.
I prefer this over Jon Skeet's suggestion of using something like an IClock interface to wrap around the DateTime properties, just because the last time I did this, I felt silly making a new interface and class to wrap around a single property. If you're going to need to test around more than one of the static DateTime properties, I definitely agree with the IClock suggestion.
For example,
public class Foo
{
private readonly Func<DateTime> timeStampProvider;
public Foo(Func<DateTime> timeStampProvider)
{
this.timeStampProvider = timeStampProvider;
}
public Foo() : this(() => DateTime.Now)
{
}
public bool CompareDate(DateTime comparisonDate)
{
// Get my timestamp
return comparisonDate > timeStampProvider();
}
}
Then, during testing,
var testFoo = new Foo(() => new DateTime(1, 1, 2010));

I generally use a mock data to validate my logic. I evolve my test scenarios around the mock data. As suggested by DBM.
Mock data is a set of known data that is generally static or configurable. Common practice is to have a XML file with all the test data and load them as and when required. I can give you an example in our Project.

Try
Assert.IsTrue(feedbackDao.FeebackDate.CompareTo(DateTime.Now) < 1);
Or
Assert.IsTrue(feedbackDao.FeebackDate - DateTime.Now < someMarginOfError);
Time is generally fairly granular - often 10's of milliseconds IIRC.

Depending on your system, DateTime.Now is not updated every millisecond or tick, it is only updated periodically. Typically 10 ms or so. See here: How frequent is DateTime.Now updated ? or is there a more precise API to get the current time?

DateTime.Now isn't 100% accurate. It increases by around 130 ms(from personal experience per tick). So it's verry likely that if your method is fast enough the date will be equal to datetime.now and not smaller.
If you want a 100% accurate timer you should use the StopWatch class.
Msdn link to stopwatch