Develop Reference

SlidingExpiration and MemoryCache

Looking at the documentation for MemoryCache I expected that if an object was accessed within the Expiration period the period would be refreshed. To be honest I think I inferred from the name 'Sliding' as much as anything.
However, it appears from this test
[Test]
public void SlidingExpiryNotRefreshedOnTouch()
{
var memoryCache = new MemoryCache("donkey")
{
{
"1",
"jane",
new CacheItemPolicy {SlidingExpiration = TimeSpan.FromSeconds(1) }
}
};
var enumerable = Enumerable.Repeat("1", 100)
.TakeWhile((id, index) =>
{
Thread.Sleep(100);
return memoryCache.Get(id) != null; // i.e. it still exists
})
.Select((id, index) => (index+2)*100.0/1000); // return the elapsed time
var expires = enumerable.Last(); // gets the last existing entry
expires.Should().BeGreaterThan(1.0);
}
It fails and exhibits the behavior that the object is ejected once the TimeSpan is complete whether or not the object has been accessed. The Linq query is executed at the enumerable.Last(); statement, at which point it will only take while the cache has not expired. As soon as it stops the last item in the list will indicate how long the item lived in the cache for.
For Clarity This question is about the behaviour of MemoryCache. Not the linq query.
Is this anyone else's expectation (i.e. that the expiration does not slide with each touch)?
Is there a mode that extends the lifetime of objects that are 'touched'?
Update I found even if I wrote a wrapper around the cache and re-added the object back to the cache every time I retrieved it, with another SlidingExpiration its still only honored the initial setting. To get it to work the way I desired I had to physically remove it from the cache before re-adding it! This could cause undesirable race conditions in a multi-threaded environment.

... new CacheItemPolicy {SlidingExpiration = TimeSpan.FromSeconds(1) }
This is not adequately documented in MSDN. You were a bit unlucky, 1 second is not enough. By a hair, use 2 seconds and you'll see it works just like you hoped it would. Tinker some more with FromMilliseconds() and you'll see that ~1.2 seconds is the happy minimum in this program.
Explaining this is rather convoluted, I have to talk about how MemoryCache avoids having to update the sliding timer every single time you access the cache. Which is relatively expensive, as you might imagine. Let's take a shortcut and take you to the relevant Reference Source code. Small enough to paste here:
internal void UpdateSlidingExp(DateTime utcNow, CacheExpires expires) {
if (_slidingExp > TimeSpan.Zero) {
DateTime utcNewExpires = utcNow + _slidingExp;
if (utcNewExpires - _utcAbsExp >= CacheExpires.MIN_UPDATE_DELTA || utcNewExpires < _utcAbsExp) {
expires.UtcUpdate(this, utcNewExpires);
}
}
}
CacheExpires.MIN_UPDATE_DELTA is the crux, it prevents UtcUpdate() from being called. Or to put it another way, at least MIN_UPDATE_DELTA worth of time has to pass before it will update the sliding timer. The CacheExpired class is not indexed by the Reference Source, a hint that they are not entirely happy about the way it works :) But a decent decompiler can show you:
static CacheExpires()
{
MIN_UPDATE_DELTA = new TimeSpan(0, 0, 1);
MIN_FLUSH_INTERVAL = new TimeSpan(0, 0, 1);
// etc...
}
In other words, hard-coded to 1 second. With no way to change it right now, that's pretty ugly. It takes ~1.2 seconds for the SlidingExpiration value in this test program because Thread.Sleep(100) does not actually sleep for 100 milliseconds, it takes a bit more. Or to put it another way, it will be the 11th Get() call that gets the sliding timer to slide in this test program. You didn't get that far.
Well, this ought to be documented but I'd guess this is subject to change. For now, you'll need to assume that a practical sliding expiration time should be at least 2 seconds.

Directory with datetime+string key, and automatic removal of old entries

I have an application that receives certain "events", uniquely identified by a 12 chars string and a DateTime. At each event is associated a result that is a string.
I need to keep these events in memory (for a maximum of for example 8 hours) and be able, in case I receive a second time the same event, being able to know I've already received it (in the last 8 hours).
Events to store will be less than 1000.
I can't use an external storage, it has to be done in memory.
My idea is to use a Dictionary where the key is a class composed of a string and a datetime, the value is the result.
EDIT: the string itself (actually a MAC address) does not identify uniquely the event, it's the MAC AND the DateTime, those two combined are unique, that's why the key must be formed by both.
The application is a server that receives a certain event from a client: the event is marked on the client by client MAC and by the client datetime (can't use a guid).
It may happen that the client retransmits the same data, and by checking the dictionary for that MAC/Datetime key I would know that I have already received that data.
Then, every hour (for example), I can foreach through the whole collection and remove all the keys where datetime is older than 8 hours.
Can you suggest a better approach to the problem or to the data formats I have chosen? In terms of performance and cleaniness of the code.
Or a better way to delete old data, with LINQ for example.
Thanks,
Mattia

The event time has to not be part of the key -- if it is, how are you going to be able to tell that you have already received this event? So you should move to a dictionary where the keys are event names and the values are tuples of date and result.
Once in a while you can trim old data from the dictionary easily with LINQ:
dictionary = dictionary
.Where(p => p.Value.DateOfEvent >= DateTime.Now.AddHours(-8))
.ToDictionary();

If requirements state that updating once per hour is good enough, and you're never having more than 1000 items in the dictionary, your solution should be perfectly adequate and probably the most easily understood by anyone else looking at your code. I'd probably recommend immutable structs for the key instead of classes, but that's it.
If there's a benefit to removing them immediately rather than once per hour, you could do something where you also add a Timer that removes it after exactly 8 hours, but then you've got to deal with thread safety and cleaning up all the timers and such. Likely not worth it.
I'd avoid the OrderedDictionary approach since it's more code, and may be slower since it has to reorder with every insert.
It's a common mantra these days to focus first on keeping code simple, only optimize when necessary. Until you have a known bottleneck and have profiled it, you never know if you're even optimizing the right thing. (And from your description, there's no telling which part will be slowest without profiling it).

I would go for a Dictionary.
This way you can searh very fast for the string (O(1)-operation).
Other collections are slower:
OrderedDictionary: is slow because it needs boxing and unboxing.
SortedDictionary: performs an O(log n) operation.
All normal arrays and lists: use an O(n/2) operation.
An example:
public class Event
{
public Event(string macAddress, DateTime time, string data)
{
MacAddress = macAddress;
Time = time;
Data = data;
}
public string MacAddress { get; set; }
public DateTime Time { get; set; }
public string Data { get; set; }
}
public class EventCollection
{
private readonly Dictionary<Tuple<string, DateTime>, Event> _Events = new Dictionary<Tuple<string, DateTime>, Event>();
public void Add(Event e)
{
_Events.Add(new Tuple<string, DateTime>(e.MacAddress, e.Time), e);
}
public IList<Event> GetOldEvents(bool autoRemove)
{
DateTime old = DateTime.Now - TimeSpan.FromHours(8);
List<Event> results = new List<Event>();
foreach(Event e in _Events.Values)
if (e.Time < old)
results.Add(e);
// Clean up
if (autoRemove)
foreach(Event e in results)
_Events.Remove(new Tuple<string, DateTime>(e.MacAddress, e.Time));
return results;
}
}

I would use an OrderedDictionary where the key is the 12 charactor identifier and the result and datetime are part of the value. Sadly OrderedDictionary is not generic (key and value are objects), so you would need to do the casting and type checking yourself. When you need to remove the old events, you can foreach through the OrderedDictionary and stop when you get to a time new enough to keep. This assumes the datetimes you use are in order when you add them to the dictionary.

Instant.Now for NodaTime

I'm trying to get a handle on using the Noda Time framework by Jon Skeet (and others).
I'm trying to store the current now(Instant). Instant is created from a long ticks, but what is the current now count of Ticks?
Is it:
Instant now = new Instant(DateTime.Now.ToUniversalTime().Ticks);
And or?
Instant now = Instant.FromDateTimeUtc(DateTime.Now.ToUniversalTime());
Are they equivalent, am I even doing this right?
PS, if Jon answer's this - I'd like to propose an Instant.Now property.
PS2 I know the title contains a tag, but it wouldn't let me have a short "Instant.Now" title.

I did a bit of research and it seems that the NodaTime way is to get the now moment according to a clock.
If you want to get the current time using the system clock, just use SystemClock.Instance.GetCurrentInstant().
However, instead of using the SystemClock.Instance directly in your code, it's preferable that you inject an IClock dependency in your time-aware classes.
This will allow you to:
provide the class with SystemClock.Instance at runtime, so the code will use the correct time
supply a fake implementation of IClock during unit testing to allow you to tweak the time as needed in order to test various scenarios (like the passing of time). There's a NodaTime.Testing project that offers such a class, called FakeClock.
I find this very useful. I think having something like new Instant() or Instant.Now return the current time would make it easier to hardcode usages of SystemClock under the covers, therefore missing the testing advantage that NodaTime offers.
For more info on unit testing with NodaTime, see this link.
Regarding your code examples: they are not equivalent.
Instant.FromDateTimeUtc(DateTime.Now.ToUniversalTime()) will indeed give you the current instant in UTC.
new Instant(DateTime.Now.ToUniversalTime().Ticks) will give you a wrong date far in the future, because the BCL's DateTime.Ticks represents the number of ticks since 1/1/0001, and NodaTime's Instant.Ticks represents the number of ticks since 1/1/1970 (see the remark here).

SystemClock.Now returns the current time as an Instant value:
Instant now = SystemClock.Instance.Now;
But you may want to heed the remarks in the documentation for the IClock interface:
IClock is intended for use anywhere you need to have access to the current time. Although it's not strictly incorrect to call SystemClock.Instance.Now directly, in the same way as you might call UtcNow, it's strongly discouraged as a matter of style for production code. We recommend providing an instance of IClock to anything that needs it, which allows you to write tests using the stub clock in the NodaTime.Testing assembly (or your own implementation).
As a simple example, suppose you have a Logger class that needs the current time. Instead of accessing SystemClock directly, use an IClock instance that's supplied via its constructor:
public class Logger
{
private readonly IClock clock;
public Logger(IClock clock)
{
this.clock = clock;
}
public void Log(string message)
{
Instant timestamp = this.clock.Now;
// Now log the message with the timestamp...
}
}
When you instantiate a Logger in your production code, you can give it SystemClock.Instance. But in a unit test for the Logger class, you can give it a FakeClock.

Difficulty comparing two DateTime instances

I'm writing a Unit Test class in C# (.NET 4.5). In one of the tests I'm checking the values of various properties after an instance of our class FeedbackDao is constructed. On construction, the FeedbackDate property of FeedbackDao is set to DateTime.Now.
FeedbackDao feedbackDao = new FeedbackDao();
// a couple of lines go here then I set up this test:
Assert.IsTrue(feedbackDao.FeedbackDate.CompareTo(DateTime.Now) < 0);
My assumption is that feedbackDao.FeedbackDate should always be just a little earlier than the current time returned by DateTime.Now, even if it's only by a millisecond, and my IsTrue test should always pass, but sometimes it passes and sometimes it fails. When I add a message like this:
Assert.IsTrue(feedbackDao.FeedbackDate.CompareTo(DateTime.Now) < 0,
feedbackDao.FeedbackDate.CompareTo(DateTime.Now).ToString());
the message sometimes reads -1 (meaning that FeedbackDate is earlier than Now) and sometimes reads 0 (meaning that the DateTime instances are equal).
Why is FeedbackDate not always earlier than Now? And, if I can't trust that comparison, how can I write a rigorous test to check the value of FeedbackDate when FeedbackDao is constructed?

My assumption is that feedbackDao.FeebackDate should always be just a little earlier than the current time returned by DateTime.Now, even if it's only by a millisecond.
What makes you think that? That would suggest that 1000 calls would have to take at least 1 second which seems unlikely.
Add to that the fact that DateTime.Now only has a practical granularity of about 10-15ms IIRC, and very often if you call DateTime.Now twice in quick succession you'll get the same value twice.
For the purpose of testability - and clean expression of dependencies - I like to use a "clock" interface (IClock) which is always used to extract the current system time. You can then write a fake implementation to control time however you see fit.
Additionally, this assertion is flawed:
Assert.IsTrue(feedbackDao.FeebackDate.CompareTo(DateTime.Now) < 0,
feedbackDao.FeebackDate.CompareTo(DateTime.Now).ToString());
It's flawed because it evaluates DateTime.Now twice... so the value that it reports isn't necessarily the same one that it checks. It would be better as:
DateTime now = DateTime.Now;
Assert.IsTrue(feedbackDao.FeebackDate.CompareTo(now) < 0,
feedbackDao.FeebackDate.CompareTo(now).ToString());
Or even better:
DateTime now = DateTime.Now;
DateTime feedbackDate = feedbackDao.FeebackDate;
Assert.IsTrue(now < feedbackDate,
feedbackDate + " should be earlier than " + now);

Your test is not that useful as it is, you're asserting that the value is less than DateTime.Now but that does not mean it was correctly set to the expected value. If the date time is not initialized it will have the DateTime.MinValue and that value will always pass the test.
This test is as valid as testing for feedbackDao.FeebackDate.CompareTo(DateTime.Now) <= 0 and with that you would not have the problem that motivated you to write this question.
You need to extract the dependency on DateTime.Now or use a mocking framework that supports mocking DateTime.Now and assert that the value is initialized to the correct one. You can check Microsoft Moles, now renamed to Fakes in VS 2012, which is the only mocking framework that I know that is free (kind of for the latest version, since it ships with VS and don't know if it is available on the express editions) and that will let you replace a call to DateTime.Now.
Update:
Without resorting to a mocking framework you could improve your test by doing something like this:
var lowerBoundary = DateTime.Now;
var dao = new FeedbackDao();
var upperBoundary = DateTime.Now;
Assert.IsTrue(dao.Date >= lowerBoundary && dao.Date <= upperBoundary);

When unit testing, I consider DateTime.Now to be an external dependency, and thus something needing to be mocked. What I've done in the past when testing scenarios involving DateTime.Now, I've just passed a Func<DateTime> in via the constructor of the class, which allows me to mock DateTime.Now during testing.
I prefer this over Jon Skeet's suggestion of using something like an IClock interface to wrap around the DateTime properties, just because the last time I did this, I felt silly making a new interface and class to wrap around a single property. If you're going to need to test around more than one of the static DateTime properties, I definitely agree with the IClock suggestion.
For example,
public class Foo
{
private readonly Func<DateTime> timeStampProvider;
public Foo(Func<DateTime> timeStampProvider)
{
this.timeStampProvider = timeStampProvider;
}
public Foo() : this(() => DateTime.Now)
{
}
public bool CompareDate(DateTime comparisonDate)
{
// Get my timestamp
return comparisonDate > timeStampProvider();
}
}
Then, during testing,
var testFoo = new Foo(() => new DateTime(1, 1, 2010));

I generally use a mock data to validate my logic. I evolve my test scenarios around the mock data. As suggested by DBM.
Mock data is a set of known data that is generally static or configurable. Common practice is to have a XML file with all the test data and load them as and when required. I can give you an example in our Project.

Try
Assert.IsTrue(feedbackDao.FeebackDate.CompareTo(DateTime.Now) < 1);
Or
Assert.IsTrue(feedbackDao.FeebackDate - DateTime.Now < someMarginOfError);
Time is generally fairly granular - often 10's of milliseconds IIRC.

Depending on your system, DateTime.Now is not updated every millisecond or tick, it is only updated periodically. Typically 10 ms or so. See here: How frequent is DateTime.Now updated ? or is there a more precise API to get the current time?

DateTime.Now isn't 100% accurate. It increases by around 130 ms(from personal experience per tick). So it's verry likely that if your method is fast enough the date will be equal to datetime.now and not smaller.
If you want a 100% accurate timer you should use the StopWatch class.
Msdn link to stopwatch

.Net DateTime Precision

within my .net domain object I am tracking each state transition. This is done by putting the state set into a state history collection. So later on, one can see an desc ordered list to find out which state was changed at what time.
So there is a method like this:
private void SetState(RequestState state)
{
var stateHistoryItem = new RequestStateHistoryItem(state, this);
stateHistoryItems.Add(stateHistoryItem);
}
When a new RequestStateHistoryItem is instantiated, the current date is automatically assigned. Like this:
protected IdentificationRequestStateHistoryItem()
{
timestamp = EntityTimestamp.New();
}
The EntityTimestamp object is an object containing the appropiate user and created and changed date.
When listing the state history, I do a descending order with Linq:
public virtual IEnumerable<RequestStateHistoryItem> StateHistoryItems
{
get { return stateHistoryItems.OrderByDescending(s => s.Timestamp.CreatedOn.Ticks); }
}
Now when a new Request is instantiated the first state Received is set in the constructor SetState(RequestState.Received). Then, without any delay and depending on some conditions, a new state Started is set. After some time (db operations) the state Finished is set.
Now when performing the descending ordering, the Received always is AFTER the Started state. When I am debugging slowly, or when putting a System.Threading.Thread.Sleep(1000) before setting the state to Started, the ordering works.
If not, as told above, the Started state's CreatedOn is OLDER then the Received CreatedOn date?!
TimeOfDay {17:04:42.9430318} FINSHED
Ticks 634019366829430318
TimeOfDay {17:04:39.5376207} RECEICED
Ticks 634019366795376207
TimeOfDay {17:04:39.5367815} STARTED
Ticks 634019366795367815
How can that be? I would understand if the received and start date is exactly the same, but I don't understand how it can even be BEFORE the other one?
I already tried new DateTimePrecise().Now, (see DateTimePrecise class) I found in another question. Same result.
Anyone knows what that could be?
Update
public virtual bool Finish()
{
// when I put the SetState(State.Received) from the constructor into here, the timestamp of finish still is BEFORE received
SetState(IdentificationRequestState.Received);
SetState(IdentificationRequestState.Finished);
// when I put the SetState(State.Received) after Finished, then the Received timestamp is BEFORE Finished
SetState(IdentificationRequestState.Finished);
SetState(IdentificationRequestState.Received);
var match = ...
if (match != null)
{
...
}
else
{
...
}
}

DateTime.Now is not accurate to the millisecond. It is only updated at larger intervals, something like 30 or 15 milliseconds (which is just the way Window's internal clock works, IIRC).
System.Diagnostics.Stopwatch is a more accurate way to measure time differences. It also doesn't have the overhead of UTC to local time conversions etc. The DateTimePrecise class uses a combination of DateTime and Stopwatch to give a more accurate time than DateTime.Now does.

You are retrieving the timestamp at an undetermined time before you add it to your collection.
The delay between retrieving it and adding it to the collection is variable - for example your thread may be pre-empted by the scheduler after getting the timestamp and before adding to the collection.
If you want strict ordering, you need to use synchronisation, something like the following every time you instantiate a history item:
lock(syncLock)
{
// Timestamp is generated here...
var stateHistoryItem = new RequestStateHistoryItem(state, this);
// ... but an indeterminate time can pass before ...
...
// ... it's added to the collection here.
stateHistoryItems.Add(stateHistoryItem);
}

Have you tried setting both the Received and Started timestamps via the same approach (i.e. moving the Received stamp out of the constructor and setting it via property or method to match how the Started status is set?).
I know it doesn't explain why, but constructors are somewhat special in the runtime. .NET constructors are designed to execute as fast as possible, so it wouldn't surprise me that there are some side-effects of the focus on performance.