I would like to detect which C# files (pocos) have changed the second time I run the application compared to the previous run. By a change, I mean any change to the file eg change to the logic inside a method or addition/removal of properties/fields. You can assume the file name stays the same. Inspecting git logs/changes is out of scope for this problem.
Potential solutions I can think of is to keep a record of the hash in the database. I can calculate the hash of the file or calculate the hash of the instance of the poco object. Calculating the hash of the file will give me a unique hash, but during runtime, all the .cs files get compiled into dll so it may not be possible to do this within a single project, I would need to have a separate project run before the main project to populate the hashes (not ideal). Or I can I hash the object instance, but I don’t think that considers the code changes in the method/functions of the poco as it’s only hashing the property/field values (correct me here if I’m wrong).
I am wondering if there a straight forward way of doing this? Potentially a solution where I can get a unique hash of the file at runtime, save it to database for second run. On the second run, compare the new hashes I just generated to the values in the database and operate on any changes that were made.
So at this point, I'm not sure what possible methods are there to tackle this issue. Any pointers or advice on how to tackle this would be helpful.
Background context to the problem:
In a CQRS system, we have projections, and any changes to the views/projections require the events to be replaying onto the projection to rebuild the view. This can be expensive if the event database is large. So I've built some tools to allow all the all views/projections to be rebuilt for application startup in development and for production, ability to optionally recreate some views for speed that have changed during the sprint. To optionally recreate views, we have to find out which views changed during the sprint and pass it in as an environment variable so the only the correct views should be rebuilt to speed up the startup time of the application. This is a manual process and human error can occur easily where we may miss a view that may have changed. I'd like automatically detect the views that have changed during the sprint at the application startup, by checking the files that have changed and use this change list to rebuild the views.
Thanks in advance for all the help :)
Use a source control system. Today Git is what most people use. Visual Studio has support for Git (see Version Control in Visual Studio). Personally I prefer to use the Git Extensions for Visual Studio.
These extensions have a browser that allows to compare commits. All code changes are highlighted. The old code is red, the new code is green.
UPDATE
You could create a hash code from the properties that are in a type, to see whether they have changed:
public static int GetPropertyHash(Type type) {
int hash = 237463;
foreach (PropertyInfo prop in type.GetProperties()) {
unchecked {
hash = hash * 17 + prop.Name.GetHashCode();
hash = hash * 17 + prop.PropertyType.FullName.GetHashCode();
}
}
return hash;
}
If this is not sufficient, then create a separate application that calculates the hash of the relevant code files and stores the result as C# code file into this project. Or use a T4 template inside the project to generate this file. The generated file could be something like this:
public static class SouceCode
{
public static Dictionary<string, int> Hash => new Dictionary<string, int> {
["MyView1.cs"] = 1382683,
["MyPoco.cs"] = -598674579,
};
}
The hash is only different if you use the built-in String.GetHashCode method (inherited from Object). If you use your own hashing method (more difficult), or an implementation such as HashCode (in .NET Core), you get what you want.
Getting a stable hash is part of the solution, already covered in other answers.
If you need access to the C# files at runtime, you'll need to copy them somehow at build time. In the project file, you can either Embed the files and then extract them from the assembly manifest, or Copy the files to bin/ and read them off disk.
Bit longer question ahead but please bear with me: A CosmosDB PM explains in this feedback thread:
In Cosmos DB, all of the resources e.g. databases, collections/tables/graphs, users, permissions, documents/items/nodes/edges, attachments are all runtime resources. You can CRUD/query these resources using runtime SDKs and REST APIs. [...] All of the “runtime resources” [...] are meant to be used by the developers directly inside their applications.
The only resource which is meant for administrative purposes is the “database account”. This resource is exposed via ARM.
So only the resource "CosmosDB" can be provisioned using ARM (Azure Resource Manager), e.g. in a CI/CD pipeline for instance with Azure DevOps.
So now my question is: What is the proper way to create the Database(s) and Collection(s) inside a CosmosDB account?
Let's say I am using am using an Azure Function that stores/reads data from a CosmosDB. Using the Function Binding I could for example use
[DocumentDB("ToDoList", "Migration", ConnectionStringSetting = "CosmosDB", CreateIfNotExists = true)] IAsyncCollector<Document> documentsToStore)
to create db and collection. Instantiating the DocumentClient manually I could use
await client.CreateDatabaseIfNotExistsAsync(database);
But: Is this the proper way to do this?? Doing it for instance in the Function binding would mean that the collection wouldn't be created until the Function is executed for the first time. That just does not feel right.
Or should one instead use for instance a Powershell script in the deployment pipeline to create db and collection after the ARM script was deployed? This is of course doable but if that would be the recommended way, one can very much argue that this should be exposed through ARM.
The answer to this question is entirely dependent on the intent of the function you are trying to build I feel. If the function assumes it can read/update/delete documents in a collection that might or might not exist then yes it a good practice to challenge the collection and then create it on the fly if it does not exist yet.
If the collection needs to be created because other tools depend on it and it is a predictable collection you can create them beforehand but if the collections name or number cannot be predicted then you are in a bind.
Basically I know what the difference is. Also, I do know (as you all) there is a lot of resources discussing the issue (e.g. here).
But today, I heard an strange and weird idea which says Add method, adds the object to application-scope cache, while the Insert method will adds the object to the session-scope cache!!!
I mean HttpContext.Current.Cache.Add will add object to a shared memory which is accessible for all users/threads/sessions/requests/etc.
But HttpContext.Current.Cache.Insert will add the object to a session-dependent cache (which actually would be the session itself. right? :D)!!!
It's the first time I'm hearing such a thing. I did search the msdn and technet.microsoft.com. I also did decompile the Cache class (via dear ReSharper) and read the entire code (and it's dependencies). But nothing came out which make a dependency between Cache and Session.
I'm just asking. Is there any relationship between Cache and Session?
Note: The app is running on a web garden in a Win 2012 R2 machine with IIS 8.5.
There are separate and in a different scope to each other.
As far as I can see, both Methods are applicable to the Cache object and calling Add will not have any interaction with the Session object
I'm wondering if there are any differences between MemoryCache and HttpRuntime.Cache, which one is preferred in ASP.NET MVC projects?
As far as I understand, both are thread safe, API is from first sight more or less the same, so is there any difference when to use which?
HttpRuntime.Cache gets the Cache for the current application.
The MemoryCache class is similar to the ASP.NET Cache class.
The MemoryCache class has many properties and methods for accessing the cache that will be familiar to you if you have used the ASP.NET Cache class.
The main difference between HttpRuntime.Cache and MemoryCache is that the latter has been changed to make it usable by .NET Framework applications that are not ASP.NET applications.
For additional reading:
Justin Mathew Blog - Caching in .Net 4.0
Jon Davis Blog - Four Methods Of Simple Caching In .NET
Update :
According to the users feedback, sometimes Jon davis blog is not working.Hence I have put the whole article as an image.Please see that.
Note : If it's not clear then just click on the image.After that it'll open on a browser.Then click again on it to zoom :)
Here is Jon Davis' article. To preserve readability, I'm cutting out the now obsolete EntLib section, the intro as well as the conclusion.
ASP.NET Cache
ASP.NET, or the System.Web.dll assembly, does have a caching mechanism. It was never intended to be used outside of a web context, but it can be used outside of the web, and it does perform all of the above expiration behaviors in a hashtable of sorts.
After scouring Google, it appears that quite a few people who have discussed the built-in caching functionality in .NET have resorted to using the ASP.NET cache in their non-web projects. This is no longer the most-available, most-supported built-in caching system in .NET; .NET 4 has an ObjectCache which I’ll get into later. Microsoft has always been adamant that the ASP.NET cache is not intended for use outside of the web. But many people are still stuck in .NET 2.0 and .NET 3.5, and need something to work with, and this happens to work for many people, even though MSDN says clearly:
Note: The Cache class is not intended for use outside of ASP.NET applications. It was designed and tested for use in ASP.NET to provide caching for Web applications. In other types of applications, such as console applications or Windows Forms applications, ASP.NET caching might not work correctly.
The class for the ASP.NET cache is System.Web.Caching.Cache in System.Web.dll. However, you cannot simply new-up a Cache object. You must acquire it from System.Web.HttpRuntime.Cache.
Cache cache = System.Web.HttpRuntime.Cache;
Working with the ASP.NET cache is documented on MSDN here.
Pros:
It’s built-in.
Despite the .NET 1.0 syntax, it’s fairly simple to use.
When used in a web context, it’s well-tested. Outside of web contexts, according to Google searches it is not commonly known to cause problems, despite Microsoft recommending against it, so long as you’re using .NET 2.0 or later.
You can be notified via a delegate when an item is removed, which is necessary if you need to keep it alive and you could not set the item’s priority in advance.
Individual items have the flexibility of any of (a), (b), or (c) methods of expiration and removal in the list of removal methods at the top of this article. You can also associate expiration behavior with the presence of a physical file.
Cons:
Not only is it static, there is only one. You cannot create your own type with its own static instance of a Cache. You can only have one bucket for your entire app, period. You can wrap the bucket with your own wrappers that do things like pre-inject prefixes in the keys and remove these prefixes when you pull the key/value pairs back out. But there is still only one bucket. Everything is lumped together. This can be a real nuisance if, for example, you have a service that needs to cache three or four different kinds of data separately. This shouldn’t be a big problem for pathetically simple projects. But if a project has any significant degree of complexity due to its requirements, the ASP.NET cache will typically not suffice.
Items can disappear, willy-nilly. A lot of people aren’t aware of this—I wasn’t, until I refreshed my knowledge on this cache implementation. By default, the ASP.NET cache is designed to destroy items when it “feels” like it. More specifically, see (c) in my definition of a cache table at the top of this article. If another thread in the same process is working on something completely different, and it dumps high-priority items into the cache, then as soon as .NET decides it needs to require some memory it will start to destroy some items in the cache according to their priorities, lower priorities first. All of the examples documented here for adding cache items use the default priority, rather than the NotRemovable priority value which keeps it from being removed for memory-clearing purposes but will still remove it according to the expiration policy. Peppering CacheItemPriority.NotRemovable in cache invocations can be cumbersome, otherwise a wrapper is necessary.
The key must be a string. If, for example, you are caching data records where the records are keyed on a long or an integer, you must convert the key to a string first.
The syntax is stale. It’s .NET 1.0 syntax, even uglier than ArrayList or Hashtable. There are no generics here, no IDictionary<> interface. It has no Contains() method, no Keys collection, no standard events; it only has a Get() method plus an indexer that does the same thing as Get(), returning null if there is no match, plus Add(), Insert() (redundant?), Remove(), and GetEnumerator().
Ignores the DRY principle of setting up your default expiration/removal behaviors so you can forget about them. You have to explicitly tell the cache how you want the item you’re adding to expire or be removed every time you add add an item.
No way to access the caching details of a cached item such as the timestamp of when it was added. Encapsulation went a bit overboard here, making it difficult to use the cache when in code you’re attempting to determine whether a cached item should be invalidated against another caching mechanism (i.e. session collection) or not.
Removal events are not exposed as events and must be tracked at the time of add.
And if I haven’t said it enough, Microsoft explicitly recommends against it outside of the web. And if you’re cursed with .NET 1.1, you not supposed to use it with any confidence of stability at all outside of the web so don’t bother.
.NET 4.0’s ObjectCache / MemoryCache
Microsoft finally implemented an abstract ObjectCache class in the latest version of the .NET Framework, and a MemoryCache implementation that inherits and implements ObjectCache for in-memory purposes in a non-web setting.
System.Runtime.Caching.ObjectCache is in the System.Runtime.Caching.dll assembly. It is an abstract class that that declares basically the same .NET 1.0 style interfaces that are found in the ASP.NET cache. System.Runtime.Caching.MemoryCache is the in-memory implementation of ObjectCache and is very similar to the ASP.NET cache, with a few changes.
To add an item with a sliding expiration, your code would look something like this:
var config = new NameValueCollection();
var cache = new MemoryCache("myMemCache", config);
cache.Add(new CacheItem("a", "b"),
new CacheItemPolicy
{
Priority = CacheItemPriority.NotRemovable,
SlidingExpiration=TimeSpan.FromMinutes(30)
});
Pros:
It’s built-in, and now supported and recommended by Microsoft outside of the web.
Unlike the ASP.NET cache, you can instantiate a MemoryCache object instance.
Note: It doesn’t have to be static, but it should be—that is Microsoft’s recommendation (see yellow Caution).
A few slight improvements have been made vs. the ASP.NET cache’s interface, such as the ability to subscribe to removal events without necessarily being there when the items were added, the redundant Insert() was removed, items can be added with a CacheItem object with an initializer that defines the caching strategy, and Contains() was added.
Cons:
Still does not fully reinforce DRY. From my small amount of experience, you still can’t set the sliding expiration TimeSpan once and forget about it. And frankly, although the policy in the item-add sample above is more readable, it necessitates horrific verbosity.
It is still not generically-keyed; it requires a string as the key. So you can’t store as long or int if you’re caching data records, unless you convert to string.
DIY: Build One Yourself
It’s actually pretty simple to create a caching dictionary that performs explicit or sliding expiration. (It gets a lot harder if you want items to be auto-removed for memory-clearing purposes.) Here’s all you have to do:
Create a value container class called something like Expiring or Expirable that would contain a value of type T, a TimeStamp property of type DateTime to store when the value was added to the cache, and a TimeSpan that would indicate how far out from the timestamp that the item should expire. For explicit expiration you can just expose a property setter that sets the TimeSpan given a date subtracted by the timestamp.
Create a class, let’s call it ExpirableItemsDictionary, that implements IDictionary. I prefer to make it a generic class with defined by the consumer.
In the the class created in #2, add a Dictionary> as a property and call it InnerDictionary.
The implementation if IDictionary in the class created in #2 should use the InnerDictionary to store cached items. Encapsulation would hide the caching method details via instances of the type created in #1 above.
Make sure the indexer (this[]), ContainsKey(), etc., are careful to clear out expired items and remove the expired items before returning a value. Return null in getters if the item was removed.
Use thread locks on all getters, setters, ContainsKey(), and particularly when clearing the expired items.
Raise an event whenever an item gets removed due to expiration.
Add a System.Threading.Timer instance and rig it during initialization to auto-remove expired items every 15 seconds. This is the same behavior as the ASP.NET cache.
You may want to add an AddOrUpdate() routine that pushes out the sliding expiration by replacing the timestamp on the item’s container (Expiring instance) if it already exists.
Microsoft has to support its original designs because its user base has built up a dependency upon them, but that does not mean that they are good designs.
Pros:
You have complete control over the implementation.
Can reinforce DRY by setting up default caching behaviors and then just dropping key/value pairs in without declaring the caching details each time you add an item.
Can implement modern interfaces, namely IDictionary<K,T>. This makes it much easier to consume as its interface is more predictable as a dictionary interface, plus it makes it more accessible to helpers and extension methods that work with IDictionary<>.
Caching details can be unencapsulated, such as by exposing your InnerDictionary via a public read-only property, allowing you to write explicit unit tests against your caching strategy as well as extend your basic caching implementation with additional caching strategies that build upon it.
Although it is not necessarily a familiar interface for those who already made themselves comfortable with the .NET 1.0 style syntax of the ASP.NET cache or the Caching Application Block, you can define the interface to look like however you want it to look.
Can use any type for keys. This is one reason why generics were created. Not everything should be keyed with a string.
Cons:
Is not invented by, nor endorsed by, Microsoft, so it is not going to have the same quality assurance.
Assuming only the instructions I described above are implemented, does not “willy-nilly” clear items for clearing memory on a priority basis (which is a corner-case utility function of a cache anyway .. BUY RAM where you would be using the cache, RAM is cheap).
Among all four of these options, this is my preference. I have implemented this basic caching solution. So far, it seems to work perfectly, there are no known bugs (please contact me with comments below or at jon-at-jondavis if there are!!), and I intend to use it in all of my smaller side projects that need basic caching. Here it is:
Github link: https://github.com/kroimon/ExpirableItemDictionary
Old Link: ExpirableItemDictionary.zip
Worthy Of Mention: AppFabric, NoSQL, Et Al
Notice that the title of this blog article indicates “Simple Caching”, not “Heavy-Duty Caching”. If you want to get into the heavy-duty stuff, you should look at dedicated, scale out solutions.
MemoryCache.Default can also serve as a "bridge" if you're migrating a classic ASP.NET MVC app to ASP.NET Core, because there's no "System.Web.Caching" and "HttpRuntime" in Core.
I also wrote a small benchmark to store a bool item 20000 times (and another benchmark to retrieve it) and MemoryCache seems to be two times slower (27ms vs 13ms - that's total for all 20k iterations) but they're both super-fast and this can probably be ignored.
MemoryCache is what it says it is, a cache stored in memory
HttpRuntime.Cache (see http://msdn.microsoft.com/en-us/library/system.web.httpruntime.cache(v=vs.100).aspx and http://msdn.microsoft.com/en-us/library/system.web.caching.cache.aspx) persists to whatever you configure it to in your application.
see for example "ASP.NET 4.0: Writing custom output cache providers"
http://weblogs.asp.net/gunnarpeipman/archive/2009/11/19/asp-net-4-0-writing-custom-output-cache-providers.aspx
Our applications use a lot of shared components. Some of them have no need for caching, for example, Windows Services which process unmailed emails. You'd never cache that result set...
Problem is, since our shared data layer has been modified to use SqlCacheDependency, our services which don't start SqlDependency fail on database calls where the data layer requests a SqlCacheDependency object.
Which leads to the question - is there a way for our data classes to test to see if the broker service is listening (ie: has SqlDependency.Start(connectionString) been called)?
The SqlDependency object itself has no Enabled or similar property. Is there any way short of forcing the calling app to tell the data layer that SqlCaching is in use for the data layer to determine the state?
Pretty much the answer is no. We ended up adding a config variable that if false or not present, causes the request to use SqlCacheDependency to be skipped.
SELECT * FROM sys.service_queues WHERE name LIKE 'SqlQueryNotificationService-%'
returns a 'SqlQueryNotificationService-[some guid]'
And if you look deep in the Non-public members of the SqlDependency _serverUserHash while debugging in the IDE, you'll find a collection that contains that same entry. If Microsoft would be so kind as to expose that, then yes.
In my case I have a class library which is used by some web applications. So I have no App.config. I also use the SqlCacheDependency in a static event. So I'm using a static boolean like:
if (!isCachingEnabled)
isCachingEnabled = SqlDependency.Start(builder.ProviderConnectionString);
So far is working but I'm open to suggestions when using Class Libraries.