Develop Reference

C# LINQ and calculations involving large datasets

This is more of a technical "how-to" or "best approach" question.
We have a current requirement to retrieve records from the database, place them into an 'in-memory' list and then perform a series of calculations on the data, i.e. maximum values, averages and some more specific custom statistics as well.
Getting the data into an 'in-memory' list is not a problem as we use NHibernate as our ORM and it does an excellent job of retrieving data from the database. The advice I am seeking is how should we best perform calculations on the resulting list of data.
Ideally I would like to create a method for each statistic, MaximumValue(), AverageValueUnder100(), MoreComplicatedStatistic() etc etc. Of course passing the required variables to each method and having it return the result. This approach would also make unit testing a breeze and provide us with excellent coverage.
Would there be a performance hit if we perform a LINQ query for each calculation or should be consolidate as many calls to each statistic method in as few LINQ queries as possible. For example it doesn't make much sense to pass the list of data to a method called AverageValueBelow100 and then pass the entire list of data to another method AverageValueBelow50 when they could effectively be performed with one LINQ query.
How can we achieve a high level of granularity and separation without sacrificing performance?
Any advice ... is the question clear enough?

Depending on the complexity of the calculation, it may be best to do it in the database. If it is signifcantly complex that you need to bring it in as objects and encur that overhead, you may want to avoid multiple iterations over your result set. you may want to consider using Aggregate. See http://geekswithblogs.net/malisancube/archive/2009/12/09/demystifying-linq-aggregates.aspx for a discussion if it. You would be able to unit test each aggregate separately, but then (potentially) project multiple aggregates within a single iteration.

I dont agree that it is best "to do it all in the database".
Well written Linq Queries will result in good SQL queries being executed against the database, which should be good enough performance wise (if you are not going to do dwh stuff). This is assuming you are using the Linq Provider for NHibernate and not Linq to Objects.
It does look good, you can change it easily and keeps your business logic in one place.
If that is too slow for your needs, you might check the SQL code created and tweak your linq queries, are try to precompile them, and in the end you can still go back to writing the beloved stored procedures - and start to spread your business logic all over the place.
Will there be a performance hit? Yeah, you might lose a few millisecs, but is that worth the price you have to pay for separating your logic?

To answer the "I would like to create a method for each statistic" concern, I would suggest you to build a kind of statistician class. Here is some pseudo code to express the idea :
class Statistician
{
public bool MustCalculateFIRSTSTATISTIC { get; set; } // Please rename me!
public bool MustCalculateSECONDSTATISTIC { get; set; } // Please rename me!
public void ProcessObject(object Object) // Replace object and Rename
{
if (MustCalculateFIRSTSTATISTIC)
CalculateFIRSTSTATISTIC(Object);
if (MustCalculateFIRSTSTATISTIC)
CalculateSECONDSTATISTIC(Object);
}
public object GetFIRSTSTATISTIC() // Replace object, Rename
{ /* ... */ }
public object GetSECONDSTATISTIC() // Replace object, Rename
{ /* ... */ }
private void CalculateFIRSTSTATISTIC(object Object) // Replace object
{ /* ... */ }
private void CalculateSECONDSTATISTIC(object Object) // Replace object
{ /* ... */ }
}
Would I have to do this, I would probably try to make it generic and use collections of delegates instead of methods, but since I don't know your context, I'll leave it to that. Also note that I only used Object members of object class, but that's only because I'm not suggesting you to use DataRows, Entities, or what not; I'll leave that to the other folks that know more then me on the subject!

How can I avoid adding getters to facilitate unit testing?

After reading a blog post mentioning how it seems wrong to expose a public getter just to facilitate testing, I couldn't find any concrete examples of better practices.
Suppose we have this simple example:
public class ProductNameList
{
private IList<string> _products;
public void AddProductName(string productName)
{
_products.Add(productName);
}
}
Let's say for object-oriented design reasons, I have no need to publicly expose the list of products. How then can I test whether AddProductName() did its job? (Maybe it added the product twice, or not at all.)
One solution is to expose a read-only version of _products where I can test whether it has only one product name -- the one I passed to AddProductName().
The blog post I mentioned earlier says it's more about interaction (i.e., did the product name get added) rather than state. However, state is exactly what I'm checking. I already know AddProductName() has been called -- I want to test whether the object's state is correct once that method has done its work.
Disclaimer: Although this question is similar to
Balancing Design Principles: Unit Testing, (1) the language is different (C# instead of Java), (2) this question has sample code, and (3) I don't feel the question was adequately answered (i.e., code would have helped demonstrate the concept).

Unit tests should test the public API.
If you have "no need to publicly expose the list of products", then why would you care whether AddProductName did its job? What possible difference would it make if the list is entirely private and never, ever affected anything else?
Find out what affect AddProductName has on the state that can be detected using the API, and test that.
Very similar question here: Domain Model (Exposing Public Properties)

You could make your accessors protected so you can mock or you could use internal so that you can access the property in a test but that IMO would be wrong as you have suggested.
I think sometimes we get so caught up in wanting to make sure that every little thing in our code is tested. Sometime we need to take a step back and ask why are we storing this value, and what is its purpose? Then instead of testing that the value gets set we can then start testing that the behaviour of the component is correct.
EDIT
One thing you can do in your scenario is to have a bastard constructor where you inject an IList and then you test that you have added a product:
public class ProductNameList
{
private IList<string> _products;
internal ProductNameList(IList<string> products)
{
_products = products;
}
...
}
You would then test it like this:
[Test]
public void FooTest()
{
var productList = new List<string>();
var productNameList = new ProductNameList(productList);
productNameList.AddProductName("Foo");
Assert.IsTrue(productList[0] == "Foo");
}
You will need to remember to make internals visable to your test assembly.

Make _products protected instead of private. In your mock, you can add an accessor.

To test if AddProductName() did it's job, instead of using a public getter for _ProductNames, make a call to GetProductNames() - or the equivalent that's defined in your API. Such a function may not necessarily be in the same class.
Now, if your API does not expose any way to get information about product names, then AddProductName() has no observable side effects (In which case, it is a meaningless function).
If AddProductName() does have side effects, but they are indirect - say, a method in ProductList that writes the list of product names to a file, then ProductList should be split into two classes - one that manages the list, and the other that calls it's Add and Get API, and performs side effects.

C# Where to load initial data in an object?

Similar to this question:
C# Constructor Design but this question is slight different.
I have a class Customer and a class CustomerManager. When an instance is created of the CustomerManager class I want to load all the customers. And this is where I got stuck. I can do this several ways:
Load all the customers in the constructor (I don't like this one because it can take a while if I have many customers)
In every method of the CustomerManager class that performs database related actions, check the local list of customers is loaded and if not, load the list:
public method FindCustomer(int id)
{
if(_customers == null)
// some code which will load the customers list
}
Create a method which loads all the customers. This method must be called before calling methods which performs database related actions:
In the class:
public LoadData()
{
// some code which will load the customers list
}
In the form:
CustomerManager manager = new CustomerManager();
manager.LoadData();
Customer customer = manager.FindCustomer(int id);
What is the best way to do this?
EDIT:
I have the feeling that I am misunderstood here. Maybe it is because I wasn't clear enough. In the CustomerManager class I have several methods which depends on the local list (_customers). So, my question is, where should I fill that list?

What you are describing is "lazy loading".
A simple approach is to have a private property like this:
private Lixt<Customer> _customers;
private List<Customer> Customers
{
get
{
if(_customers == null)
_customers = LoadData();
return _customers;
}
}
Then, you refer to Customers internally. The customers will be loaded the first time they are needed but no earlier.
This is such a common pattern that .Net 4.0 added a Lazy<T> class that does this for you.
I that case, you just define it as a private like this:
private Lazy<List<Customer>> _customers = new Lazy<List<Customer>>(LoadData);
Then, you simply refer to your customers in code:
_customers.Value
The class will initialize the value with your LoadData() method.
If you are not on .Net 4.0 yet, the Lazy<T> class is very easy to implement.

Use a property for accessing the customers. Have that check if the customers are loaded.

Well, it depends. All your options have advantages and disadvantages.
The good thing about options 1 and 3 is that the user has full control over when the (lengthy) data loading operation is performed. Whether option 1 or 3 is better depends on whether it makes sense to create the Manager and load the data later or not. Personally, I prefer a separate LoadData method if it's a lengthy operation, but that might be a matter of taste.
The good thing about option 2 is that the data will not be loaded if it is not needed. The drawback is that the (lengthy) load occurs as a side-affect of the first access, which makes your program "less deterministic".
In principle, all the options you have presented are fine and valid choices. It really depends on your requirements.

Is it ok to use C# Property like this

One of my fellow developer has a code similar to the following snippet
class Data
{
public string Prop1
{
get
{
// return the value stored in the database via a query
}
set
{
// Save the data to local variable
}
}
public void SaveData()
{
// Write all the properties to a file
}
}
class Program
{
public void SaveData()
{
Data d = new Data();
// Fetch the information from database and fill the local variable
d.Prop1 = d.Prop1;
d.SaveData();
}
}
Here the Data class properties fetch the information from DB dynamically. When there is a need to save the Data to a file the developer creates an instance and fills the property using self assignment. Then finally calls a save. I tried arguing that the usage of property is not correct. But he is not convinced.
This are his points
There are nearly 20 such properties.
Fetching all the information is not required except for saving.
Instead of self assignment writing an utility method to fetch all will have same duplicate code in the properties.
Is this usage correct?

I don't think that another developer who will work with the same code will be happy to see :
d.Prop1 = d.Prop1;
Personally I would never do that.
Also it is not the best idea to use property to load data from DB.
I would have method which will load data from DB to local variable and then you can get that data using property. Also get/set logically must work with the same data. It is strange to use get for getting data from DB but to use set to work with local variable.

Properties should really be as lightweight as possible.
When other developers are using properties, they expect them to be intrinsic parts of the object (that is, already loaded and in memory).
The real issue here is that of symmetry - the property get and set should mirror each other, and they don't. This is against what most developers would normally expect.
Having the property load up from database is not recommended - normally one would populate the class via a specific method.

This is pretty terrible, imo.
Properties are supposed to be quick / easy to access; if there's really heavy stuff going on behind a property it should probably be a method instead.
Having two utterly different things going on behind the same property's getter and setter is very confusing. d.Prop1 = d.Prop1 looks like a meaningless self-assignment, not a "Load data from DB" call.
Even if you do have to load twenty different things from a database, doing it this way forces it to be twenty different DB trips; are you sure multiple properties can't be fetched in a single call? That would likely be much better, performance-wise.

"Correct" is often in the eye of the beholder. It also depends how far or how brilliant you want your design to be. I'd never go for the design you describe, it'll become a maintenance nightmare to have the CRUD actions on the POCOs.
Your main issue is the absense of separations of concerns. I.e., The data-object is also responsible for storing and retrieving (actions that need to be defined only once in the whole system). As a result, you end up with duplicated, bloated and unmaintainable code that may quickly become real slow (try a LINQ query with a join on the gettor).
A common scenario with databases is to use small entity classes that only contain the properties, nothing more. A DAO layer takes care of retrieving and filling these POCOs with data from the database and defined the CRUD actions only ones (through some generics). I'd suggest NHibernate for the ORM mapping. The basic principle explained here works with other ORM mappers too and is explained here.
The reasons, esp. nr 1, should be a main candidate for refactoring this into something more maintainable. Duplicated code and logic, when encountered, should be reconsidered strongly. If the gettor above is really getting the database data (I hope I misunderstand that), get rid of it as quickly as you can.
Overly simplified example of separations of concerns:
class Data
{
public string Prop1 {get; set;}
public string Prop2 {get; set;}
}
class Dao<T>
{
SaveEntity<T>(T data)
{
// use reflection for saving your properies (this is what any ORM does for you)
}
IList<T> GetAll<T>()
{
// use reflection to retrieve all data of this type (again, ORM does this for you)
}
}
// usage:
Dao<Data> myDao = new Dao<Data>();
List<Data> allData = myDao.GetAll();
// modify, query etc using Dao, lazy evaluation and caching is done by the ORM for performance
// but more importantly, this design keeps your code clean, readable and maintainable.
EDIT:
One question you should ask your co-worker: what happens if you have many Data (rows in database), or when a property is a result of a joined query (foreign key table). Have a look at Fluent NHibernate if you want a smooth transition from one situation (unmaintainable) to another (maintainable) that's easy enough to understand by anybody.

If I were you I would write a serialize / deserialize function, then provide properties as lightweight wrappers around the in-memory results.
Take a look at the ISerialization interface: http://msdn.microsoft.com/en-us/library/system.runtime.serialization.iserializable.aspx

This would be very hard to work with,
If you set the Prop1, and then get Prop1, you could end up with different results
eg:
//set Prop1 to "abc"
d.Prop1 = "abc";
//if the data source holds "xyz" for Prop1
string myString = d.Prop1;
//myString will equal "xyz"
reading the code without the comment you would expect mystring to equal "abc" not "xyz", this could be confusing.
This would make working with the properties very difficult and require a save every time you change a property for it to work.

As well as agreeing with what everyone else has said on this example, what happens if there are other fields in the Data class? i.e. Prop2, Prop3 etc, do they all go back to the database, each time they are accessed in order to "return the value stored in the database via a query". 10 properties would equal 10 database hits. Setting 10 properties, 10 writes to the database. That's not going to scale.

In my opinion, that's an awful design. Using a property getter to do some "magic" stuff makes the system awkward to maintain. If I would join your team, how should I know that magic behind those properties?
Create a separate method that is called as it behaves.

When to use Properties and Methods?

I'm new to the .NET world having come from C++ and I'm trying to better understand properties. I noticed in the .NET framework Microsoft uses properties all over the place. Is there an advantage for using properties rather than creating get/set methods? Is there a general guideline (as well as naming convention) for when one should use properties?

It is pure syntactic sugar. On the back end, it is compiled into plain get and set methods.
Use it because of convention, and that it looks nicer.
Some guidelines are that when it has a high risk of throwing Exceptions or going wrong, don't use properties but explicit getters/setters. But generally even then they are used.

Properties are get/set methods; simply, it formalises them into a single concept (for read and write), allowing (for example) metadata against the property, rather than individual members. For example:
[XmlAttribute("foo")]
public string Name {get;set;}
This is a get/set pair of methods, but the additional metadata applies to both. It also, IMO, simply makes it easier to use:
someObj.Name = "Fred"; // clearly a "set"
DateTime dob = someObj.DateOfBirth; // clearly a "get"
We haven't duplicated the fact that we're doing a get/set.
Another nice thing is that it allows simple two-way data-binding against the property ("Name" above), without relying on any magic patterns (except those guaranteed by the compiler).

There is an entire book dedicated to answering these sorts of questions: Framework Design Guidelines from Addison-Wesley. See section 5.1.3 for advice on when to choose a property vs a method.
Much of the content of this book is available on MSDN as well, but I find it handy to have it on my desk.

Consider reading Choosing Between Properties and Methods. It has a lot of information on .NET design guidelines.

properties are get/set methods

Properties are set and get methods as people around here have explained, but the idea of having them is making those methods the only ones playing with the private values (for instance, to handle validations).
The whole other logic should be done against the properties, but it's always easier mentally to work with something you can handle as a value on the left and right side of operations (properties) and not having to even think it is a method.
I personally think that's the main idea behind properties.

I always think that properties are the nouns of a class, where as methods are the verbs...

First of all, the naming convention is: use PascalCase for the property name, just like with methods. Also, properties should not contain very complex operations. These should be done kept in methods.
In OOP, you would describe an object as having attributes and functionality. You do that when designing a class. Consider designing a car. Examples for functionality could be the ability to move somewhere or activate the wipers. Within your class, these would be methods. An attribute would be the number of passengers within the car at a given moment. Without properties, you would have two ways to implement the attribute:
Make a variable public:
// class Car
public int passengerCount = 4;
// calling code
int count = myCar.passengerCount;
This has several problems. First of all, it is not really an attribute of the vehicle. You have to update the value from inside the Car class to have it represent the vehicles true state. Second, the variable is public and could also be written to.
The second variant is one widley used, e. g. in Java, where you do not have properties like in c#:
Use a method to encapsulate the value and maybe perform a few operations first.
// class Car
public int GetPassengerCount()
{
// perform some operation
int result = CountAllPassengers();
// return the result
return result;
}
// calling code
int count = myCar.GetPassengerCount();
This way you manage to get around the problems with a public variable. By asking for the number of passengers, you can be sure to get the most recent result since you recount before answering. Also, you cannot change the value since the method does not allow it. The problem is, though, that you actually wanted the amount of passengers to be an attribute, not a function of your car.
The second approach is not necessarily wrong, it just does not read quite right. That's why some languages include ways of making attributes look like variables, even though they work like methods behind the scenes. Actionscript for example also includes syntax to define methods that will be accessed in a variable-style from within the calling code.
Keep in mind that this also brings responsibility. The calling user will expect it to behave like an attribute, not a function. so if just asking a car how many passengers it has takes 20 seconds to load, then you probably should pack that in a real method, since the caller will expect functions to take longer than accessing an attribute.
EDIT:
I almost forgot to mention this: The ability to actually perform certain checks before letting a variable be set. By just using a public variable, you could basically write anything into it. The setter method or property give you a chance to check it before actually saving it.

Properties simply save you some time from writing the boilerplate that goes along with get/set methods.
That being said, a lot of .NET stuff handles properties differently- for example, a Grid will automatically display properties but won't display a function that does the equivalent.
This is handy, because you can make get/set methods for things that you don't want displayed, and properties for those you do want displayed.

The compiler actually emits get_MyProperty and set_MyProperty methods for each property you define.

Although it is not a hard and fast rule and, as others have pointed out, Properties are implemented as Get/Set pairs 'behind the scenes' - typically Properties surface encapsulated/protected state data whereas Methods (aka Procedures or Functions) do work and yield the result of that work.
As such Methods will take often arguments that they might merely consume but also may return in an altered state or may produce a new object or value as a result of the work done.
Generally speaking - if you need a way of controlling access to data or state then Properties allow the implementation that access in a defined, validatable and optimised way (allowing access restriction, range & error-checking, creation of backing-store on demand and a way of avoiding redundant setting calls).
In contrast, methods transform state and give rise to new values internally and externally without necessarily repeatable results.
Certainly if you find yourself writing procedural or transformative code in a property, you are probably really writing a method.

Also note that properties are available via reflection. While methods are, too, properties represent "something interesting" about the object. If you are trying to display a grid of properties of an object-- say, something like the Visual Studio form designer-- then you can use reflection to query the properties of a class, iterate through each property, and interrogate the object for its value.

Think of it this way, Properties encapsulate your fields (commoningly marked private) while at the same time provides your fellow developers to either set or get the field value. You can even perform routine validation in the property's set method should you desire.

Properties are not just syntactic sugar - they are important if you need to create object-relational mappings (Linq2Sql or Linq2Entities), because they behave just like variables while it is possible to hide the implementation details of the object-relational mapping (persistance). It is also possible to validate a value being assigned to it in the getter of the property and protect it against assigning unwanted values.
You can't do this with the same elegance with methods. I think it is best to demonstrate this with a practical example.
In one of his articles, Scott Gu creates classes which are mapped to the Northwind database using the "code first" approach. One short example taken from Scott's blog (with a little modification, the full article can be read at Scott Gu's blog here):
public class Product
{
[Key]
public int ProductID { get; set; }
public string ProductName { get; set; }
public Decimal? UnitPrice { get; set; }
public bool Discontinued { get; set; }
public virtual Category category { get; set; }
}
// class Category omitted in this example
public class Northwind : DbContext
{
public DbSet<Product> Products { get; set; }
public DbSet<Category> Categories { get; set; }
}
You can use entity sets Products, Categories and the related classes Product and Category just as if they were normal objects containing variables: You can read and write them and they behave just like normal variables. But you can also use them in Linq queries, persist them (store them in the database and retrieve them).
Note also how easy it is to use annotations (C# attributes) to define the primary key (in this example ProductID is the primary key for Product).
While the properties are used to define a representation of the data stored in the database, there are some methods defined in the entity set class which control the persistence: For example, the method Remove() marks a given entity as deleted, while Add() adds a given entity, SaveChanges() makes the changes permanent. You can consider the methods as actions (i.e. you control what you want to do with the data).
Finally I give you an example how naturally you can use those classes:
// instantiate the database as object
var nw = new NorthWind();
// select product
var product = nw.Products.Single(p => p.ProductName == "Chai");
// 1. modify the price
product.UnitPrice = 2.33M;
// 2. store a new category
var c = new Category();
c.Category = "Example category";
c.Description = "Show how to persist data";
nw.Categories.Add(c);
// Save changes (1. and 2.) to the Northwind database
nw.SaveChanges();