Develop Reference

.NET 6.0 application some times takes long time to create objects

We see a strange behaviour in my .NET 6.0 application in my production environment. Shortly described the applications endpoint does the follwoing:
Fetches a list of objects from a database
Maps the objects from SQLReader to a model (with some custom logic)
Returns the list
Seems simple and it runs very fast for the most time being. But some times we see a very long running task between the mapping of SQLReader output to the model class.
The mapping follows this object creation:
private static MyCustomObject MapRecord(int indexNo, long id, IDataReader reader)
return new MyCustomObject(indexNo,
reader.GetDateTime(DATE),
id,
reader.GetString(ID),
reader.TryRead(TEXT, string.Empty),
reader.GetDecimal(NUMBER1),
reader.GetDecimal(NUMBER2),
reader.TryRead(DATA, string.Empty),
reader.TryRead(CATEGORY_ID, 0),
GetCategoryScores(reader.TryRead(CATEGORIES, string.Empty)),
reader.TryRead<bool?>(SOMEBOOL, null),
reader.TryRead(SOMEOTHERBOOL, false),
reader.TryRead(COMMENT, string.Empty),
);
}
At first we suspected the GetCategoryScores, which does a string split in two dimensions:
private static IList<(int, double)>? GetCategoryScores(string? scores)
{
if (scores != null)
{
return scores.Split(';' , StringSplitOptions.RemoveEmptyEntries)
.Select(x =>
{
var split = x.Split(':');
var categoryScore = string.IsNullOrEmpty(split[1]) ? 0 : double.Parse(split[1], CultureInfo.InvariantCulture);
return (int.Parse(split[0]), categoryScore);
}).ToList();
}
return null;
}
My Jaeger traces showed that, it was NOT the case, hence i can only see the "TryRead" function to be the sinner. It looks like this:
public static T? TryRead<T>(this IDataReader reader, int ordinal, T? defaultOnNull) =>
reader.IsDBNull(ordinal)
? defaultOnNull
: (T)reader[ordinal];
My other suspicion is that it is when the GC collects unused objects, it takes time. Usually the execution time of the MapRecord function is negligible, but when it takes time it takes between 1s and 1.5s.
The application is deployed as a docker container and runs on a kubernetes cluster.
Has anyone seen anything like this before and have any suggestions of what to do?

If it's not GC, my guess is that it's the reader having to wait for data being streamed back from the database, most likely because other queries are locking tables and delaying the result, but maybe also due to general CPU/disk contention or networking issues.

Creating an object via lambda factory vs direct "new Type()" syntax

For example, consider a utility class SerializableList:
public class SerializableList : List<ISerializable>
{
public T Add<T>(T item) where T : ISerializable
{
base.Add(item);
return item;
}
public T Add<T>(Func<T> factory) where T : ISerializable
{
var item = factory();
base.Add(item);
return item;
}
}
Usually I'd use it like this:
var serializableList = new SerializableList();
var item1 = serializableList.Add(new Class1());
var item2 = serializableList.Add(new Class2());
I could also have used it via factoring, like this:
var serializableList = new SerializableList();
var item1 = serializableList.Add(() => new Class1());
var item2 = serializableList.Add(() => new Class2());
The second approach appears to be a preferred usage pattern, as I've been lately noticing on SO. Is it really so (and why, if yes) or is it just a matter of taste?

Given your example, the factory method is silly. Unless the callee requires the ability to control the point of instantiation, instantiate multiple instances, or lazy evaluation, it's just useless overhead.
The compiler will not be able to optimize out delegate creation.
To reference the examples of using the factory syntax that you gave in comments on the question. Both examples are trying (albeit poorly) to provide guaranteed cleanup of the instances.
If you consider a using statement:
using (var x = new Something()) { }
The naive implementation would be:
var x = new Something();
try
{
}
finally
{
if ((x != null) && (x is IDisposable))
((IDisposable)x).Dispose();
}
The problem with this code is that it is possible for an exception to occur after the assignment of x, but before the try block is entered. If this happens, x will not be properly disposed, because the finally block will not execute. To deal with this, the code for a using statement will actually be something more like:
Something x = null;
try
{
x = new Something();
}
finally
{
if ((x != null) && (x is IDisposable))
((IDisposable)x).Dispose();
}
Both of the examples that you reference using factory parameters are attempting to deal with this same issue. Passing a factory allows for the instance to be instantiated within the guarded block. Passing the instance directly allows for the possibility of something to go wrong along the way and not have Dispose() called.
In those cases, passing the factory parameter makes sense.

Caching
In the example you have provided it does not make sense as others have pointed out. Instead I will give you another example,
public class MyClass{
public MyClass(string file){
// load a huge file
// do lots of computing...
// then store results...
}
}
private ConcurrentDictionary<string,MyClass> Cache = new ....
public MyClass GetCachedItem(string key){
return Cache.GetOrAdd(key, k => new MyClass(key));
}
In above example, let's say we are loading a big file and we are calculating something and we are interested in end result of that calculation. To speedup my access, when I try to load files through Cache, Cache will return me cached entry if it has it, only when cache does not find the item, it will call the Factory method, and create new instance of MyClass.
So you are reading files many times, but you are only creating instance of class that holds data just once. This pattern is only useful for caching purpose.
But if you are not caching, and every iteration requires to call new operator, then it makes no sense to use factory pattern at all.
Alternate Error Object or Error Logging
For some reason, if creation fails, List can create an error object, for example,
T defaultObject = ....
public T Add<T>(Func<T> factory) where T : ISerializable
{
T item;
try{
item = factory();
}catch(ex){
Log(ex);
item = defaultObject;
}
base.Add(item);
return item;
}
In this example, you can monitor factory if it generates an exception while creating new object, and when that happens, you Log the error, and return something else and keep some default value in list. I don't know what will be practical use of this, but Error Logging sounds better candidate here.

No, there's no general preference of passing the factory instead of the value. However, in very particular situations, you will prefer to pass the factory method instead of the value.
Think about it:
What's the difference between passing the parameter as a value, or
passing it as a factory method (e.g. using Func<T>)?
Answer is simple: order of execution.
In the first case, you need to pass the value, so you must obtain it before calling the target method.
In the second case, you can postpone the value creation/calculation/obtaining till it's needed by the target method.
Why would you want to postpone the value creation/calculation/obtaining? obvious things come to mind:
Processor-intensive or memory-intensive creation of the value, that you want to happen only in case the value is really needed (on-demand). This is Lazy loading then.
If the value creation depends on parameters that are accessible by the target method but not from outside of it. So, you would pass Func<T, T> instead of Func<T>.

The question compares methods with different purposes. The second one should be named CreateAndAdd<T>(Func<T> factory).
So depending what functionality is required, should be used one or another method.

How to make NHibernate considered property to always be dirty when using dynamic update or insert?

I am looking for help on an issue with NHibernate which has been bugging me for a while now. Long story short:
I’m looking for a way to, in the first level cache, “reset” a property on an entity each time I do an update or an insert.
What I want to achieve is that the property in question will always be considered to be dirty by NHibernate when using dynamic update or insert.
The backstory for this is that I know that, if the transaction was successful, the column that I want to “reset” will be set to Null in the database by a trigger. On the flip side, the first level cache does not know this, and thus NHibernate will think that the property was not updated when I set it to the same value as I did on the previous update/insert. The catch is that my trigger is dependent on this value being set. The resulting mess is that if I want to use dynamic update or insert I’m only able to update/insert an entity once without “refreshing” it afterwards (which I really don’t want to do).
Tips or help would be much appreciated, because I’ve really hit a wall here

NHibernate provides many places for extension. Among them is the Session IInterceptor. There is documentation with many details:
http://nhibernate.info/doc/nh/en/index.html#objectstate-interceptors
In this case, we can create our custom one, which will be observing our entity (for example Client) and a property which must be updated every time (for example Code). So our implementation could look like this:
public class MyInterceptor : EmptyInterceptor
{
public override int[] FindDirty(object entity, object id, object[] currentState, object[] previousState, string[] propertyNames, NHibernate.Type.IType[] types)
{
var result = new List<int>();
// we do not care about other entities here
if(!(entity is Client))
{
return null;
}
var length = propertyNames.Length;
// iterate all properties
for(var i = 0; i < length; i++)
{
var areEqual = currentState[i].Equals(previousState[i]);
var isResettingProperty = propertyNames[i] == "Code";
if (!areEqual || isResettingProperty)
{
result.Add(i); // the index of "Code" property will be added always
}
}
return result.ToArray();
}
}
NOTE: This is just an example! Apply your own logic for checking the dirty properties.
And we have to wrap Session this way:
var interceptor = new MyInterceptor()
_configuration.SetInterceptor(interceptor);
And this is it. While Client is marked as dynamic-update, the property Code will always be set as dirty
<class name="Client" dynamic-update="true" ...

How can I implement Query Interception in a LINQ to Entities query? (c#)

I'm trying to implement encrypted columns in EF4 and using the CTP5 features to allow simple use of POCO's to query the database. Sorry that this is a lot of words, but I hope the below gives enough to explain the need and the problem!
So, bit of background, and my progress so far:
The intention is that if you query the tables without using our DAL then the data is rubbish, but I don't want the developers to worry about if/when/how the data is encrypted.
For simplicity, at this stage I'm working on the assumption any string column will be encrypted.
Now, I have successfully implemented this for returning the data using the Objectmaterialized event, and for data commits using the SavingChanges event.
So given the following class:
public class Thing
{
public int ID { get; set; }
[Required]
public string Name { get; set; }
public DateTime Date { get; set; }
public string OtherString { get; set; }
}
The below query returns all the required values and the POCO materialized has clear data in it.
var things = from t in myDbContext.Things
select t;
where myDbContext.Things is a DbSet<Thing>
Likewise, passing an instance of Thing to Things.Add()
(with clear string data in the Name and/or OtherString values)
and then calling myDbContext.SaveChanges() encrypts the strings before it gets to the data store.
Now, the problem I have is in this query:
var things = from t in myDbContext.Things
where t.Name == "Hairbrush"
select t;
This results in the unencrypted value being compared to the encrypted value in the DB. Obviously I don't want to get all the records from the database, materialize them, and then filter the results based on any supplied Where clause... so what I need to do is: intercept that query and rewrite it by encrypting any strings in the Where clause.
So I've looked at:
writing a query provider, but that doesn't seem like the right solution... (is it?)
writing my own IQueryable wrapper for the DbSet which will capture the expression, run over it using an expression tree visitor and then forward the new expression to the DbSet...
Attempts at both have left me somewhat lost! I prefer the second solution i think since it feels a bit neater, and is probably clearer to other developers in future. But I'm happy to go with either or another better option!!
The main thing I am struggling with is when/how the LINQ expression is applied to the object... I think i've got myself a bit confused as to where the expression executes in the IQueryable object thus I'm not sure which method I need to implement in my wrapper to then grab and manipulate the expression being passed in...
I'm sure I'm missing something fairly obvious here and I'm waiting for that light bulb moment... but its not coming!!
Any help will be very gratefully received!

Thought I'd let you know what my final solution was.
In the end I have gone a wrapper class which implements a Where method, but without going to the extent of implementing IQueryable entirely. LINQ will still execute against the class (at least to the extent that I want/need it to) and will call the Where method with the expression from the LINQ.
I then traverse this ExpressionTree and replace my strings with encrypted values before forwarding the new expressiontree to the internal DbSet. and then returning the result.
Its pretty crude, and has its limitation, but works for our particular circumstance without problem.
Thanks,
Ben

you should use the QueryInterceptor attribute, search here in SO or in google and you find examples on how to use it.
a snippet:
[QueryInterceptor("Orders")]
public Expression<Func<Order, bool>> FilterOrders()
{
return o => o.Customer.Name == /* Current principal name. */;
}
// Insures that the user accessing the customer(s) has the appropriate
// rights as defined in the QueryRules object to access the customer
// resource(s).
[QueryInterceptor ("Customers")]
public Expression<Func<Customer, bool>> FilterCustomers()
{
return c => c.Name == /* Current principal name. */ &&
this.CurrentDataSource.QueryRules.Contains(
rule => rule.Name == c.Name &&
rule.CustomerAllowedToQuery == true
);
}

You can use David Fowler's Query Interceptor:
https://github.com/davidfowl/QueryInterceptor
One example of its use:
IQueryable q = ...;
IQueryable modifed = q.InterceptWith(new MyInterceptor());
And on class MyInterceptor:
protected override Expression VisitBinary(BinaryExpression node) {
if (node.NodeType == ExpressionType.Equal) {
// Change == to !=
return Expression.NotEqual(node.Left, node.Right);
}
return base.VisitBinary(node);
}

Using the LINQ to SQL OnLoaded() method results in an empty sequence for my aggregate root?

I can't find much information on the LINQ to SQL OnLoaded() partial method used with entity types. I'm running into an issue and it must be due to my lack of understanding how the method is supposed to be used or some other oversight on my part.
I have an aggregate root, Account, which has a collection of entities, of type Balance, called BalanceHistory. I have defined a property in the class called CurrentBalance which is not mapped to any column in my source table. Instead, I use OnLoaded to set its value as the most recent entry from the BalanceHistory collection.
partial void OnLoaded()
{
var balance = this.BalanceHistory
.OrderByDescending(b => b.Datestamp)
.First();
this.CurrentBalance = balance.Value;
}
The BalanceHistory collection is loaded with the Account entity, i.e. it is not lazy loaded. Here are the load options for my DataContext class.
public AccountRepository()
{
var connection = ConfigurationManager.ConnectionStrings["DbConnectionString"].ConnectionString;
this._dataContext = new DbDataContext(connection);
var options = new DataLoadOptions();
options.LoadWith<Account>(a => a.BalanceHistory);
this._dataContext.LoadOptions = options;
}
The problem I'm running into is that whenever I try to retrieve Account objects from my DataContext class, I get an InvalidOperationException saying "Sequence contains no elements." However, if I use the debugger and look through the DataContext object, I can see that only the Account objects are missing. All of the Balance objects in the BalanceHistory collection are there. However, if I comment out the OnLoaded method the set of Accounts is retrieved without error (but of course, the CurrentBalance property is not populated).
So I must be doing something wrong. Does anyone have any suggestions? Besides figuring out what I'm doing wrong, I'd also be interested in hearing any ideas for a better implementation.
Thanks!

Presumably CurrentBalance is a property added in a partial class implementation. In that case, I would calculate it the first time that the property is invoked rather than populating it when the parent object is loaded.
private decimal? currentBalance;
public decimal CurrentBalance
{
get
{
if (!this.currentBalance.HasValue)
{
var balance = this.BalanceHistory
.OrderByDescending(b => b.Datestamp)
.FirstOrDefault();
this.currentBalance = balance == null ? 0M : balance.Value;
}
return this.currentBalance.Value;
}
}