Replace a SQL process that uses MERGE with LINQ in C# code - c#

I need to transfer a process from SQL Server to my application code. The T-SQL process uses MERGE, as shown below, to conditionally update or insert.
-- Synchronize the InterestRates table with refreshed/new data from ImportRates_Stg table
MERGE InterestRates AS TARGET
USING ImportRates_Stg AS SOURCE
ON (TARGET.Effective = SOURCE.EffectiveDate)
-- When records are matched on the Effective date, update the records if there is any change to the Rate
WHEN MATCHED AND TARGET.Effective = SOURCE.EffectiveDate
THEN UPDATE SET TARGET.Rate = SOURCE.Rate
-- When no records are matched on the Effective date,
-- insert the incoming records from ImportRates_Stg table to InterestRates table
WHEN NOT MATCHED BY TARGET
THEN INSERT (Effective, Rate) VALUES (SOURCE.EffectiveDate, Rate);
I need to reproduce this functionality in C#, and am thinking that LINQ would likely be the best way to do this, but so far all of my attempts have failed. Here is the code that I have so far. Importing the data to a list from an excel file is all working. It is when I get to the actual logic to replace the SQL MERGE which is not working...
public async Task<IActionResult> OnPostAsync()
{
// Perform an initial check to catch FileUpload class attribute violations.
if (!ModelState.IsValid)
{
return Page();
}
string filePath = RatesBatchImportFilepath + Path.GetFileName(Request.Form.Files["RatesExtract"].FileName);
using (FileStream fileStream = new FileStream(filePath, FileMode.Create))
{
await Request.Form.Files["RatesExtract"].CopyToAsync(fileStream);
}
var newRates = new List<InterestRate>();
using (var wb = new XLWorkbook(filePath, XLEventTracking.Disabled))
{
var ws = wb.Worksheet(1);
DataTable dataTable = ws.RangeUsed().AsTable().AsNativeDataTable();
if (dataTable.Rows.Count > 0)
{
foreach (DataRow dataRow in dataTable.Rows)
{
if (dataRow.ItemArray.All(x => string.IsNullOrEmpty(x?.ToString()))) continue;
newRates.Add(new InterestRate()
{
Effective = Convert.ToDateTime(dataRow["PeriodEndingDate"]),
Rate = Convert.ToDecimal(dataRow["AY01NetPerf"])
});
};
}
}
IQueryable<InterestRate> existingRates = from s in _context.InterestRates
orderby s.Effective descending
select s;
foreach (var oldRate in existingRates)
{
DateTime ourDate = oldRate.Effective;
var thisUpdateQuery =
from thisRate in newRates
where thisRate.Effective == ourDate
select thisRate;
foreach (var rate in thisUpdateQuery)
{
oldRate.Effective = rate.Effective;
newRates.Remove(rate); // this causes an error.
}
}
foreach (var rate in newRates)
{
Rates.Add(rate);
}
_context.SaveChanges();
return RedirectToPage("./Index");
}
Here is the Error: InvalidOperationException: Collection was modified; enumeration operation may not execute.

here you are a merge function with all options like merge in sql:
public static async Task Merge<T>(this List<T> target, List<T> source, Func<T, T, bool> mergeOn, Func<T, T, Task> onMatched = null, Func<List<T>, Task> whenNotMatchedByTarget = null, Func<List<T>, Task> whenNotMatchedBySource = null)
{
var sourceTemp = JsonConvert.DeserializeObject<List<T>>(JsonConvert.SerializeObject(source, new JsonSerializerSettings() { ReferenceLoopHandling = ReferenceLoopHandling.Ignore }));
var notMatchedByTarget = new List<T>();
bool isMatched;
for (int i = 0; i < target?.Count; i++)
{
isMatched = false;
for (int j = 0; j < sourceTemp?.Count; j++)
{
if (mergeOn(target[i], sourceTemp[j]))
{
if (onMatched != null)
await onMatched(target[i], sourceTemp[j]);
sourceTemp.RemoveAt(j);
isMatched = true;
break;
}
}
if (!isMatched)
notMatchedByTarget.Add(target[i]);
}
if (whenNotMatchedByTarget != null)
await whenNotMatchedByTarget(notMatchedByTarget);
if (whenNotMatchedBySource != null)
await whenNotMatchedBySource(sourceTemp);
}
then use it like:-
await targetList.Merge(source: sourceList,
mergeOn: (t, s) => t.Id == s.Id,
onMatched: Update,
whenNotMatchedByTarget: Delete,
whenNotMatchedBySource: Add);
Task Update(sourceType old, sourceType new){};
Task Delete(List<sourceType> listToBeDeleted){};
Task Add(List<sourceType> listToBeAdded){};

Here's an extension method that I think works --
public static List<T> Merge<T>(this List<T> list, Func<T, T, bool> mergeOnFunc, Action<T, T> ifMatchedByTargetAction)
{
List<T> mergedList = new List<T>();
list.ForEach(rec =>
{
if (!mergedList.Any(x => mergeOnFunc(rec, x)))
mergedList.Add(rec); // if the record hasn't been added yet, add it
else // if the record has already been added, merge the changes
ifMatchedByTargetAction(mergedList.Where(x => mergeOnFunc(rec, x)).First(), rec);
});
return mergedList;
}
Here's an example of how you'd use the extension method (assumes you've already batched all of your mergeables into a single list w. concat or something similar -- sorry, it's not an ideal use case, but it's the one I have).
classAttendanceList = classAttendanceList.Merge(
(a, b) => { return a.StudentName == b.StudentName; }, // this is your "ON"
(a, b) => { a.DaysAttended += b.DaysAttended; }); // this is your "WHEN MATCHED"

Related

Project on update/create (set values from another object en masse) in LINQ2DB?

When using LINQ2DB for my application I tried to use entity-DTO mapping using Expression<Func<Entity, DTO>> and vice versa like described here: https://github.com/linq2db/linq2db/issues/1283#issuecomment-413509043
This works great for projecting using a select, but what do I do when I need to update/insert a new record? I've skimmed over Update and Set extension methods but couldn't find anything.
What I am trying to achieve is basically expression-based two-way mapping between an entity class and a DTO, kinda like AutoMapper's projection for EF but manually written per-DTO, in the form of two expressions for two-way conversion.
Sadly I am not an expert in expression trees and LINQ to SQL translation, so would appreciate if anyone suggests something that works like this:
Expression<Func<SomeDTO, SomeEntityTable>> projectExpr =
x => new SomeEntity
{
ID = x.ID,
Name = x.Name,
// ...
}; // this is just so that I can write two mapping expressions per DTO and don't ever repeat them, for stuff like CRUD
// ...
using var db = ConnectionFactory.Instance.GetMainDB();
await db.SomeEntityTable
.Where(e => e.ID == dto.ID)
.Set(dto, projectExpr) // dto is of SomeDTO type here; this will set ONLY the values that are written in the expression
.Set(e => e.LastEditedAt, DateTime.Now()) // able to append some more stuff after
.UpdateAsync();
// similar for insert operation, using the same expression
These extension methods should provide needed mapping:
using var db = ConnectionFactory.Instance.GetMainDB();
await db.SomeEntityTable
.Where(e => e.ID == dto.ID)
.AsUpdatable()
.Set(dto, projectExpr) // new extension method
.Set(e => e.LastEditedAt, DateTime.Now())
.UpdateAsync();
await db.SomeEntityTable
.AsValueInsertable()
.Values(dto, projectExpr) // new extension method
.Value(e => e.LastEditedAt, DateTime.Now())
.InsertAsync();
And implementation:
public static class InsertUpdateExtensions
{
private static MethodInfo _withIUpdatable = Methods.LinqToDB.Update.SetUpdatableExpression;
private static MethodInfo _withIValueInsertable = Methods.LinqToDB.Insert.VI.ValueExpression;
public static IUpdatable<TEntity> Set<TEntity, TDto>(
this IUpdatable<TEntity> updatable,
TDto obj,
Expression<Func<TDto, TEntity>> projection)
{
var body = projection.GetBody(Expression.Constant(obj));
var entityParam = Expression.Parameter(typeof(TEntity), "e");
var pairs = EnumeratePairs(body, entityParam);
foreach (var pair in pairs)
{
updatable = (IUpdatable<TEntity>)_withIUpdatable.MakeGenericMethod(typeof(TEntity), pair.Item1.Type)
.Invoke(null,
new object?[]
{
updatable,
Expression.Lambda(pair.Item1, entityParam),
Expression.Lambda(pair.Item2)
})!;
}
return updatable;
}
public static IValueInsertable<TEntity> Values<TEntity, TDto>(
this IValueInsertable<TEntity> insertable,
TDto obj,
Expression<Func<TDto, TEntity>> projection)
{
var body = projection.GetBody(Expression.Constant(obj));
var entityParam = Expression.Parameter(typeof(TEntity), "e");
var pairs = EnumeratePairs(body, entityParam);
foreach (var pair in pairs)
{
insertable = (IValueInsertable<TEntity>)_withIValueInsertable.MakeGenericMethod(typeof(TEntity), pair.Item1.Type)
.Invoke(null,
new object?[]
{
insertable,
Expression.Lambda(pair.Item1, entityParam),
Expression.Lambda(pair.Item2)
})!;
}
return insertable;
}
private static IEnumerable<Tuple<Expression, Expression>> EnumeratePairs(Expression projection, Expression entityPath)
{
switch (projection.NodeType)
{
case ExpressionType.MemberInit:
{
var mi = (MemberInitExpression)projection;
foreach (var b in mi.Bindings)
{
if (b.BindingType == MemberBindingType.Assignment)
{
var assignment = (MemberAssignment)b;
foreach (var p in EnumeratePairs(Expression.MakeMemberAccess(entityPath, assignment.Member),
assignment.Expression))
{
yield return p;
}
}
}
break;
}
case ExpressionType.New:
{
var ne = (NewExpression)projection;
if (ne.Members != null)
{
for (var index = 0; index < ne.Arguments.Count; index++)
{
var expr = ne.Arguments[index];
var member = ne.Members[index];
foreach (var p in EnumeratePairs(Expression.MakeMemberAccess(entityPath, member), expr))
{
yield return p;
}
}
}
break;
}
case ExpressionType.MemberAccess:
{
yield return Tuple.Create(projection, entityPath);
break;
}
default:
throw new NotImplementedException();
}
}
}

How to pass a list of strings through webapi and get the results without those strings?

My code already gets the table without containing a string. How can I get a list without containing a list of strings? I want to get the result of SELECT * FROM table WHERE column NOT IN ('x' ,'y');
public IEnumerable<keyart1> Get(string keyword)
{
List<keyart1> keylist;
using (dbEntities5 entities = new dbEntities5())
{
keylist = entities.keyart1.Where(e => e.keyword != keyword).ToList();
var result = keylist.Distinct(new ItemEqualityComparer());
return result;
}
}
I think i found the answer if anybody interested
public IEnumerable<keyart1> Get([FromUri] string[] keyword1)
{
List<keyart1> keylist;
List<IEnumerable<keyart1>> ll;
using (dbEntities5 entities = new dbEntities5())
{
ll = new List<IEnumerable<keyart1>>();
foreach (var item in keyword1)
{
keylist = entities.keyart1.Where(e => e.keyword != item).ToList();
var result = keylist.Distinct(new ItemEqualityComparer());
ll.Add(result);
}
var intersection = ll.Aggregate((p, n) => p.Intersect(n).ToList());
return intersection;
}
}

C# Calculate field inside LINQ Query

I need some help to calculate a property inside my Linq query.
I know I need to use "let" somewhere, but I can't figure it out!
So, first I have this method to get my list from Database:
public BindingList<Builders> GetListBuilders()
{
BindingList<Builders> builderList = new BindingList<Builders>();
var ctx = new IWMJEntities();
var query = (from l in ctx.tblBuilders
select new Builders
{
ID = l.BuilderID,
Projeto = l.NomeProjeto,
Status = l.Status,
DataPedido = l.DataPedido,
DataPendente = l.DataPendente,
DataEntregue = l.DataEntregue,
DataAnulado = l.DataAnulado
});
foreach (var list in query)
builderList.Add(list);
return builderList;
}
Then, I have a function to calculate the Days between Dates accordingly to Status:
public int GetDays()
{
int Dias = 0;
foreach (var record in GetListBuilders)
{
if (record.Status == "Recebido")
{
Dias = GetBusinessDays(record.DataPedido, DateTime.Now);
}
else if (record.Status == "Pendente")
{
Dias = GetBusinessDays(record.DataPedido, (DateTime)record.DataPendente);
}
else if (record.Status == "Entregue")
{
Dias = GetBusinessDays(record.DataPedido, (DateTime)record.DataEntregue);
}
else if (record.Status == "Anulado")
{
Dias = GetBusinessDays(record.DataPedido, (DateTime)record.DataAnulado);
}
}
return Dias;
}
I need to call the GetDays in a DataGridView to give the days for each record.
My big problem is, How do I get this? include it in Linq Query? Calling GetDays() (need to pass the ID from each record to GetDays() function)!?
Any help?
Thanks
I think it would be easier to create an extension method:
public static int GetBusinessDays(this Builders builder) // or type of ctx.tblBuilders if not the same
{
if (builder == null) return 0;
switch(builder.status)
{
case "Recebido": return GetBusinessDays(builder.DataPedido, DateTime.Now);
case "Pendente": return GetBusinessDays(builder.DataPedido, (DateTime)builder.DataPendente);
case "Entregue": return GetBusinessDays(builder.DataPedido, (DateTime)builder.DataEntregue);
case "Anulado": GetBusinessDays(builder.DataPedido, (DateTime)builder.DataAnulado);
default: return 0;
}
}
Then, call it like that:
public BindingList<Builders> GetListBuilders()
{
BindingList<Builders> builderList = new BindingList<Builders>();
var ctx = new IWMJEntities();
var query = (from l in ctx.tblBuilders
select new Builders
{
ID = l.BuilderID,
Projeto = l.NomeProjeto,
Status = l.Status,
DataPedido = l.DataPedido,
DataPendente = l.DataPendente,
DataEntregue = l.DataEntregue,
DataAnulado = l.DataAnulado,
Dias = l.GetBusinessDays()
});
foreach (var list in query)
builderList.Add(list);
return builderList;
}
To do better, to convert a object to a new one, you should create a mapper.
Why does it need to be a part of the query? You can't execute C# code on the database. If you want the calculation to be done at the DB you could create a view.
You're query is executed as soon as the IQueryable is enumerated at the foreach loop. Why not just perform the calculation on each item as they are enumerated and set the property when you are adding each item to the list?

Working with multiple resultset in .net core

While retrieving the results using stored procedure how can I retrieve and store multiple result set in view model in .net core
For e.g. from stored procedure I am returning records for below two queries
Select * LMS_Survey
Select * from LMS_SurveyQuestion
Select * from LMS_SurveyQuestionOptionChoice
and below is view model for two tables
public class LMS_SurveyTraineeViewModel
{
public LMS_SurveyDetailsViewModel SurveyDetailsViewModel { get; set; }
public LMS_SurveyQuestionsViewModel SurveyQuestionsViewModel { get; set; }
public LMS_SurveyQuestionOptionChoiceViewModel SurveyQuestionOptionChoiceViewModel { get; set; }
}
This is how I am executing the stored procedure
public List<LMS_SurveyTraineeViewModel> GetTraineeSurvey(int surveyID)
{
try
{
List<LMS_SurveyTraineeViewModel> modelList = new List<LMS_SurveyTraineeViewModel>();
modelList = dbcontext.Set<LMS_SurveyTraineeViewModel>().FromSql("LMSSP_GetTraineeSurvey #surveyID = {0},#LanguageID = {1}", surveyID, AppTenant.SelectedLanguageID).ToList();
return modelList;
}
catch (Exception ex)
{
throw ex;
}
}
How can stored the multiple result set using stored procedure in view model ?
Currently, EF Core doesn't not support this. There is an open issue to address this.
https://github.com/aspnet/EntityFramework/issues/8127
Update 12th Sep 2018: This is still not a priority for EF Core even for release 3.0; so best use Dapper or plain ADO.NET when you have multiple results scenario
Update 25th Jun 2020: still on the backlog for EF Core even for release 5.0; so best use Dapper or plain ADO.NET when you have multiple results scenario
Update 7th Feb 2021: still on the backlog for EF Core
Update 8th Aug 2022: still on the backlog for EF Core, looks like its not a high priority use-case. Recommend to follow alternatives like using straight ADO.NET or Dapr or the below workaround for this
In the interim an alternative solution can be achieved via extension method(s)
public static async Task<IList<IList>> MultiResultSetsFromSql(this DbContext dbContext, ICollection<Type> resultSetMappingTypes, string sql, params object[] parameters)
{
var resultSets = new List<IList>();
var connection = dbContext.Database.GetDbConnection();
var parameterGenerator = dbContext.GetService<IParameterNameGeneratorFactory>()
.Create();
var commandBuilder = dbContext.GetService<IRelationalCommandBuilderFactory>()
.Create();
foreach (var parameter in parameters)
{
var generatedName = parameterGenerator.GenerateNext();
if (parameter is DbParameter dbParameter)
commandBuilder.AddRawParameter(generatedName, dbParameter);
else
commandBuilder.AddParameter(generatedName, generatedName);
}
using var command = connection.CreateCommand();
command.CommandType = CommandType.Text;
command.CommandText = sql;
command.Connection = connection;
for (var i = 0; i < commandBuilder.Parameters.Count; i++)
{
var relationalParameter = commandBuilder.Parameters[i];
relationalParameter.AddDbParameter(command, parameters[i]);
}
var materializerSource = dbContext.GetService<IEntityMaterializerSource>();
if (connection.State == ConnectionState.Closed)
await connection.OpenAsync();
using var reader = await command.ExecuteReaderAsync();
foreach (var pair in resultSetMappingTypes.Select((x, i) => (Index: i, Type: x)))
{
var i = pair.Index;
var resultSetMappingType = pair.Type;
if (i > 0 && !(await reader.NextResultAsync()))
throw new InvalidOperationException(string.Format("No result set at index {0}, unable to map to {1}.", i, resultSetMappingType));
var type = resultSetMappingType;
var entityType = dbContext.GetService<IModel>()
.FindEntityType(type);
if (entityType == null)
throw new InvalidOperationException(string.Format("Unable to find a an entity type (or query type) matching '{0}'", type));
var relationalTypeMappingSource = dbContext.GetService<IRelationalTypeMappingSource>();
var columns = Enumerable.Range(0, reader.FieldCount)
.Select(x => new
{
Index = x,
Name = reader.GetName(x)
})
.ToList();
var relationalValueBufferFactoryFactory = dbContext.GetService<IRelationalValueBufferFactoryFactory>();
int discriminatorIdx = -1;
var discriminatorProperty = entityType.GetDiscriminatorProperty();
var entityTypes = entityType.GetDerivedTypesInclusive();
var instanceTypeMapping = entityTypes.Select(et => new
{
EntityType = et,
Properties = et.GetProperties()
.Select(x =>
{
var column = columns.FirstOrDefault(y => string.Equals(y.Name,
x.GetColumnName() ?? x.Name, StringComparison.OrdinalIgnoreCase)) ?? throw new InvalidOperationException(string.Format("Unable to find a column mapping property '{0}'.", x.Name));
if (x == discriminatorProperty)
discriminatorIdx = column.Index;
return new TypeMaterializationInfo(x.PropertyInfo.PropertyType, x, relationalTypeMappingSource, column.Index);
})
.ToArray()
})
.Select(x => new
{
EntityType = x.EntityType,
Properties = x.Properties,
ValueBufferFactory = relationalValueBufferFactoryFactory.Create(x.Properties)
})
.ToDictionary(e => e.EntityType.GetDiscriminatorValue() ?? e.EntityType, e => e)
;
var resultSetValues = (IList)Activator.CreateInstance(typeof(List<>).MakeGenericType(type));
while (await reader.ReadAsync())
{
var instanceInfo = discriminatorIdx < 0 ? instanceTypeMapping[entityType] : instanceTypeMapping[reader[discriminatorIdx]];
var valueBuffer = instanceInfo.ValueBufferFactory.Create(reader);
var materializationAction = materializerSource.GetMaterializer(instanceInfo.EntityType);
resultSetValues.Add(materializationAction(new MaterializationContext(valueBuffer, dbContext)));
}
resultSets.Add(resultSetValues);
}
return resultSets;
}
And the extension typed methods
public static async Task<(IReadOnlyCollection<T1> FirstResultSet, IReadOnlyCollection<T2> SecondResultSet)> MultiResultSetsFromSql<T1, T2>(this DbContext dbContext, string sql, params object[] parameters)
{
var resultSetMappingTypes = new[]
{
typeof(T1), typeof(T2)
};
var resultSets = await MultiResultSetsFromSql(dbContext, resultSetMappingTypes, sql, parameters);
return ((IReadOnlyCollection<T1>)resultSets[0], (IReadOnlyCollection<T2>)resultSets[1]);
}
public static async Task<(IReadOnlyCollection<T1> FirstResultSet, IReadOnlyCollection<T2> SecondResultSet, IReadOnlyCollection<T3> ThirdResultSet)> MultiResultSetsFromSql<T1, T2, T3>(this DbContext dbContext, string sql, params object[] parameters)
{
var resultSetMappingTypes = new[]
{
typeof(T1), typeof(T2), typeof(T3)
};
var resultSets = await MultiResultSetsFromSql(dbContext, resultSetMappingTypes, sql, parameters);
return ((IReadOnlyCollection<T1>)resultSets[0], (IReadOnlyCollection<T2>)resultSets[1], (IReadOnlyCollection<T3>)resultSets[2]);
}
Currently, EF Core doesn't not support this. see this example for retrieve multiple result sets.
https://github.com/nilendrat/EfCoreMultipleResults/
It works with this tiny change on EF core 5 according to Ricky G answer
change
command.CommandType = CommandType.Text;
to
command.CommandType = CommandType.StoredProcedure;
and as sql parameter value for this extension method type your stored procedure name "dbo.testproc"
example of usage:
var t1 = await _context.MultiResultSetsFromSql(new [] {typeof(proctestprocResult) },"dbo.testproc", sqlParameters);
works for me

Generic DbDataReader to List<T> mapping

I am having a slight issue (more like an annoyance) with my property binding data access classes. The problem is that the mapping fails when there exists no column in the reader for corresponding property in class.
Code
Here is the mapper class:
// Map our datareader object to a strongly typed list
private static IList<T> Map<T>(DbDataReader dr) where T : new()
{
try
{
// initialize our returnable list
List<T> list = new List<T>();
// fire up the lamda mapping
var converter = new Converter<T>();
while (dr.Read())
{
// read in each row, and properly map it to our T object
var obj = converter.CreateItemFromRow(dr);
// add it to our list
list.Add(obj);
}
// reutrn it
return list;
}
catch (Exception ex)
{
return default(List<T>);
}
}
Converter class:
/// <summary>
/// Converter class to convert returned Sql Records to strongly typed classes
/// </summary>
/// <typeparam name="T">Type of the object we'll convert too</typeparam>
internal class Converter<T> where T : new()
{
// Concurrent Dictionay objects
private static ConcurrentDictionary<Type, object> _convertActionMap = new ConcurrentDictionary<Type, object>();
// Delegate action declaration
private Action<IDataReader, T> _convertAction;
// Build our mapping based on the properties in the class/type we've passed in to the class
private static Action<IDataReader, T> GetMapFunc()
{
var exps = new List<Expression>();
var paramExp = Expression.Parameter(typeof(IDataReader), "o7thDR");
var targetExp = Expression.Parameter(typeof(T), "o7thTarget");
var getPropInfo = typeof(IDataRecord).GetProperty("Item", new[] { typeof(string) });
var _props = typeof(T).GetProperties();
foreach (var property in _props)
{
var getPropExp = Expression.MakeIndex(paramExp, getPropInfo, new[] { Expression.Constant(property.Name, typeof(string)) });
var castExp = Expression.TypeAs(getPropExp, property.PropertyType);
var bindExp = Expression.Assign(Expression.Property(targetExp, property), castExp);
exps.Add(bindExp);
}
// return our compiled mapping, this will ensure it is cached to use through our record looping
return Expression.Lambda<Action<IDataReader, T>>(Expression.Block(exps), new[] { paramExp, targetExp }).Compile();
}
internal Converter()
{
// Fire off our mapping functionality
_convertAction = (Action<IDataReader, T>)_convertActionMap.GetOrAdd(typeof(T), (t) => GetMapFunc());
}
internal T CreateItemFromRow(IDataReader dataReader)
{
T result = new T();
_convertAction(dataReader, result);
return result;
}
}
Exception
System.IndexOutOfRangeException {"Mileage"}
Stacktrace
at System.Data.ProviderBase.FieldNameLookup.GetOrdinal(String fieldName)
at System.Data.SqlClient.SqlDataReader.GetOrdinal(String name)
at System.Data.SqlClient.SqlDataReader.get_Item(String name)
at lambda_method(Closure , IDataReader , Typing )
at o7th.Class.Library.Data.Converter`1.CreateItemFromRow(IDataReader dataReader) in d:\Backup Folder\Development\o7th Web Design\o7th.Class.Library.C-Sharp\o7th.Class.Library\Data Access Object\Converter.cs:line 50
at o7th.Class.Library.Data.Wrapper.Map[T](DbDataReader dr) in d:\Backup Folder\Development\o7th Web Design\o7th.Class.Library.C-Sharp\o7th.Class.Library\Data Access Object\Wrapper.cs:line 33
Question
How can I fix it, so that it will not fail when I have an extra property that the reader may not have as column and vice versa? Of course the quick band-aid would be to simply add NULL As Mileage to this query in example, however, this is not a solution to the problem :)
Here's Map<T> using reflection:
// Map our datareader object to a strongly typed list
private static IList<T> Map<T>(DbDataReader dr) where T : new()
{
try
{
// initialize our returnable list
List<T> list = new List<T>();
T item = new T();
PropertyInfo[] properties = (item.GetType()).GetProperties();
while (dr.Read()) {
int fc = dr.FieldCount;
for (int j = 0; j < fc; ++j) {
var pn = properties[j].Name;
var gn = dr.GetName(j);
if (gn == pn) {
properties[j].SetValue(item, dr[j], null);
}
}
list.Add(item);
}
// return it
return list;
}
catch (Exception ex)
{
// Catch an exception if any, an write it out to our logging mechanism, in addition to adding it our returnable message property
_Msg += "Wrapper.Map Exception: " + ex.Message;
ErrorReporting.WriteEm.WriteItem(ex, "o7th.Class.Library.Data.Wrapper.Map", _Msg);
// make sure this method returns a default List
return default(List<T>);
}
}
Note:
This method is 63% slower than using expression trees...
As noted in comments, the problem is that there exists no column in the reader for the specified property. The idea is to loop by the column names of reader first, and check to see if matching property exists. But how do one get the list of column names beforehand?
One idea is to use expression trees itself to build the list of column names from the reader and check it against properties of the class. Something like this
var paramExp = Expression.Parameter(typeof(IDataRecord), "o7thDR");
var loopIncrementVariableExp = Expression.Parameter(typeof(int), "i");
var columnNamesExp = Expression.Parameter(typeof(List<string>), "columnNames");
var columnCountExp = Expression.Property(paramExp, "FieldCount");
var getColumnNameExp = Expression.Call(paramExp, "GetName", Type.EmptyTypes,
Expression.PostIncrementAssign(loopIncrementVariableExp));
var addToListExp = Expression.Call(columnNamesExp, "Add", Type.EmptyTypes,
getColumnNameExp);
var labelExp = Expression.Label(columnNamesExp.Type);
var getColumnNamesExp = Expression.Block(
new[] { loopIncrementVariableExp, columnNamesExp },
Expression.Assign(columnNamesExp, Expression.New(columnNamesExp.Type)),
Expression.Loop(
Expression.IfThenElse(
Expression.LessThan(loopIncrementVariableExp, columnCountExp),
addToListExp,
Expression.Break(labelExp, columnNamesExp)),
labelExp));
would be the equivalent of
List<string> columnNames = new List<string>();
for (int i = 0; i < reader.FieldCount; i++)
{
columnNames.Add(reader.GetName(i));
}
One may continue with the final expression, but there is a catch here making any further effort along this line futile. The above expression tree will be fetching the column names every time the final delegate is called which in your case is for every object creation, which is against the spirit of your requirement.
Another approach is to let the converter class have a pre-defined awareness of the column names for a given type, by means of attributes (see for an example) or by maintaining a static dictionary like (Dictionary<Type, IEnumerable<string>>). Though it gives more flexibility, the flip side is that your query need not always include all the column names of a table, and any reader[notInTheQueryButOnlyInTheTableColumn] would result in exception.
The best approach as I see is to fetch the column names from the reader object, but only once. I would re-write the thing like:
private static List<string> columnNames;
private static Action<IDataReader, T> GetMapFunc()
{
var exps = new List<Expression>();
var paramExp = Expression.Parameter(typeof(IDataRecord), "o7thDR");
var targetExp = Expression.Parameter(typeof(T), "o7thTarget");
var getPropInfo = typeof(IDataRecord).GetProperty("Item", new[] { typeof(string) });
foreach (var columnName in columnNames)
{
var property = typeof(T).GetProperty(columnName);
if (property == null)
continue;
// use 'columnName' instead of 'property.Name' to speed up reader lookups
//in case of certain readers.
var columnNameExp = Expression.Constant(columnName);
var getPropExp = Expression.MakeIndex(
paramExp, getPropInfo, new[] { columnNameExp });
var castExp = Expression.TypeAs(getPropExp, property.PropertyType);
var bindExp = Expression.Assign(
Expression.Property(targetExp, property), castExp);
exps.Add(bindExp);
}
return Expression.Lambda<Action<IDataReader, T>>(
Expression.Block(exps), paramExp, targetExp).Compile();
}
internal T CreateItemFromRow(IDataReader dataReader)
{
if (columnNames == null)
{
columnNames = Enumerable.Range(0, dataReader.FieldCount)
.Select(x => dataReader.GetName(x))
.ToList();
_convertAction = (Action<IDataReader, T>)_convertActionMap.GetOrAdd(
typeof(T), (t) => GetMapFunc());
}
T result = new T();
_convertAction(dataReader, result);
return result;
}
Now that begs the question why not pass the data reader directly to constructor? That would be better.
private IDataReader dataReader;
private Action<IDataReader, T> GetMapFunc()
{
var exps = new List<Expression>();
var paramExp = Expression.Parameter(typeof(IDataRecord), "o7thDR");
var targetExp = Expression.Parameter(typeof(T), "o7thTarget");
var getPropInfo = typeof(IDataRecord).GetProperty("Item", new[] { typeof(string) });
var columnNames = Enumerable.Range(0, dataReader.FieldCount)
.Select(x => dataReader.GetName(x));
foreach (var columnName in columnNames)
{
var property = typeof(T).GetProperty(columnName);
if (property == null)
continue;
// use 'columnName' instead of 'property.Name' to speed up reader lookups
//in case of certain readers.
var columnNameExp = Expression.Constant(columnName);
var getPropExp = Expression.MakeIndex(
paramExp, getPropInfo, new[] { columnNameExp });
var castExp = Expression.TypeAs(getPropExp, property.PropertyType);
var bindExp = Expression.Assign(
Expression.Property(targetExp, property), castExp);
exps.Add(bindExp);
}
return Expression.Lambda<Action<IDataReader, T>>(
Expression.Block(exps), paramExp, targetExp).Compile();
}
internal Converter(IDataReader dataReader)
{
this.dataReader = dataReader;
_convertAction = (Action<IDataReader, T>)_convertActionMap.GetOrAdd(
typeof(T), (t) => GetMapFunc());
}
internal T CreateItemFromRow()
{
T result = new T();
_convertAction(dataReader, result);
return result;
}
Call it like
List<T> list = new List<T>();
var converter = new Converter<T>(dr);
while (dr.Read())
{
var obj = converter.CreateItemFromRow();
list.Add(obj);
}
There are a number of improvements that I can suggest, though.
The generic new T() you're calling in CreateItemFromRow is slower, it uses reflection behind the scenes. You can delegate that part to expression trees as well which should be faster
Right now GetProperty call isn't case insensitive, meaning your column names will have to exactly match the property name. I would make it case insensitive using one of those Bindings.Flag.
I'm not sure at all why you are using a ConcurrentDictionary as a caching mechanism here. A static field in a generic class <T> will be unique for every T. The generic field itself can act as cache. Also why is the Value part of ConcurrentDictionary of type object?
As I said earlier, it's not the best to strongly tie a type and the column names (which you're doing by caching one particular Action delegate per type). Even for the same type your queries can be different selecting different set of columns. It's best to leave it to data reader to decide.
Use Expression.Convert instead of Expression.TypeAs for value type conversion from object.
Also note that reader.GetOrdinal is much faster way to perform data reader lookups.
I would re-write the whole thing like:
readonly Func<IDataReader, T> _converter;
readonly IDataReader dataReader;
private Func<IDataReader, T> GetMapFunc()
{
var exps = new List<Expression>();
var paramExp = Expression.Parameter(typeof(IDataRecord), "o7thDR");
var targetExp = Expression.Variable(typeof(T));
exps.Add(Expression.Assign(targetExp, Expression.New(targetExp.Type)));
//does int based lookup
var indexerInfo = typeof(IDataRecord).GetProperty("Item", new[] { typeof(int) });
var columnNames = Enumerable.Range(0, dataReader.FieldCount)
.Select(i => new { i, name = dataReader.GetName(i) });
foreach (var column in columnNames)
{
var property = targetExp.Type.GetProperty(
column.name,
BindingFlags.Public | BindingFlags.Instance | BindingFlags.IgnoreCase);
if (property == null)
continue;
var columnNameExp = Expression.Constant(column.i);
var propertyExp = Expression.MakeIndex(
paramExp, indexerInfo, new[] { columnNameExp });
var convertExp = Expression.Convert(propertyExp, property.PropertyType);
var bindExp = Expression.Assign(
Expression.Property(targetExp, property), convertExp);
exps.Add(bindExp);
}
exps.Add(targetExp);
return Expression.Lambda<Func<IDataReader, T>>(
Expression.Block(new[] { targetExp }, exps), paramExp).Compile();
}
internal Converter(IDataReader dataReader)
{
this.dataReader = dataReader;
_converter = GetMapFunc();
}
internal T CreateItemFromRow()
{
return _converter(dataReader);
}

Categories

Resources