Develop Reference

Dynamic linq doesn't seem to be working inside a GroupBy clause

I have a problem where the we have a lot of data grouped into what are called stages. The front end wants to display this data, grouped in stages, before rendering it. The number of stages is variable. So originally they were making an api per stage to bunching up the data before binding to a grid control.
When the number of stages become quite high, it becomes too slow. The browser will queue api requests and the overall response time would be very long. Better - we decided - would be to have a single api call that can deal with grouping AND paging. That is, you could say 'take=30' and it would give you 30 records of EACH stage, for however many stages there are.
So the query on the back end looks like what I have below. The problems is that the sorting of the data, before the skip and take is on a dynamic field, and this is where it falls over. Dynamic Linq doesn't seem to work (although it works perfectly fine for normal queries outside the GroupBy call), and I haven't been able to get any of the huge number of extension methods out there working either. They are all some variation of the errors you would find below: (Keep in mind the problem is getting it working with linq to sql)
TestEntities context = new TestEntities();
List<TestTable1> result;
// This works (Hard coding at design time)
result = context.TestTable1.GroupBy(x => x.StageId)
.SelectMany(x => x.OrderBy(y => y.StageId).Skip(1).Take(1)).ToList();
// This works (using a hard coded function at design time)
Func<TestTable1, int> orderByExpression = x => x.StageId;
result = context.TestTable1.GroupBy(x => x.StageId)
.SelectMany(x => x.OrderBy(orderByExpression).Skip(1).Take(1)).ToList();
// This doesn't work. Dynamic linq
result = context.TestTable1.GroupBy(x => x.StageId)
.SelectMany(x => x.AsQueryable().OrderBy("Name").Skip(1).Take(1)).ToList();
// This doesn't work. Can't convert an object to a primitive type
Func<TestTable1, object> orderByObjectExpression = x => x.StageId;
result = context.TestTable1.GroupBy(x => x.StageId)
.SelectMany(x => x.AsQueryable().OrderBy(orderByObjectExpression).Skip(1).Take(1)).ToList();
// This doesn't work. Same as above
Func<TestTable1, dynamic> orderByDynamicExpression = x => x.StageId;
result = context.TestTable1.GroupBy(x => x.StageId)
.SelectMany(x => x.AsQueryable().OrderBy(orderByObjectExpression).Skip(1).Take(1)).ToList();
Console.WriteLine(JsonConvert.SerializeObject(result));
Console.ReadKey();

EF Core query from Include but no need to return in result

Okay, so this is probably from not knowing how to use EF core correctly as this is my 2nd day using but I appear to have to run .Include() in order to query against the "inner join" this method creates.
I've got this bit working and filtering down my results, but I don't want to return the include in the model as it's making a 256kb JSON response turn into a 2.8Mb JSON response.
public IEnumerable<Property> GetFeaturedProperties()
{
var properties = _db.Properties
.Include(p => p.GeoInfo)
.Include(p => p.FeaturedUntil)
.ToArray();
var featuredProperties = new List<Property>();
var DateTimeNow = DateTime.Now;
var midnightToday = DateTimeNow.AddHours(-DateTimeNow.Hour)
.AddMinutes(-DateTimeNow.Minute)
.AddSeconds(-DateTimeNow.Second-1);
for (var i = 0; i < properties.Count(); i++)
{
var property = properties[i];
if(property.FeaturedUntil.Any(p => p.FeaturedDate >= midnightToday))
featuredProperties.Add(property);
}
return featuredProperties;
}
So the offending line is .Include(p => p.FeaturedUntil). As this is an Array of dates that can be up anything from 10-1000 rows per joined row. It includes ALL data, even historical so this is really racking up data.
Can I run my query and then run something to .RemoveInclude(p => p.FeaturedUntil)?

You don't need to load the navigation properties in order to apply filtering. When you access a navigation property inside LINQ to Entities query, it's translated to the corresponding SQL construct, including JOINs. No real object/collections are involved. The whole query (with some exceptions) executes at server (database) side.
In your case, the following simple query will do the job:
public IEnumerable<Property> GetFeaturedProperties()
{
var DateTimeNow = DateTime.Now;
var midnightToday = DateTimeNow.AddHours(-DateTimeNow.Hour)
.AddMinutes(-DateTimeNow.Minute)
.AddSeconds(-DateTimeNow.Second-1);
return _db.Properties
.Include(p => p.GeoInfo) // assuming you want to return this data
.Where(p => p.FeaturedUntil.Any(p => p.FeaturedDate >= midnightToday))
.ToList();
}
For more info, see How Queries Work documentation topic.

What is the fastest way to sort an EF-to-Linq query?

Using Entity Framework, in theory which is faster:
// (1) sort then select/project
// in db, for entire table
var results = someQuery
.OrderBy(q => q.FieldA)
.Select(q => new { q.FieldA, q.FieldB })
.ToDictionary(q => q.FieldA, q => q.FieldB);
or
// (2) select/project then sort
// in db, on a smaller data set
var results = someQuery
.Select(q => new { q.FieldA, q.FieldB })
.OrderBy(q => q.FieldA)
.ToDictionary(q => q.FieldA, q => q.FieldB);
or
// (3) select/project then materialize then sort
// in object space
var results = someQuery
.Select(q => new { q.FieldA, q.FieldB })
.ToDictionary(q => q.FieldA, q => q.FieldB)
.OrderBy(q => q.FieldA); // -> this won't compile, but you get the question
I'm no SQL expert, but it intuitively seems that 2 is faster than 1... is that correct? And how does that compare to 3, because in my experience with EF almost everything is faster when done on the db.
PS I have no perf tools in my environment, and not sure how to test this, hence the question.

Your query is compiling and being executed at the moment you call ToDictionary, so both 1 and 2 should be the same and produce the same query: you get a SELECT FieldA, FieldB FROM table ORDER BY FieldA in both cases.
Third is different: you first execute the SQL query (without the ORDER BY clause), then your sort the returned set in-memory (data is not sorted by the DB provider, but by the client). This might be faster or slower depending on the amount of data, the server's and client's hardware, and how is your database designed (indexes, etc.), the network infrastructure, and so on.
There's no way to tell which one will be faster with the information you provided
PS: this makes no sense as a Dictionary doesn't really care about order (I don't think 3 would compile since Dictionary<>, if I'm not mistaken, doesn't have OrderBy), but change ToDictionary to ToList and there's your performance answer

IPagedList.MVC, set the total size?

I am building an application using asp.net MVC 5 and have a grid working with IPagedList.MVC version 4.5.0.0, AutoMapper and Entity Framework.
In the project I have a BusinessLayer which is what my Action talks to, as I don't want the Action method to talk to Entity Framework directly. So my BLL has the following method:
public IPagedList<ActiveContractViewModel> GetAllContracts(string regNumFilter, int page)
{
var lstcontractViewModel = new List<ActiveContractViewModel>();
using (ActiveContractRepository activeContractRepos = new ActiveContractRepository(new UnitOfWork()))
{
var activeContractList = activeContractRepos.All.OrderByDescending(x => x.Id).Include(c => c.Contractor);
if (regNumFilter.Trim().Length > 0)
{
activeContractList = activeContractRepos.All.Where(x => x.RegistrationNumber.Contains(regNumFilter)).OrderByDescending(x => x.Id).Include(c => c.Contractor);
}
foreach (var activeContract in activeContractList)
{
Mapper.CreateMap<DomainClasses.ActiveContract, ActiveContractViewModel>().ForMember(dest => dest.ContractorModel, opts => opts.MapFrom(src => new ContractorViewModel
{
Id = src.Contractor.Id,
Name = src.Contractor.Name,
ContactPerson = src.Contractor.ContactPerson,
Phone = src.Contractor.Phone,
Fax = src.Contractor.Fax,
Address = src.Contractor.Address,
VendorNumber = src.Contractor.VendorNumber,
FederalTaxId = src.Contractor.FederalTaxId
}
));
Mapper.AssertConfigurationIsValid();
lstcontractViewModel.Add(Mapper.Map<ActiveContractViewModel>(activeContract));
}
}
return lstcontractViewModel.ToPagedList(page, 20);
}
I'm mapping my ActiveContract class (from Entity Framework) to a model (ActiveContractVieWModel) it works fine, data is returned and paging works. But I noticed while debugging that the foreach loop would also go through all records, if I have 2500 records it loops through all building a large list, which is then use on the ToPageList method.
Is there a better way to get around this, so i can build my model and fill it with just the 20 records I need and have the IPagedList know the total size?

I ended up looking into IPageList.MVC some more and saw the author had posted about this:
https://github.com/troygoode/pagedlist#example-2-manual-paging
"In some cases you do not have access something capable of creating an IQueryable, such as when using .Net's built-in MembershipProvider's GetAllUsers method. This method offers paging, but not via IQueryable. Luckily PagedList still has your back (note the use of StaticPagedList):"
I switched to using the StaticPagedList and it works better now, just grabbing the number of records I want and paging works as well.

Thats because you are retrieving all items. In your LINQ queries, you are only filtering by regNumFilter
var activeContractList = activeContractRepos
.All
.OrderByDescending(x => x.Id)
.Include(c => c.Contractor);
activeContractList = activeContractRepos
.All
.Where(x => x.RegistrationNumber.Contains(regNumFilter))
.OrderByDescending(x => x.Id)
.Include(c => c.Contractor);
To retrieve a specific number of rows(In you case 20 items per page), use Skip() and Take() before you iterate the results.
Sample code:
var activeContractList = activeContractList
.Skip(20 * page)
.Take(20);
foreach (var activeContract in activeContractList)
{
....
}

I m not using automapper but valueinjecter.
What i do is "mapping" IList<TModel> to IList<TViewModel> then you wont have the code. it will be more clean.
PagedList is not aware of TEntity, it just does paging. Dont forget paging is lazy.
With Valueinjecter I have following code:
https://github.com/fatagun/NetCollab/blob/master/NetCollab.Web/Mappers/Mapper.cs

I think what you need is AutoMapper Queryable-Extensions.
When ever you use the IEnumerable<T> interface the underline Entity query is materialized.
So the query will be run against your whole table well before ToPagedList "takes" and "skips" some pages. Instead you will have to go with the IQueryable<T> variations.
When using an ORM such as NHibernate or Entity Framework with AutoMapper's standard Mapper.Map functions, you may notice that the ORM will query all the fields of all the objects within a graph when AutoMapper is attempting to map the results to a destination type

Linq to sql expression tree execution zone issue

I have got a bit of an issue and was wondering if there is a way to have my cake and eat it.
Currently I have a Repository and Query style pattern for how I am using Linq2Sql, however I have got one issue and I cannot see a nice way to solve it. Here is an example of the problem:
var someDataMapper = new SomeDataMapper();
var someDataQuery = new GetSomeDataQuery();
var results = SomeRepository.HybridQuery(someDataQuery)
.Where(x => x.SomeColumn == 1 || x.SomeColumn == 2)
.OrderByDescending(x => x.SomeOtherColumn)
.Select(x => someDataMapper.Map(x));
return results.Where(x => x.SomeMappedColumn == "SomeType");
The main bits to pay attention to here are Mapper, Query, Repository and then the final where clause. I am doing this as part of a larger refactor, and we found that there were ALOT of similar queries which were getting slightly different result sets back but then mapping them the same way to a domain specific model. So take for example getting back a tbl_car and then mapping it to a Car object. So a mapper basically takes one type and spits out another, so exactly the same as what would normally happen in the select:
// Non mapped version
select(x => new Car
{
Id = x.Id,
Name = x.Name,
Owner = x.FirstName + x.Surname
});
// Mapped version
select(x => carMapper.Map(x));
So the car mapper is more re-usable on all areas which do similar queries returning same end results but doing different bits along the way. However I keep getting the error saying that Map is not able to be converted to SQL, which is fine as I dont want it to be, however I understand that as it is in an expression tree it would try to convert it.
{"Method 'SomeData Map(SomeTable)' has no supported translation to SQL."}
Finally the object that is returned and mapped is passed further up the stack for other objects to use, which make use of Linq to SQL's composition abilities to add additional criteria to the query then finally ToList() or itterate on the data returned, however they filter based on the mapped model, not the original table model, which I believe is perfectly fine as answered in a previous question:
Linq2Sql point of retrieving data
So to sum it up, can I use my mapping pattern as shown without it trying to convert that single part to SQL?

Yes, you can. Put AsEnumerable() before the last Select:
var results = SomeRepository.HybridQuery(someDataQuery)
.Where(x => x.SomeColumn == 1 || x.SomeColumn == 2)
.OrderByDescending(x => x.SomeOtherColumn)
.AsEnumerable()
.Select(x => someDataMapper.Map(x));
Please note, however, that the second Where - the one that operates on SomeMappedColumn - will now be executed in memory and not by the database. If this last where clause significantly reduces the result set this could be a problem.
An alternate approach would be to create a method that returns the expression tree of that mapping. Something like the following should work, as long as everything happening in the mapping is convertible to SQL.
Expression<Func<EntityType, Car>> GetCarMappingExpression()
{
return new Expression<Func<EntityType, Car>>(x => new Car
{
Id = x.Id,
Name = x.Name,
Owner = x.FirstName + x.Surname
});
}
Usage would be like this:
var results = SomeRepository.HybridQuery(someDataQuery)
.Where(x => x.SomeColumn == 1 || x.SomeColumn == 2)
.OrderByDescending(x => x.SomeOtherColumn)
.Select(GetCarMappingExpression());