Develop Reference

Translate/Separate IQueryable expressions?

consider the following scenario:
public class DBEntry {
public string Id;
}
public class ComputedEntry {
public string Id;
public int ComputedIndex;
}
IQueryable<DBEntry> databaseQueryable; // Somewhere hidden behind the API
IQueryable<ComputedEntry> entryQueryable; // Usable with the API
Let's assume each DBEntry has a unique Id and not much else. A ComputedEntry has a 1:n relationship with DBEntry, meaning that a DBEntrycan be expanded into more than a single ComputedEntryduring execution of the query.
Now, I am trying to query entryQueryable to get a range of computed indices, e.g:
entryQueryable.Where(dto => dto.ComputedIndex < 10 && dto.Id == "some-id");
What I'm looking for is a way of separating the given query expression to only push down the relevant parts of the query to the databaseQueryable. In the example above something like this should happen (probably in the implementation of IQueryableProvider.Execute when using the entryQueryable):
var results = databaseQueryable.Where(e => e.Id == "some-id").ToList();
int i = 0;
return results.Select(e => new ComputedEntry(e.Id, i++));
So basically I'd like the query to be separated and the relevant/compatible parts should be pushed down to the databaseQueryable.
The obvious question would be: How should I approach this? I tried to figure out a way of separating the expression with an ExpressionVisitor, but haven't been very successful here and it seems like this is a rather complex task.
Any ideas? Maybe there is an already existing method of optimizing/translating the query I am not aware of? I have looked through the documentation but couldn't find anything useful here.
Many thanks for your suggestions!

Getting all dates between two dates using datepickers and Entity Framework 6

I have two datetime pickers on my form. I want a function that will return all datetimes from a specific table (which are values of a specific column) between those two dates.
My method looks like this:
public DateTime[] GetAllArchiveDates(string username = null)
{
var result = new DateTime[0];
if (username != null)
{
result = this._context.archive.OrderBy(s => s.IssuingDate).Where(s => s.insertedBy == username).Select(s => s.issuing_date).Distinct().ToArray();
}
else
{
result = this._context.archive.OrderBy(s => s.IssuingDate).Select(s => s.issuing_date).Distinct().ToArray();
}
return result;
}
But I am getting this error:
System.NotSupportedException: 'The specified type member 'IssuingDate' is not supported in LINQ to Entities. Only initializers, entity members, and entity navigation properties are supported.'
How to do this?

The cause of your error message
You should be aware about the differences between IEnumerable and IQueryable.
An object of a class that implements IEnumerable holds everything to enumerate over the sequence of items it represents. You can ask for the first item of the sequence, and once you've got one, you can ask for the next item, until there are no more items.
On the other hand, an object of a class that implements IQueryable holds everything to ask another process to provide data to create an IEnumerable sequence. To do this, it holds an Expression and a Provider.
The Expression is a generic representation of what kind of IEnumerable must be created once you start enumerating the IQueryable.
The Provider knows who must execute the query, and it knows how to translate the Expression into a format that the executor understands, for instance SQL.
There are two kinds of LINQ statements. Those that use deferred execution, and those that don't. The deferred functions can be recognized, because they return IQueryable<TResult> (or IEnumerable). Examples are Where, Select, GroupBy, etc.
The non-deferred functions return a TResult: ToList, ToDictionary, FirstOrDefault, Max.
As long as you concatenate deferred LINQ functions, the query is not executed, only the Expression is changed. Once you start enumerating, either explicitly using GetEnumerator and MoveNext, or implicitly using foreach, ToList, Max, etc, the Expression is sent to the Provider who will translate it to SQL and execute the query. The result is represented as an IEnumerable, on which the GetEnumerator is performed.
What has this to do with my question?
Because the Expression must be translated into SQL, it can't hold anything that you invented. After all, SQL doesn't know your functions. In fact, there are a lot of standard functions that can't be used in an IQueryable. See Supported and unsupported LINQ functions
Alas you forgot to give us the archive class definition, but I think that it is not a POCO: It contains functions and properties that do more than just get / set. I think that IssuingDate is not just get / set.
For IQueryables you should keep your classes simple: use only {get; set;} during your query, nothing more. Other functions can be called after you've materialized your IQueryable into something IEnumerable which is to be executed within your local process
Back to your question
So you have a database with a table Archive with at least columns IssuingDate and InsertedBy. It seems that InsertedBy is just a string. It could be a foreign key to a table with users. This won't influence the answer very much.
Following the entity framework code first conventions this leads to the following classes
class Archive
{
public int Id {get; set;}
public DateTime IssuingDate {get; set;}
public string InsertedBy {get; set;}
...
}
public class MyDbContext : DbContext
{
public DbSet<Archive> Archives {get; set;}
}
By the way, is there a proper reason you deviate so often from Microsoft standards about naming identifiers, especially pluralization and camel casing?
Anyway, your requirement
I have two datetime pickers on my form. I want a function that will return all datetimes from a specific table (which are values of a specific column) between those two dates.
Your code seems to do a lot more, but let's first write an extension function that meets your requirement. I'll write it as an extension method of your archive class. This will keep your archive class simple (only {get; set;}), yet it adds functionality to the class. Writing it as an extension function also enables you to use these functions as if they were any other LINQ function. See Extension methods demystified
public static IQueryable<Archive> BetweenDates(this IQueryable<Archive> archives,
DateTime startDate,
DateTime endDate)
{
return archives.Where(archive => startDate <= archive.IssuingDate
&& archive.IssuingDate <= endDate);
}
If I look at your code, you don't do anything of selecting archives between dates. You do something with a userName, ordering, select distinct... It is a bit strange that you first Order all your million archives, and then decide to keep only the ten archives that belong to userName, and if you have several same issuing dates you decide to remove the duplicates. Wouldn't it be more efficient to first limit the number of issuing dates before you start ordering them?
public static IQueryable<archive> ToIssuingDatesOfUser(this IQueryable<archive> archives,
string userName)
{
// first limit the number of archives, depdning on userName,
// then select the IssuingDate, remove duplicates, and finally Order
var archivesOfUser = (userName == null) ? archives :
archives.Where(archive => archive.InsertedBy == userName);
return archivesOfUser.Select(archive => archive.IssuingDate)
.Distinct()
.OrderBy(issuingDate => issuingDate);
}
Note: until now, I only created IQueryables. So only the Expression is changed, which is fairly efficient. The database is not communicated yet.
Example of usage:
Requirement: given a userName, a startDate and an endDate, give me the unique issuingDates of all archives that are issued by this user, in ascending order
public ICollection<string> GetIssuingDatesOfUserBetweenDates(string userName,
DateTime startDate,
DateTime endDate)
{
using (var dbContext = new MyDbContext(...))
{
return dbContext.Archives
.BetweenDates(startDate, endDate)
.ToIssuingDatesOfUser(userName)
.ToList();
}
}

Dapper multilevel nesting

I am trying to use Dapper in my project to speed up data loading (currently using EF6)
Here is my SQL
String SQL = #"select vwArtikli_Grid_V2.ArtikalID
,vwArtikli_Grid_V2.ArtikalNaziv
,Artikli_TagLista.ArtikalTagListaID
,Artikli_TagLista.ArtikalTagID
,Artikli_Stanje.ArtikalStanjeID
,Artikli_Stanje.ObjekatID
,Artikli_Stanje.Stanje
,Artikli_Tagovi.GrupaID
,Artikli_Tagovi.ArtikalTagGrupaID
,Artikli_Tagovi.ArtikalTagNaziv
,Artikli_Tagovi.ArtikalTagPrint
,Artikli_Tagovi.ArtikalTagSlika
,Artikli_Tagovi.ArtikalTagID
,vwArtikli_Grid_V2.ArtikalID
from Artikli_Tagovi
inner join Artikli_TagLista on Artikli_Tagovi.ArtikalTagID = Artikli_TagLista.ArtikalTagID
right outer join vwArtikli_Grid_V2 on Artikli_TagLista.ArtikalID = vwArtikli_Grid_V2.ArtikalID
left outer join Artikli_Stanje on vwArtikli_Grid_V2.ArtikalID = Artikli_Stanje.ArtikalID;
I am using my Entity Framework Entities as POCOs and they are
VwArtikliGridV2, Artikli_TagLista, Artikli_Tagovi, Artikli_Stanje
VwArtikliGridV2 has two properties
public virtual ICollection<Artikli_TagLista> Artikli_TagLista { get; set; }
public virtual ICollection<Artikli_Stanje> Artikli_Stanje { get; set; }
and Artikli_TagLista has
public virtual Artikli_Tagovi Artikli_Tagovi { get; set; }
What is the easiest way to execute the query and map my data to the POCOs or Entities ?
I tried
Dapper.Mapper
var Artikli = cn.Query<VwArtikliGridV2, Artikli_TagLista, Artikli_Stanje, Artikli_Tagovi> (SQL);
but it didnot work
I also tried Slapper.AutoMapper
List<dynamic> ArtikliUM = cn.Query<dynamic>(SQL).ToList();
Slapper.AutoMapper.Configuration.AddIdentifiers(typeof(Artikli_Tagovi), new List<string> { "ArtikalTagID" });
Slapper.AutoMapper.Configuration.AddIdentifiers(typeof(Artikli_TagLista), new List<string> { "ArtikalTagListaID" });
Slapper.AutoMapper.Configuration.AddIdentifiers(typeof(Artikli_Stanje), new List<string> { "ArtikalStanjeID" });
Artikli = (Slapper.AutoMapper.MapDynamic<VwArtikliGridV2>(ArtikliUM) as IEnumerable<VwArtikliGridV2>).ToList();
But it also did not work.
I can map VwArtikliGridV2 but i cant map any of the nested objects. They are always null.
What can I try ?

Dapper maps things flat. At first this feels like a big pain because EF nests things so easily. But once you get over the initial pain you realize how simple, predictable, and performance-oriented it is. Maybe an extra 15 min. of query-crafting for untouchable performance.
I answered a similar question here return a list of data via stored proc to dapper.
You should be able to return multiple datasets (one for the parent item, one for the underlying tags, etc.) and combine them in your app tier.

As BlackjacketMack said in his answer, I think multiple data sets are the way to go.
You might want to look in the Dapper documentation at the features Multi Mapping to split single rows into multiple objects, and QueryMultiple to read multiple result sets from a single query.
Obviously, it would mean modifying your query to return multiple result sets, but could achieve what you are looking for.
Examples can be found here: https://github.com/StackExchange/Dapper/blob/master/Dapper.Tests/MultiMapTests.cs
https://github.com/StackExchange/Dapper/blob/master/Dapper.Tests/QueryMultipleTests.cs
In particular, you might want to take a look at method public void TestMultiMapThreeTypesWithGridReader() in MultiMapTests.cs. I found that very useful to help me understand a similar problem of parent with child collections, where the child contains different object types returned in a single row.

How can I implement Query Interception in a LINQ to Entities query? (c#)

I'm trying to implement encrypted columns in EF4 and using the CTP5 features to allow simple use of POCO's to query the database. Sorry that this is a lot of words, but I hope the below gives enough to explain the need and the problem!
So, bit of background, and my progress so far:
The intention is that if you query the tables without using our DAL then the data is rubbish, but I don't want the developers to worry about if/when/how the data is encrypted.
For simplicity, at this stage I'm working on the assumption any string column will be encrypted.
Now, I have successfully implemented this for returning the data using the Objectmaterialized event, and for data commits using the SavingChanges event.
So given the following class:
public class Thing
{
public int ID { get; set; }
[Required]
public string Name { get; set; }
public DateTime Date { get; set; }
public string OtherString { get; set; }
}
The below query returns all the required values and the POCO materialized has clear data in it.
var things = from t in myDbContext.Things
select t;
where myDbContext.Things is a DbSet<Thing>
Likewise, passing an instance of Thing to Things.Add()
(with clear string data in the Name and/or OtherString values)
and then calling myDbContext.SaveChanges() encrypts the strings before it gets to the data store.
Now, the problem I have is in this query:
var things = from t in myDbContext.Things
where t.Name == "Hairbrush"
select t;
This results in the unencrypted value being compared to the encrypted value in the DB. Obviously I don't want to get all the records from the database, materialize them, and then filter the results based on any supplied Where clause... so what I need to do is: intercept that query and rewrite it by encrypting any strings in the Where clause.
So I've looked at:
writing a query provider, but that doesn't seem like the right solution... (is it?)
writing my own IQueryable wrapper for the DbSet which will capture the expression, run over it using an expression tree visitor and then forward the new expression to the DbSet...
Attempts at both have left me somewhat lost! I prefer the second solution i think since it feels a bit neater, and is probably clearer to other developers in future. But I'm happy to go with either or another better option!!
The main thing I am struggling with is when/how the LINQ expression is applied to the object... I think i've got myself a bit confused as to where the expression executes in the IQueryable object thus I'm not sure which method I need to implement in my wrapper to then grab and manipulate the expression being passed in...
I'm sure I'm missing something fairly obvious here and I'm waiting for that light bulb moment... but its not coming!!
Any help will be very gratefully received!

Thought I'd let you know what my final solution was.
In the end I have gone a wrapper class which implements a Where method, but without going to the extent of implementing IQueryable entirely. LINQ will still execute against the class (at least to the extent that I want/need it to) and will call the Where method with the expression from the LINQ.
I then traverse this ExpressionTree and replace my strings with encrypted values before forwarding the new expressiontree to the internal DbSet. and then returning the result.
Its pretty crude, and has its limitation, but works for our particular circumstance without problem.
Thanks,
Ben

you should use the QueryInterceptor attribute, search here in SO or in google and you find examples on how to use it.
a snippet:
[QueryInterceptor("Orders")]
public Expression<Func<Order, bool>> FilterOrders()
{
return o => o.Customer.Name == /* Current principal name. */;
}
// Insures that the user accessing the customer(s) has the appropriate
// rights as defined in the QueryRules object to access the customer
// resource(s).
[QueryInterceptor ("Customers")]
public Expression<Func<Customer, bool>> FilterCustomers()
{
return c => c.Name == /* Current principal name. */ &&
this.CurrentDataSource.QueryRules.Contains(
rule => rule.Name == c.Name &&
rule.CustomerAllowedToQuery == true
);
}

You can use David Fowler's Query Interceptor:
https://github.com/davidfowl/QueryInterceptor
One example of its use:
IQueryable q = ...;
IQueryable modifed = q.InterceptWith(new MyInterceptor());
And on class MyInterceptor:
protected override Expression VisitBinary(BinaryExpression node) {
if (node.NodeType == ExpressionType.Equal) {
// Change == to !=
return Expression.NotEqual(node.Left, node.Right);
}
return base.VisitBinary(node);
}

"Member access 'DataType MemberName' of 'Namespace.ClassName' not legal on type 'System.Collections.Generic.IEnumerable`1[Namespace.ClassName]."

I would love a solution to my current problem, but I would love EVEN MORE if I can understand what that error actually means.
I have LINQ to SQL classes for two tables in my DB: Providers and Assignments. I added the following member to the Provider class:
public IEnumerable<Assignment> Assignments
{
get
{
return (new linqDataContext())
.Assignments
.Where(a => a.ProviderID == this.ProviderID);
}
}
Then, I bind a GridView using a query that pulls from the parent Provider and uses the child member Assignments, like this:
protected void PopulateProviders()
{
linqDataContext context = new linqDataContext();
var list = from p in context.Providers
where (p.Assignments.Count(a => a.Category == ddlCategory.SelectedValue) > 0)
select p;
lvProviders.DataSource = list;
lvProviders.DataBind();
}
At .DataBind(), when it actually runs the query, it throws the following error:
Member access 'System.String Category' of 'Namespace.Assignment' not legal on type 'System.Collections.Generic.IEnumerable`1[Namespace.Assignment].
I've tried checking for nulls (a => a != null && a.Category ...) but that hasn't worked. I'm not sure what to try next. I think that, if I knew what the error is trying to tell me, I could find the solution. As it stands, I don't know why member access would be illegal.

That Assignments property is all wrong. First of all, property getters should not have side-effects, and more importantly, entity classes should never have reverse dependencies on the DataContext. Linq to SQL has no way to decipher this query; it's relying on a property that does all sorts of crazy stuff that Linq to SQL can't hope to understand.
Get rid of that Assignments property now. Instead of doing that, you need to write this query as a join:
int category = (int)ddlCategory.SelectedValue;
var providers =
from p in context.Providers
join a in context.Assignments
on p.ProviderID equals a.ProviderID
into g
where g.Count(ga => ga.Category == category) > 0
select p;
That should do what you're trying to do if I understood the intent of your code correctly.
One last side note: You never dispose properly of the DataContext in any of your methods. You should wrap it like so:
using (var context = new linqDataContext())
{
// Get the data here
}

I think somewhere it doesn't know that type that is in the IEnumerable. You are trying to call a method that is not part of the IEnumerable inteface.
Why don't you just move the query from the property out to the PopulateProviders() method?

Remove your custom-defined Assignments property. In the your Linq-To-SQL dbml file, create an association between Providers and Assignments, with Providers as the parent property, and ProviderID as the Participating Property for both entities. LINQ will generate a property "IEnumerable Assignments" based on matches between ProviderID using a consistent DataContext.