Develop Reference

Linq Queryable from OrganizationServiceProxy throws exception when comparing two fields

I've tried to reduce this example to remove the OrganizationServiceProxy/XrmServiceContext, but now I believe the issue is originating there. I'm pulling data from a Dynamics 365 instance using the code generated by CrmSvcUtil.exe from Microsoft. Here is the relevant snippet:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
CrmServiceClient client = new CrmServiceClient(userName, CrmServiceClient.MakeSecureString(password), string.Empty, orgName, isOffice365: true);
using (OrganizationServiceProxy service = client.OrganizationServiceProxy)
{
XrmServiceContext crm = new XrmServiceContext(service);
var q = crm.CreateQuery<Account>();
List<Account> accounts = q.Where(x => x.AccountNumber != x.dynamics_integrationkey).ToList();
}
I get the following exception:
variable 'x' of type '{company-specific}.Account' referenced from scope '', but it is not defined
This is happening on the last line when the Linq is actually executed. It's only when it's created through the Microsoft.Xrm.Sdk.Client.OrganizationServiceContext that I get the error.
Things I've checked:
I get the error when comparing any two fields on the same object (not just Account), even when both are the same exact data type.
If I replace crm.CreateQuery... with a hand-populated List...AsQueryable() the Linq works fine
Most frustratingly, if I work through an intermediary list with List<Account> a = q.ToList(); to rasterize the result set first then this all works fine. It takes much longer to run because I've forfeited any lazy loading, but it works without error.
I've looked at other answers related to the referenced from scope '' but not defined Linq error, but they are all about malformed Linq. I know this Linq is well formed because it works fine if I change the underlying Queryable.
Clearly something is different between a List...ToQueryable() and the Microsoft.Xrm.Sdk.Linq.Query<> created by the XrmServiceContext (which inherits CreateQuery from Microsoft.Xrm.Sdk.Client.OrganizationServiceContext), but I don't know what it is.

The query you are trying to build (compare a field value with another field value on the same entity) is not possible in Dynamics.
The LINQ Provider still uses QueryExpression under the hood, meaning that your LINQ condition will be converted in a ConditionExpression, and that comparison is not possible.
Similar questions:
https://social.microsoft.com/Forums/en-US/d1026ed7-56fd-4f54-b382-bac2fc5e46a7/linq-and-queryexpression-compare-entity-fields-in-query?forum=crmdevelopment
Dynamics QueryExpression - Find entity records where fieldA equals fieldB

This morning I realized that my answer contained the flaw of comparing two fields against each other in a LINQ query. My bad. Thank you Guido for calling that out as impossible.
I generate my proxy classes with a 3rd party tool, so I'm not familiar with the XrmServiceContext class. But, I have confirmed that XrmServiceContext inherits from Microsoft.Xrm.Client.CrmOrganizationServiceContext, which inherits from Microsoft.Xrm.Sdk.Client.OrganizationServiceContext
I use OrganizationServiceContext for LINQ queries.
If you retrieve all the accounts first by calling ToList() on the query, you can then do the comparison of the two fields. If you have too many Accounts to load all at once, you can page through them and store the mismatched ones as you go.
And, I would probably retrieve a subset of the fields rather than the whole record (as shown).
var client = new CrmServiceClient(userName, CrmServiceClient.MakeSecureString(password), string.Empty, orgName, isOffice365: true);
using (var ctx = new OrganizationServiceContext(client))
{
var q = from a in ctx.CreateQuery<Account>()
where a.AccountNumber != null
&& a.dynamics_integrationkey != null
select new Account
{
Id = a.AccountId,
AccountId = a.AccountId,
Name = a.Name,
AccountNumber = a.AccountNumber,
dynamics_integrationkey = a.dynamics_integrationkey
};
var accounts = q.ToList();
var mismatched = accounts.Where(a => a.AccountNumber != a.dynamics_integrationkey).ToList()
}

Fetching only required columns from table using LINQ to modify and save record gives a typical error

So what I was trying was to fetch only those columns from table which has to be updated and I tried it as below:
var user = (from u in context.tbl_user where u.e_Id == model.Id select
new { u.first_name, u.last_name, u.role, u.email, u.account_locked })
.FirstOrDefault();
But when I tried to assign new value to the fetched data as below
user.first_name = model.FirstName;
I saw below error getting displayed
Property or indexer 'anonymous type: string first_name, string
last_name, string role, string email, bool account_locked.first_name'
cannot be assigned to -- it is read only
But when I retrieved all the values from table without filtering as below it worked fine.
var user = (from u in context.tbl_user where u.e_Id == model.Id select u).FirstOrDefault();
Why it doesn't work for first query. I've read in many sites that it is good to retrieve only required properties from database in terms of performance and security. But I am really not able to understand what's wrong with the first approach I opted. Any explanations are much appreciated.
Update
Are there any other ways to fetch only required column and update them and store them back?

Anonymous Types properties are read-only so you can not change them.
Stop doing micro-optimizing or premature-optimization on your code. Try to write code that performs correctly, then if you face a performance problem later then profile your application and see where is the problem. If you have a piece of code which have performance problem due to finding the shortest and longest string then start to optimize this part.
We should forget about small efficiencies, say about 97% of the time:
premature optimization is the root of all evil. Yet we should not pass
up our opportunities in that critical 3% - Donald Knuth
If you want to just fetch specific columns you can create a custom class and fill the properties in your query like others have mentioned.

As others said anonymous type is read only, for achieving what you want, you will have to create a type for it with properties that are required:
public class User
{
public string FirstName {get;set;}
public string LastName {get;set;}
.....................
.....................
}
and then, you have to use it in your linq query:
var user = (from u in context.tbl_user
where u.e_Id == model.Id
select new User
{
FirstName = u.first_name,
LastName = u.last_name,
......................,
.....................
}).FirstOrDefault();

Anonymous types are read-only by design.
In order to retrieve something that you can edit, you have to create a class and select your entities into that class, as done here: How to use LINQ to select into an object?
Or here:
var user = (from u in context.tbl_user where u.e_Id == model.Id select
new User_Mini { u.first_name, u.last_name, u.role, u.email, u.account_locked })
.FirstOrDefault();
Note: you won't be able to call context.SubmitChnages() when editing this new object. You could do something like this though: LINQ to SQL: how to update the only field without retrieving whole entity
This will allow you to update only certain parts of the object.

Linq query to get all pending Loans

I have two tables in lightswitch LOANS(Id(default),..) and RELEASES(Id(default),Loan,..).i want to create a screen with all pending loans to be shown in a datagrid.so i created a wcf RIA class library.i wanto return all the loans that have no releases yet.what would be the linq query for that.
i tried this from other SO post but it gave a null reference exception.Nullreference exception was unhandled by user code.Object reference not set to an instance of object
Loan to Release has 1 : 0/1 (one loan to zero or one release)relationship
a loan can have zero or one relationship.a release must have a loan.
[Query(IsDefault = true)]
public IQueryable<PendingLoans> GetPendingLoans()
{
var res = from l in this.context.Loans
join r in this.context.Releases
on l equals r.Loan
where r.Loan == null
select new PendingLoans { BillNo = l.BillNo };
return res.AsQueryable<PendingLoans>();
}

Try this, this is linq but using lambdas instead of the query syntax
[Query(IsDefault = true)]
public IQueryable<PendingLoans> GetPendingLoans()
{
var res = this.context.Loans.Where(l=>!l.Releases.Any()).Select(l=> new PendingLoans { BillNo = l.BillNo }).AsQueryable();
return res;
}
If you want to use the query syntax this is effectively the same thing
[Query(IsDefault = true)]
public IQueryable<PendingLoans> GetPendingLoans()
{
var res = from l in this.context.Loans
where !l.Releases.Any()
select new PendingLoans { BillNo = l.BillNo };
return res.AsQueryable();
}
Now one thing I will say is that because you're converting to a PendingLoan before you say "AsQueryable", you've already enumerated over your data set (converting from one type of object to another). Therefore this will not have the same benefits of late bindings you may be trying to get out of the "AsQueryable" portion (You've already executed against the db), so you may be better off just returning an IEnumerable and forgetting about the AsQueryable part since you've already enumerated once.

Counts in Breeze.js

I have a complex query that returns item counts. If I run a query on the client, will it always return the objects, or is there a way to just return the item counts without sending the object array in the payload? I tried doing something like
var query = breeze.EntityQuery.from('Items').inlineCount(true);
but that still pulls all the records down. Any solutions?

I don't know if this exactly answers your question, but you would need to query the records in order to know how many there are (to my knowledge, there may be a more efficient way to tie Breeze directly into a SQL command that is way over my head) so you could do something like -
var query = breeze.EntityQuery.from('Items')
.take(0)
.inlineCount(true);
Edited the answer - this would return no objects and simply get the count.

The inlineCount answer already provided is absolutely correct.
Another alternative is to calculate the counts on the server and just send down the "summary". For example this server side controller method will return an array of two element objects to the client:
[HttpGet]
public Object CustomerCountsByCountry() {
return ContextProvider.Context.Customers.GroupBy(c => c.Country).Select(g => new {g.Key, Count = g.Count()});
}
This would be called via
EntityQuery.from("CustomerCountsByCountry")
.using(myEntityManager).execute()
.then(function(data) {
var results = data.results;
results.forEach(function(r) {
var country = r.Key;
var count = r.Count
});
});

var query = breeze.EntityQuery.from('Items')
.take(0)
.inlineCount(true);

C# EF / LINQ hack fix hitting perfomance? Other way of fixing?

I've been learning C# / LINQ / ASP.NET / MVC 3 / EF for a few months now comming from Java / Icefaces / Ibatis base (Real world uses .NET D;). I really enjoy LINQ / Entity Framework from the .NET Framework but I'm having a few issues understand what's really happening behind the scenes.
Here's my problem:
I'm using a AJAX / JSON fed jQuery datatable (that I highly recommend to anyone in need of a free web datatable system by the way). I have a method in my MVC3 application that returns a JSON result of the data needed by the table, doing the sorting and all. Everything is working nicely and smoothly. However, I'm having a concern with the "dirty" hack I had to do to make this work.
Here's the complete code:
//inEntities is the Entity Framework Database Context
//It includes the following entities:
// Poincon
// Horaire
// HoraireDetail
//Poincon, Horaire and HoraireDetail are "decorated" using the Metadata technic which
//adds properties methods and such to the Entity (Like getEmploye which you will see in
//the following snippet)
//
//The Entity Employe is not a database data and therefor not handled by the EF.
//Instead, it is a simple object with properties that applies Lazy Loading to get an
//Employe Name based off of his Employe ID in the Active Directory. An employe object
//can be constructed with his Employe ID which will expose the possibility of getting
//the Employe Name from the AD if needed.
[HttpPost]
public JsonResult List(FormCollection form)
{
String sEcho;
int iDisplayStart;
int iDisplayLength;
String sSearch;
int iSortingCols;
Dictionary<String, String> sorting;
try
{
sEcho = form["sEcho"];
iDisplayStart = int.Parse(form["iDisplayStart"]);
iDisplayLength = int.Parse(form["iDisplayLength"]);
sSearch = form["sSearch"];
iSortingCols = int.Parse(form["iSortingCols"]);
sorting = new Dictionary<string,string>();
for (int i = 0; i < iSortingCols; i++)
sorting.Add(form["mDataProp_" + form["iSortCol_" + i]].ToUpper(), form["sSortDir_" + i].ToUpper());
}
catch
{
HttpContext.Response.StatusCode = 500;
return null;
}
var qPoincon = inEntities.Poincons.AsEnumerable();
var lPoincon = qPoincon.Select(o => new
{
o.id,
emp = o.getEmploye(),
o.poinconStart,
o.poinconEnd,
o.commentaire,
o.codeExceptions
}).AsEnumerable();
//Search
lPoincon = lPoincon.Where(p => (p.emp.empNoStr.Contains(sSearch) || p.emp.empNom.Contains(sSearch) || (p.commentaire != null && p.commentaire.Contains(sSearch))));
//Keep count
int iTotalDisplayRecords = lPoincon.Count();
//Sorting
foreach(KeyValuePair<String,String> col in sorting)
{
switch (col.Key)
{
case "EMPNO":
if (col.Value == "ASC")
lPoincon = lPoincon.OrderBy(h => h.emp.empNo);
else
lPoincon = lPoincon.OrderByDescending(h => h.emp.empNo);
break;
case "POINCONSTART":
if (col.Value == "ASC")
lPoincon = lPoincon.OrderBy(h => h.poinconStart);
else
lPoincon = lPoincon.OrderByDescending(h => h.poinconStart);
break;
case "POINCONEND":
if (col.Value == "ASC")
lPoincon = lPoincon.OrderBy(h => h.poinconEnd);
else
lPoincon = lPoincon.OrderByDescending(h => h.poinconEnd);
break;
case "COMMENTAIRE":
if (col.Value == "ASC")
lPoincon = lPoincon.OrderBy(h => h.commentaire);
else
lPoincon = lPoincon.OrderByDescending(h => h.commentaire);
break;
}
}
//Paging
lPoincon = lPoincon.Skip(iDisplayStart).Take(iDisplayLength);
//Building Response
var jdt = new
{
iTotalDisplayRecords = iTotalDisplayRecords,
iTotalRecords = inEntities.Poincons.Count(),
sEcho = sEcho,
aaData = lPoincon
};
return Json(jdt);
}
As you can see, when I'm grabbing the entire list of "Poincons" from the EF and turning it into a Enumerable. From my current understanding, turning the LINQ query into a Enumerable "kills" the link to the EF, or in other words, will generate the SQL required to get that list at that point instead of keeping the LINQ data until the end and execute a percise query that will return only the data you require. After turning this LINQ Query into a Enumerable, I'm heavily filtering the LINQ (since there is paging, sorting, searching in the datatable). This leads me to thinkg that what my code is currently doing is "Grab all the "Poincons" from the database and put it into the web server's memory as a Enumerable, do your work with the Enumerable then serialize the result as a JSON string and send it to the client.
If I'm correct, the performance hit is quite heavy when you hit the couple thousand of entries (which will happen quite fast once in production... everytime an employe comes to work, it will add 1 entry. 100 employes, ~300 work days a year, you get the idea).
The reason for this hack is that the EF does not know what "getEmploye" method of "Poincon" is, therefor throwing an exception at runtime similar to this:
LINQ to Entities ne reconnaît pas la méthode « PortailNorclair.Models.Employe getEmploye() », et cette dernière ne peut pas être traduite en expression de magasin.
Approximated traduction (If anyone can let me know in a comment how to configure IIS / ASP.NET to display errors in english while keeping the globalization in a foreign language, I would be really grateful. French information about error messages is sometimes lacking):
LINQ to Entity does not recognize the method " PortailNorclair.Models.Employe getEmploye()" and the following could not be translated to a SQL expression.
The "getEmploye" method instances and returns a Employe object with the employe id found in the Poincon object. That Employe object has properties that "lazy loads" information like the employe name from the Active Directory.
So the question is: How can I avoid the performance hit from using .AsEnumerable() on the non-filtered list of objects?
Thanks a lot!

The "getEmploye" method instances and returns a Employe object with
the employe id found in the Poincon object. That Employe object has
properties that "lazy loads" information like the employe name from
the Active Directory.
You should be storing the Employee Name in the database, so you can then order, sort, skip and take in your Linq Query without having to load every employee object.
If empNoStr, empNom, and empNo were all in the database, you could retrieve just the records you want, and call getEmploye() (loading whatever else you need from active directory, or wherever) for each of those.

There are some classes on which your program performs its main work.
There are other classes which represent to database rows.
If you keep them separated, you can also separate actions you intend to occur in the database from actions you intend to perform locally. This makes it trivial to avoid loading the full table, when specific rows are required.
I see you're also doing Paging locally, while the database can do that and save your webserver some memory.