Develop Reference

Linq Queryable from OrganizationServiceProxy throws exception when comparing two fields

I've tried to reduce this example to remove the OrganizationServiceProxy/XrmServiceContext, but now I believe the issue is originating there. I'm pulling data from a Dynamics 365 instance using the code generated by CrmSvcUtil.exe from Microsoft. Here is the relevant snippet:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
CrmServiceClient client = new CrmServiceClient(userName, CrmServiceClient.MakeSecureString(password), string.Empty, orgName, isOffice365: true);
using (OrganizationServiceProxy service = client.OrganizationServiceProxy)
{
XrmServiceContext crm = new XrmServiceContext(service);
var q = crm.CreateQuery<Account>();
List<Account> accounts = q.Where(x => x.AccountNumber != x.dynamics_integrationkey).ToList();
}
I get the following exception:
variable 'x' of type '{company-specific}.Account' referenced from scope '', but it is not defined
This is happening on the last line when the Linq is actually executed. It's only when it's created through the Microsoft.Xrm.Sdk.Client.OrganizationServiceContext that I get the error.
Things I've checked:
I get the error when comparing any two fields on the same object (not just Account), even when both are the same exact data type.
If I replace crm.CreateQuery... with a hand-populated List...AsQueryable() the Linq works fine
Most frustratingly, if I work through an intermediary list with List<Account> a = q.ToList(); to rasterize the result set first then this all works fine. It takes much longer to run because I've forfeited any lazy loading, but it works without error.
I've looked at other answers related to the referenced from scope '' but not defined Linq error, but they are all about malformed Linq. I know this Linq is well formed because it works fine if I change the underlying Queryable.
Clearly something is different between a List...ToQueryable() and the Microsoft.Xrm.Sdk.Linq.Query<> created by the XrmServiceContext (which inherits CreateQuery from Microsoft.Xrm.Sdk.Client.OrganizationServiceContext), but I don't know what it is.

The query you are trying to build (compare a field value with another field value on the same entity) is not possible in Dynamics.
The LINQ Provider still uses QueryExpression under the hood, meaning that your LINQ condition will be converted in a ConditionExpression, and that comparison is not possible.
Similar questions:
https://social.microsoft.com/Forums/en-US/d1026ed7-56fd-4f54-b382-bac2fc5e46a7/linq-and-queryexpression-compare-entity-fields-in-query?forum=crmdevelopment
Dynamics QueryExpression - Find entity records where fieldA equals fieldB

This morning I realized that my answer contained the flaw of comparing two fields against each other in a LINQ query. My bad. Thank you Guido for calling that out as impossible.
I generate my proxy classes with a 3rd party tool, so I'm not familiar with the XrmServiceContext class. But, I have confirmed that XrmServiceContext inherits from Microsoft.Xrm.Client.CrmOrganizationServiceContext, which inherits from Microsoft.Xrm.Sdk.Client.OrganizationServiceContext
I use OrganizationServiceContext for LINQ queries.
If you retrieve all the accounts first by calling ToList() on the query, you can then do the comparison of the two fields. If you have too many Accounts to load all at once, you can page through them and store the mismatched ones as you go.
And, I would probably retrieve a subset of the fields rather than the whole record (as shown).
var client = new CrmServiceClient(userName, CrmServiceClient.MakeSecureString(password), string.Empty, orgName, isOffice365: true);
using (var ctx = new OrganizationServiceContext(client))
{
var q = from a in ctx.CreateQuery<Account>()
where a.AccountNumber != null
&& a.dynamics_integrationkey != null
select new Account
{
Id = a.AccountId,
AccountId = a.AccountId,
Name = a.Name,
AccountNumber = a.AccountNumber,
dynamics_integrationkey = a.dynamics_integrationkey
};
var accounts = q.ToList();
var mismatched = accounts.Where(a => a.AccountNumber != a.dynamics_integrationkey).ToList()
}

NpGsql EntityFramework 6 - "An operation is already in progress"

I am working on a project to connect to PostgreSQL database using NpGsql EntityFramework 6. I am getting the exception in question heading, when I try to execute the query in GetAdminUsersCount:
public class GenieRepository : IDisposable
{
GenieDbContext db = new GenieDbContext();
public IEnumerable<User> GetUsers()
{
return db.Users;
}
}
public int GetAdminUsersCount()
{
return repo.GetUsers().Where(u => u.Role.RoleName == "Administrator").Count();
}
What is the reason for this error and how to resolve it?

Use a ToList<T> right after the LINQ query like this:
using (ElisContext db = new ElisContext()) {
var q = from a in db.aktie select a;
List<aktie> akties = q.ToList<aktie>();
foreach (aktie a in akties) {
Console.WriteLine("aktie: id {0}, name {1}, market name {2}"
, a.id, a.name, a.marked.name);
}
}
Note the q.ToList<T> which does the trick. .NET delays the execution of the linq statement to the latest moment, which may be part of the problem. I have tried to use q in the foreach without success.

The issue is caused by the return type of the GetUsers() method. Since it is IEnumerable<User>, LINQ-to-SQL will load the result into memory (row by row) and subsequent Where, OrderBy, Select, Count, etc. calls will be executed in memory. When the Where condition is evaluated, the original result set is still being iterated over, but the relation User.Role needs to be resolved on the DB (that's where the "An operation is already in progess" error message comes from).
If the return type is changed to IQueryable<User>, the query won't be executed until the Count method is called, and furthermore, the whole query will be translated into SQL returning only the count without ever loading any User records into memory.
See also this related question/answer: Returning IEnumerable<T> vs. IQueryable<T>
The answer suggesting to call ToList() on the query, will load the entire result set into memory, which makes your error go away, but depending on the size of the result set, could also be very inefficient.

ASP.NET MVC 3 / Razor - Strange Caching Issue

I'm using a LinqToSql query to select a list of groups from a database and spew out a table. I've written a custom class to cache the results of this query for better performance. Trouble is, whenever I implement the caching class, I get weird appending behaviour from the outputting statement.
My results are outputted in the format
Test Group (1)
where "Test Group" is the name, and (1) is the number of members within that group. Here's the code that appends the count to the name (from the view)
<td>#group.group_name (#group.num_total)</td>
When I pull this from a live linq query returning groups, everything works as expected.
However, when I use my caching class, every successive page load adds the number on to the end of the group title:
Test Group (1) (1) (1) (1) (1) (1)
This only happens when I use the caching class (included below). I've been over the cache class and there's no reason I can see why this would be happening.
I can think of several workarounds for this issue, so its not a show stopper, but I'm curious as to what the fudge is going on. Any ideas?
Caching Class:
public class Cache
{
public static int user_id {
get { return
Convert.ToInt32(
Membership.GetUser(
HttpContext.Current.User.Identity.Name
).ProviderUserKey
);
}
}
public static void GetGroups_InvalidateCache()
{
if (HttpContext.Current.Cache["GetGroups_" + user_id] != null)
HttpContext.Current.Cache.Remove("GetGroups_" + user_id);
}
public static ICollection<Groups> GetGroups()
{
if (HttpContext.Current.Cache["GetGroups_" + user_id] == null)
{
using(DBContext db = new DBContext())
{
var Groups = (from g in db.Groups
where g.user_id == user_id
select g).ToList();
HttpContext.Current.Cache.Insert(
"GetGroups_" + user_id,
Groups,
null,
DateTime.Now.AddMinutes(5),
TimeSpan.Zero
);
}
}
return HttpContext.Current.Cache["GetGroups_" + user_id]
as ICollection<Groups>;
}
}
UPDATE:
I've now implemented Adam Tuliper and Paul Tyng's suggestions of calling the data context with the using clause, ending the linq statement with ToList() and using ICollection instead of IQueryable. The problem is still occurring.
Another interesting observation: The issue only happens if I navigate away to another page and return. If I simply refresh the page, it doesn't happen (Although any previous number additions still remain when I refresh)

Instead of returning IQueryable, try using simply IEnumerable and also using
using(DBContext db = new DBContext())
{
var Groups =
(from g in db.Groups
where g.user_id == user_id
select g).ToList();
...
}
Also dispose of your context as in the above statement (with the using clause)
The ToList() forces the execution "now" - I think you are potentially having deferred execution issues

Bulk inserts and duplicate records with LINQ to SQL

Is there a "best practice" way of handling bulk inserts (via LINQ) but discard records that may already be in the table? Or I am going to have to either do a bulk insert into an import table then delete duplicates, or insert one record at a time?
08/26/2010 - EDIT #1:
I am looking at the Intersect and Except methods right now. I am gathering up data from separate sources, converting into a List, want to "compare" to the target DB then INSERT just the NEW records.
List<DTO.GatherACH> allACHes = new List<DTO.GatherACH>();
State.IState myState = null;
State.Factory factory = State.Factory.Instance;
foreach (DTO.Rule rule in Helpers.Config.Rules)
{
myState = factory.CreateState(rule.StateName);
List<DTO.GatherACH> stateACHes = myState.GatherACH();
allACHes.AddRange(stateACHes);
}
List<Model.ACH> newRecords = new List<Model.ACH>(); // Create a disconnected "record set"...
foreach (DTO.GatherACH record in allACHes)
{
var storeInfo = dbZach.StoreInfoes.Where(a => a.StoreCode == record.StoreCode && (a.TypeID == 2 || a.TypeID == 4)).FirstOrDefault();
Model.ACH insertACH = new Model.ACH
{
StoreInfoID = storeInfo.ID,
SourceDatabaseID = (byte)sourceDB.ID,
LoanID = (long)record.LoanID,
PaymentID = (long)record.PaymentID,
LastName = record.LastName,
FirstName = record.FirstName,
MICR = record.MICR,
Amount = (decimal)record.Amount,
CheckDate = record.CheckDate
};
newRecords.Add(insertACH);
}
The above code builds the newRecords list. Now, I am trying to get the records from this List that are not in the DB by comparing on the 3 field Unique Index:
AchExceptComparer myComparer = new AchExceptComparer();
var validRecords = dbZach.ACHes.Intersect(newRecords, myComparer).ToList();
The comparer looks like:
class AchExceptComparer : IEqualityComparer<Model.ACH>
{
public bool Equals(Model.ACH x, Model.ACH y)
{
return (x.LoanID == y.LoanID && x.PaymentID == y.PaymentID && x.SourceDatabaseID == y.SourceDatabaseID);
}
public int GetHashCode(Model.ACH obj)
{
return base.GetHashCode();
}
}
However, I am getting this error:
LINQ to Entities does not recognize the method 'System.Linq.IQueryable1[MisterMoney.LARS.ZACH.Model.ACH] Intersect[ACH](System.Linq.IQueryable1[MisterMoney.LARS.ZACH.Model.ACH], System.Collections.Generic.IEnumerable1[MisterMoney.LARS.ZACH.Model.ACH], System.Collections.Generic.IEqualityComparer1[MisterMoney.LARS.ZACH.Model.ACH])' method, and this method cannot be translated into a store expression.
Any ideas? And yes, this is completely inline with the original question. :)

You can't do bulk inserts with LINQ to SQL (I presume you were referring to LINQ to SQL when you said "LINQ"). However, based on what you're describing, I'd recommend checking out the new MERGE operator of SQL Server 2008.
Inserting, Updating, and Deleting Data by Using MERGE
Another example here.

I recommend you just write the SQL yourself to do the inserting, I find it is a lot faster and you can get it to work exactly how you want it to. When I did something similar to this (just a one-off program) I just used a Dictionary to hold the ID's I had inserted already, to avoid duplicates.
I find LINQ to SQL is good for one record or a small set that does its entire lifespan in the LINQ to SQL.
Or you can try to use SQL Server 2008's Bulk Insert .

One thing to watch out for is if you queue more than 2000 or so records without calling SubmitChanges() - TSQL has a limit on the number of statements per execution, so you cannot simply queue up every record and then call SubmitChanges() as this will throw an SqlException, you need to periodically clear the queue to avoid this.

"Member access 'DataType MemberName' of 'Namespace.ClassName' not legal on type 'System.Collections.Generic.IEnumerable`1[Namespace.ClassName]."

I would love a solution to my current problem, but I would love EVEN MORE if I can understand what that error actually means.
I have LINQ to SQL classes for two tables in my DB: Providers and Assignments. I added the following member to the Provider class:
public IEnumerable<Assignment> Assignments
{
get
{
return (new linqDataContext())
.Assignments
.Where(a => a.ProviderID == this.ProviderID);
}
}
Then, I bind a GridView using a query that pulls from the parent Provider and uses the child member Assignments, like this:
protected void PopulateProviders()
{
linqDataContext context = new linqDataContext();
var list = from p in context.Providers
where (p.Assignments.Count(a => a.Category == ddlCategory.SelectedValue) > 0)
select p;
lvProviders.DataSource = list;
lvProviders.DataBind();
}
At .DataBind(), when it actually runs the query, it throws the following error:
Member access 'System.String Category' of 'Namespace.Assignment' not legal on type 'System.Collections.Generic.IEnumerable`1[Namespace.Assignment].
I've tried checking for nulls (a => a != null && a.Category ...) but that hasn't worked. I'm not sure what to try next. I think that, if I knew what the error is trying to tell me, I could find the solution. As it stands, I don't know why member access would be illegal.

That Assignments property is all wrong. First of all, property getters should not have side-effects, and more importantly, entity classes should never have reverse dependencies on the DataContext. Linq to SQL has no way to decipher this query; it's relying on a property that does all sorts of crazy stuff that Linq to SQL can't hope to understand.
Get rid of that Assignments property now. Instead of doing that, you need to write this query as a join:
int category = (int)ddlCategory.SelectedValue;
var providers =
from p in context.Providers
join a in context.Assignments
on p.ProviderID equals a.ProviderID
into g
where g.Count(ga => ga.Category == category) > 0
select p;
That should do what you're trying to do if I understood the intent of your code correctly.
One last side note: You never dispose properly of the DataContext in any of your methods. You should wrap it like so:
using (var context = new linqDataContext())
{
// Get the data here
}

I think somewhere it doesn't know that type that is in the IEnumerable. You are trying to call a method that is not part of the IEnumerable inteface.
Why don't you just move the query from the property out to the PopulateProviders() method?

Remove your custom-defined Assignments property. In the your Linq-To-SQL dbml file, create an association between Providers and Assignments, with Providers as the parent property, and ProviderID as the Participating Property for both entities. LINQ will generate a property "IEnumerable Assignments" based on matches between ProviderID using a consistent DataContext.