Develop Reference

Split a string in a linq select expression

The geoStartLoc holds the string in this format "54.5,44.5". I am trying to split and store the results in lat and longitude. I am getting the following error
LINQ to Entities does not recognize the method 'System.String[] Split(Char[])' method, and this method cannot be translated into a store expression. (select statement)
var data = from UberTrip in db.UberTrips
group UberTrip by new { UberTrip.startLoc, UberTrip.geoStartLoc }
into startLocGroup
select new LocationGroup() {
startLocation = startLocGroup.Key.startLoc,
latitude = startLocGroup.Key.geoStartLoc.Split(',').Count().ToString(),
//longitude= startLocGroup.Key.
countTrips = startLocGroup.Count()
};

This is a schema problem in your database. It's bad to ever store delimited values in a column. You should really have two columns: one for geoLatitude, and one for geoLongitude.
But since you probably can't make that change on your own, what you'll need to do is make sure the data is pulled down to your program's code and then split it there. Right now, linq is trying to take the expression tree created by this code and convert it to an SQL query, and it can't do it for the Split() call because not all supported database targets have an analogous Split method available. You need to save that part for after the data is loaded to memory in your program.
To accomplish this, just retrieve the full geoStartLoc string (into an anonymous type if you have to), use .ToList() to force the query compile and retrieve all the data, and then use a .Select() to convert to your LocationGroup objects.

Whenever you get this error message you are performing a query AsQueryable instead of AsEnumerable.
The main difference is that AsQueryable will usually be performed in another process like a database, while AsEnumerable will be performed in-memory in your process.
An IEnumerable object knows how to create an Enumerator. An Enumerator, knows how to do two things: "Give me the first element of your sequence", and "Give me the next element of your sequence (or null if there is no next element).
If your query is AsEnumerable it knows can use all classes and data from your process to create the enumerator. Therefore it can call functions like String.Split.
The AsQueryable does not hold all data to create the enumerator, it holds an Expression and a Provider. The Provider knows where the query needs to be performed on (usually a database, but it can also be a json string or a web service), and it knows how to translate the Expression into a format that the executor understands. In your case, the Provider knows how to translate the Expression into a SQL statement suitable for your database.
Of course SQL does not know your own defined functions. Although there are a lot of .NET functions that can be translated into SQL, not every function can. String.Split is one of them.
See: Supported and Unsupported LINQ Methods (LINQ to Entities)
To solve your problems you could bring the queried data to local memory. This is done using the extension function Enumerable.AsEnumerable(). After that you can use your sequence as if it is an IEnumerable.
The disadvantage of AsEnumerable is that it brings the queried data to local memory. This would be a waste if you remove a lot of this data to create your end result. Therefore make sure that you do your AsEnumerable with not too much data.
Luckily you need the Split in your final Select, so all data you need as input of your final select is thrown away, unless you will do a Skip / Take / FirstOrDefault / etc. In that case it is better to limit your Select by doing your Skip / Take etc before AsEnumerable().
I'm more familiar with MethodSyntax.
Your query divided into smaller steps (if desired make one LINQ):
// still AsQueryable:
var startLocGroups = db.UberTrips
.GroupBy(uberTrip => new {uberTip.startLoc, uberTrip.geoStartLoc)
// make asEnumerable
var localStartLocGroups = startLocGroups.AsEnumerable();
// now you can do your Select
var result = localStartLocGroups
.Select(group => new LocationGroup()
{
startLocation = startLocGroup.Key.startLoc,
latitude = group.Key.geoStartLoc.Split(',')
.Count()
.ToString(),
//longitude= sgroup.Key.
countTrips = group.Count(),
});

Since that isn't supported by the underlying Linq to Entities you'll have to do it after you get the data but before you return the results.
This way you do the query to get the data without invoking the string split function and then modify the results into the proper format.
var data = from UberTrip in db.UberTrips
group UberTrip by new { UberTrip.startLoc, UberTrip.geoStartLoc } into startLocGroup
select new {
startLocation = startLocGroup.Key.startLoc,
geoStartLoc = startLocGroup.Key.geoStartLoc,
countTrips = startLocGroup.Count()
};
return data.Select(trip => new LocationGroup() {
startLocation = trip.startLocation,
latitude = trip.geoStartLoc.Split(',')[0],
longitude= trip.geoStartLoc.Split(',')[1],
countTrips = trip.countTrips
}).ToList();

Use a NaturalSortComparer in a LINQ Where Clause

Say I have a table Table1 with a string field [ProductString] with values:
Alpha, alphanumeric or numeric: eg ABC, B4, U2, C 5, 100, U1, U5, U6, U11
I want to be able to take a where clause like "ProductString >= U5", and pass this to a LINQ statement as a string so it evaluates
Table1.Where(t=> t.ProductString >= 'U5');
Normally this would return results U5 and U6.
However, this I want to be able to use a NaturalSortComparer somehow so that the results returned are U5, U6 and U11.
I know how to use the comparer in an OrderBy, by I wanted to be able to use it at the Where stage.

Using natural sort comparer:
var comparer = new NaturalComparer();
Table1.Where(t=>
comparer.Compare(t.ProductString, "U5") >= 0);
Presuming all your product strings is on the format U%number% then why not abuse that fact?
Table1.Where(t=> int.Parse(t.ProductString.Replace("U","")) >= 5);
If you're using LINQ to Entities I'm not certain this will compile to a store expression (i.e that SQL knows what to do with this - I guess it should).

I'm a little confused, given the accepted answer, about whether this question relates to LINQ to Entities or not. The accepted answer doesn't appear to be a solution that would work in the LINQ to Entities context, but the comments on the question by the OP seem to confirm that this is being executed in the database context. Anyway, this answer is specifically targeted toward LINQ to Entities.
I think doing this in SQL Server would be hard, but not impossible. The problem is that .NET knows what NaturalSortComparer is, but SQL Server (where you want the query to ultimately take place) has no such concept. The best idea I can think of would consist of 2 parts:
Create a UDF (User Defined Function) in SQL server that will give a product that is orderable via natural sort: CREATE FUNCTION Naturalize(#val as nvarchar(max)) RETURNS nvarchar(1000). There's a pretty cool answer here that creates a UDF wrapper around a CLR function to accomplish just that.
Next create a function mapping for your DbContext that maps the UDF above to a function that can be called inside an EF query against the DbContext. Something like this:
[DbFunction("MyContext", "Naturalize")]
public static string Naturalize(this string value)
{
throw new NotSupportedException("This function can only be invoked from LINQ to Entities.");
}
Once you've got these two pieces in place, you can readily use this new function inside an entity query to compare strings using the Naturalized value in the comparison:
Table1.Where(t=> t.ProductString.Naturalize() >= "U5".Naturalize());
Bear in mind that the UDF will be executed against every row contained in the query, which is the whole table in the above example. You'll want to make sure to pare down your query to something manageable before applying the function as a sub-query. Or you may want to try applying some type of UDF-based index on the table in question.

If you are going to be doing searches like this a lot, then what will be the best thing to do is add two new fields to your table, [ProductCode] & [ProductNumber] which separate the two parts of the [ProductString].
Then you comparison becomes:
Table1.Where(t=> t.ProductCode == "U" && t.ProductNumer > 5);

Concatenating three lists into one with LINQ throws an exception

Ok, I must be doing something dumb, but shouldn't this work? I have following three lists:
var commonViews = (from v in context.TPM_VIEWS where v.VIEWID < 0 select v); // IQueryable<TPM_VIEWS>
var ownedViews = (from v in context.TPM_VIEWS where v.OWNERID == userId && v.VIEWID > 0 select v); // IQueryable<TPM_VIEWS>
var sharedViews = (from v in context.TPM_USER.Include("TPM_VIEWS2") where v.USERID == userId select v).First().TPM_VIEWS2; // EntityCollection<TPM_VIEWS>
Each list has the proper values and count. I can return any one of these lists:
return commonViews.ToList();
And I can return a any two of these lists:
return commonViews.Concat(ownedViews).ToList();
However, when I try to return all three:
return commonViews.Concat(ownedViews).Concat(sharedViews).ToList();
I get the exception:
Unable to create a constant value of type 'Entity.TPM_VIEWS'. Only
primitive types or enumeration types are supported in this context.
What am I doing wrong? All three values are indeed enumerable. Mostly, I'm asking this question because it's the best possible way to guarantee I'll notice the problem 30 seconds after posting.
UPDATE:
I'm 93% sure the problem is here:
var sharedViews = (from v in context.TPM_USER.Include("TPM_VIEWS2") where v.USERID == userId select v).First().TPM_VIEWS2;
This looks like an enumerable list of TPM_VIEWS object, and I can call ToList() on it and get the correct data, but it doesn't play well with the other lists.
UPDATE 2:
This actually works. Points to the person who can tell me why!
commonViews.ToList().Concat(ownedViews.ToList()).Concat(sharedViews.ToList()).ToList();

The problem is that Concat() on an EF IQueryable<T> will turn the entire concatenation into a single query.
When you call .Concat(sharedViews), you're passing a scalar (pre-loaded) collection of your nested entity class.
EF doesn't know how to convert that into a query, so it complains.
You can make it faster by calling AsEnumerable() instead of ToList().

This actually works. Points to the person who can tell me why!
commonViews.ToList().Concat(ownedViews.ToList()).Concat(sharedViews.ToList()).ToList();
That's because each of the original queries is executed separately; you're only concatenating the results in memory. There seems to be a bug in the Entity Framework query translator when you combine the 3 queries, but when you call ToList on each of them, they're no longer EF queries, they're just lists, so they're concatenated using Linq to Objects.

Differences when using IEnumerable and IQueryable as a type of ObjectSet

As I understand it when I use LINQ extension methods (with lambda expression syntax) on IQueryable that is in the fact instance of ObjectSet they are translated into LINQ to SQL queries. What I mean is that command
IQueryable<User> users = db.UserSet;
var users32YearsOld = users.Where(user => user.Age == 32);
is exactly the same as
IQueryable<User> users = db.UserSet;
var users32YearsOld = from user in users where user.Age == 32 select user;
So non of them hits database until they users32YearsOld are enumerated in for cycle or such. (Hope I understand this correctly).
But what is going to happen if I don't mask that ObjectSet as IQueryable but as IEnumerable ? So if the type of it is IEnumerable ?
IEnumerable<User> users = db.UserSet;
var users32YearsOld = users.Where(user => user.Age == 32);
Is it going to hit the database immediately (if so then when ? Right on the first line or on the second) ? Or is it going to behave as the previous command that is will not hit database until users32YearsOld is enumerated ? Will there be any difference if I use following instead ?
IEnumerable<User> users = db.UserSet;
var users32YearsOld = from user in users where user.Age == 32 select user;
Thank you

Undeleting my answer because I just tested it and it works exactly as I described:
None of mentioned queries will hit the database because there was no enumeration. The difference between IQueryable query and IEnumerable query is that in the case of IQueryable the filtering will be executed on the database server whereas in the case of IEnumerable all objects will be loaded from the database to a memory and the filtering will be done in .NET code (linq-to-objects). As you can imagine that is usually performance killer.
I wrote simple test in my project:
[TestMethod]
public void Test()
{
// ObjectQuery<Department> converted ot IEnumerable<Department>
IEnumerable<Department> departmetns = CreateUnitOfWork().GetRepository<Department>().GetQuery();
// No query execution here - Enumerable has also deffered exection
var query = departmetns.Where(d => d.Id == 1);
// Queries ALL DEPARTMENTS here and executes First on the retrieved result set
var result = departmetns.First();
}

Here's a simple explanation:
IEnumerable<User> usersEnumerable = db.UserSet;
IQueryable<User> usersQueryable = db.UserSet;
var users = /* one of usersEnumerable or usersQueryable */;
var age32StartsWithG = users.Where(user => user.Age == 32)
.Where(user => user.Name.StartsWith("G");
If you use usersEnumerable, when you start enumerating over it, the two Wheres will be run in sequence; first the ObjectSet will fetch all objects and the objects will be filtered down to those of age 32, and then these will be filtered down to those whose name starts with G.
If you use usersQueryable, the two Wheres will return new objects which will accumulate the selection criteria, and when you start enumerating over it, it will translate all of the criteria to a query. This makes a noticeable difference.
Normally, you don't need to worry, since you'll either say var users or ObjectSet users when you declare your variable, which means that C# will know that you are interested in invoking the most specific method that's available on ObjectSet, and the IQueryable query operator methods (Where, Select, ...) are more specific than the IEnumerable methods. However, if you pass around objects to methods that take IEnumerable parameters, they might end up invoking the wrong methods.
You can also use the way this works to your advantage by using the AsEnumerable() and AsQueryable() methods to start using the other approach. For example, var groupedPeople = users.Where(user => user.Age > 15).AsEnumerable().GroupBy(user => user.Age); will pull down the right users with a database query and then group the objects locally.
As other have said, it's worth repeating that nothing happens until you start enumerating the sequences (with foreach). You should now understand why it couldn't be any other way: if all results were retrieved at once, you couldn't build up queries to be translated into a more efficient query (like an SQL query).

The difference is that IEnumerable performs the filters if they are more, one at a time. For example, from 100 elements will output 20 by the first filter and then will filter second time the needed 10. It will make one query to the database but will download unnecessary data. Using IQueryable will download again with one query but only the required 10 items. The following link gives some excellent examples of how these queries work:
https://filteredcode.wordpress.com/2016/04/29/ienumerable-vs-iqueryable-part-2-practical-questions/

You are correct about IQueryable. As for IEnumerable, it would hit the database immediately upon assigning IEnumerable user.
There is no real difference between using Linq Extensions vs. syntax in the example you provided. Sometimes one or the other will be more convenient (see linq-extension-methods-vs-linq-syntax), but IMO it's more about personal preference.

How to deal with datatypes returned by LINQ

I'm new to LINQ, I've used LINQ to SQL to link to two tables, it does return data, which is cool. What I'm trying to understand is what datatype is being returned and how do I work with this datatype?
I'm used to dealing with datatables. Are we throwing out datatables (and all the other ADO.Net object like rows, datasets etc.) now if using LINQ? If so, what are we replacing that with and how can I use it to do everything I did before with datatables? Also--does it make sense to replace datables, was there a deficiency with them?
Here is some code:
protected IEnumerable<string> GetMarketCodes()
{
LINQOmniDataContext db = new LINQOmniDataContext();
var mcodes = from p in db.lkpMarketCodes
orderby 0
select p;
return (IEnumerable<string>) mcodes;
}
This code does currently return data (I can see it in debug), but errors at the "return" line, because apparently my datatype is not IEnumerables, which was my best guess. So, one thing I'd like to understand as well is what datatype is my data being put into and how to return it to the calling function.

It is returning an IQueryable<lkpMarketCode>, assuming that that lkpMarketCode is the type of data in db.lkpMarketCodes. If you want the strings, you need to select p.SomeProperty;, not just select p;.
You shouldn't need to cast (since IQueryable<T> implements IEnumerable<T>); it should also tell you this if you hover on mcodes.

I find it more convenient to return List<>'s so I know what I'm dealing with. So your code would be:
protected List<string> GetMarketCodes()
{
LINQOmniDataContext db = new LINQOmniDataContext();
var mcodes = from p in db.lkpMarketCodes
orderby 0
select p.SomeProperty;
return mcodes.ToList();
}
Having said that, I've hardly used LINQ-to-SQL so there are probably better ways around..

It's returning an IQueryable object.
How does your table look like? I'm guessing the error is because your lkpMarketCodes table is not just one string column. It's returning the whole table.
If you want to return just an IEnumerable of strings, you'll have to return something that looks like this (I'm sure the syntax is a bit off):
var mcodes = from p in db.lkpMarketCodes
orderby 0
select new { p.StringColumnName };

LINQ returns IQueryable<type>'s. This is a superset of IEnumerable. The reason you are getting an error is that your query is not returning an IQueryable<string> it's returning an IQueryable<lkpMarketCodes>. lkpMarketCodes is most likely an object, which can be thought of as similar to a row of records.
LINQ is a Object-Relational mapper, it maps Columns and Rows to Fields and Objects.
You can do pretty much all the same things that you could in ADO, but it works with objects rather than generic rows, so it's more type safe.
In your example, i'm going to assume that lkpMarketCodes is a table, and that table consists of at least two fields, mcode and description.
If you want to return an IEnumerable<string> of mcode's, you would do something like this:
protected IEnumerable<string> GetMarketCodes()
{
LINQOmniDataContext db = new LINQOmniDataContext();
var mcodes = from p in db.lkpMarketCodes
orderby 0
select p.mcode;
return mcodes;
}
This will return your IEnumerable<string> of codes. One trick you can use to find out types is to simply use the variable after its declaration, then hover your mouse over the variable name and a popup will tell you it's type.
That is, if you hover over the return mcodes, it will tell you the type, but it will not tell you the type if you hover over the var mcodes.