Develop Reference

MongoDB and returning collections efficiently

I am very new to Mongo (this is actually day 1) and using the C# driver that is available for it. One thing that I want to know (as I am not sure how to word it in Google) is how does mongo handle executing queries when I want to grab a part of the collection.
What I mean by this is that I know that with NHibernate and EF Core, the query is first built and it will only fire when you cast it. So say like an IQueryable to IEnnumerable, .ToList(), etc.
Ex:
//Query is fired when I call .ToList, until that point it is just building it
context.GetLinqQuery<MyObject>().Where(x => x.a == 'blah').ToList();
However, with Mongo's examples it appears to me that if I want to grab a filtered result I will first need to get the collection, and then filter it down.
Ex:
var collection = _database.GetCollection<MyObject>("MyObject");
//Empty filter for ease of typing for example purposes
var filter = Builders<MyObject>.Filter.Empty;
var collection.Find(filter).ToList();
Am I missing something here, I do not think I saw any overload in the GetCollection method that will accept a filter. Does this mean that it will first load the whole collection into memory, then filter it? Or will it still be building the query and only execute it once I call either .Find or .ToList on it?
I ask this because at work we have had situations where improper positioning of .ToList() would result is seriously weak performance. Apologies if this is not the right place to ask.
References:
https://docs.mongodb.com/guides/server/read_queries/

The equivalent to your context.GetLinqQuery<MyObject>() would be to use AsQueryable:
collection.AsQueryable().Where(x => x.a == "blah").ToList();
The above query will be executed server side* and is equivalent to:
collection.Find(Builders<MyObject>.Filter.Eq(x => x.a, "blah")).ToEnumerable().ToList();
* The docs state that:
Only LINQ queries that can be translated to an equivalent MongoDB query are supported. If you write a LINQ query that can’t be translated you will get a runtime exception and the error message will indicate which part of the query wasn’t supported.

Nested Linq query returns three times the result?

I am trying to work with dynamic data and running into some odd things with LINQ that I can't find much information online. I want to point out that this issue I run into happens on any nested collection.
I want to take a collection of dynamic data, then filter it with a where query. That where query simply checks all the values to see if it contains "FL" and then I want it to return the dynamic collection... not just the fields that contain FL.
I've explicitly put in the type in the where clause to make it easier to read online, it is redundant otherwise.
IEnumerable<dynamic> query = from agent in agentRecords
from values in (ExpandoObject)agent
where ((KeyValuePair<string, object>)values).Value.ToString().Contains("FL")
select agent;
The query works, but returns 3 times the expected result.(I get 9 agents instead of 3, multiple duplicates.)
I am able to filter it by calling distinct, but something tells me I am not doing this right.
The other way to do this is by using LINQ extension methods
var result = agentRecords.Cast<ExpandoObject>().Where(x => x.Any(y => y.Value.ToString().Contains("FL")));
According to https://learn.microsoft.com/en-us/dotnet/csharp/linq/query-expression-basics, there are multiple examples of "multiple/nested from" linq queries and it doesn't seem to run into this duplicate result problem... what am I overlooking?

Instead of cross joining each agent with its collection of values, test each agent once:
IEnumerable<dynamic> query = from agent in agentRecords
where (from values in (ExpandoObject)agent
select ((KeyValuePair<string, object>)values).Value.ToString().Contains("FL")).Any()
select agent;
Lambda syntax does seem clearer to me, which looks to be identical to your expression:
IEnumerable<dynamic> query2 = agentRecords.Where(agent => ((ExpandoObject)agent).Any(((KeyValuePair<string, object>)values).Value.ToString().Contains("FL")));

from a in agentRecords
where (from i in (ExpandoObject)a
where (((KeyValuePair<string, object>)i).Value.ToString().Contains("FL")
select i).Count() > 0
select a;

Linq query timing out, how to streamline query

Our front end UI has a filtering system that, in the back end, operates over millions of rows. It uses a an IQueryable that is built up over the course of the logic, then executed all at once. Each individual UI component is ANDed together (for example, Dropdown1 and Dropdown2 will only return rows that have both of what is selected in common). This is not a problem. However, Dropdown3 has has two types of data in it, and the checked items need to be ORd together, then ANDed with the rest of the query.
Due to the large amount of rows it is operating over, it keeps timing out. Since there are some additional joins that need to happen, it is somewhat tricky. Here is my code, with the table names replaced:
//The end list has driver ids in it--but the data comes from two different places. Build a list of all the driver ids.
driverIds = db.CarDriversManyToManyTable.Where(
cd =>
filter.CarIds.Contains(cd.CarId) && //get driver IDs for each car ID listed in filter object
).Select(cd => cd.DriverId).Distinct().ToList();
driverIds = driverIds.Concat(
db.DriverShopManyToManyTable.Where(ds => filter.ShopIds.Contains(ds.ShopId)) //Get driver IDs for each Shop listed in filter object
.Select(ds => ds.DriverId)
.Distinct()).Distinct().ToList();
//Now we have a list solely of driver IDs
//The query operates over the Driver table. The query is built up like this for each item in the UI. Changing from Linq is not an option.
query = query.Where(d => driverIds.Contains(d.Id));
How can I streamline this query so that I don't have to retrieve thousands and thousands of IDs into memory, then feed them back into SQL?

There are several ways to produce a single SQL query. All they require to keep the parts of the query of type IQueryable<T>, i.e. do not use ToList, ToArray, AsEnumerable etc. methods that force them to be executed and evaluated in memory.
One way is to create Union query containing the filtered Ids (which will be unique by definition) and use join operator to apply it on the main query:
var driverIdFilter1 = db.CarDriversManyToManyTable
.Where(cd => filter.CarIds.Contains(cd.CarId))
.Select(cd => cd.DriverId);
var driverIdFilter2 = db.DriverShopManyToManyTable
.Where(ds => filter.ShopIds.Contains(ds.ShopId))
.Select(ds => ds.DriverId);
var driverIdFilter = driverIdFilter1.Union(driverIdFilter2);
query = query.Join(driverIdFilter, d => d.Id, id => id, (d, id) => d);
Another way could be using two OR-ed Any based conditions, which would translate to EXISTS(...) OR EXISTS(...) SQL query filter:
query = query.Where(d =>
db.CarDriversManyToManyTable.Any(cd => d.Id == cd.DriverId && filter.CarIds.Contains(cd.CarId))
||
db.DriverShopManyToManyTable.Any(ds => d.Id == ds.DriverId && filter.ShopIds.Contains(ds.ShopId))
);
You could try and see which one performs better.

The answer to this question is complex and has many facets that, individually, may or may not help in your particular case.
First of all, consider using pagination. .Skip(PageNum * PageSize).Take(PageSize) I doubt your user needs to see millions of rows at once in the front end. Show them only 100, or whatever other smaller number seems reasonable to you.
You've mentioned that you need to use joins to get the data you need. These joins can be done while forming your IQueryable (entity framework), rather than in-memory (linq to objects). Read up on join syntax in linq.
HOWEVER - performing explicit joins in LINQ is not the best practice, especially if you are designing the database yourself. If you are doing database first generation of your entities, consider placing foreign-key constraints on your tables. This will allow database-first entity generation to pick those up and provide you with Navigation Properties which will greatly simplify your code.
If you do not have any control or influence over the database design, however, then I recommend you construct your query in SQL first to see how it performs. Optimize it there until you get the desired performance, and then translate it into an entity framework linq query that uses explicit joins as a last resort.
To speed such queries up, you will likely need to perform indexing on all of the "key" columns that you are joining on. The best way to figure out what indexes you need to improve performance, take the SQL query generated by your EF linq and bring it on over to SQL Server Management Studio. From there, update the generated SQL to provide some predefined values for your #p parameters just to make an example. Once you've done this, right click on the query and either use display estimated execution plan or include actual execution plan. If indexing can improve your query performance, there is a pretty good chance that this feature will tell you about it and even provide you with scripts to create the indexes you need.

It looks to me that using the instance versions of the LINQ extensions is creating several collections before you're done. using the from statement versions should cut that down quite a bit:
driveIds = (from var record in db.CarDriversManyToManyTable
where filter.CarIds.Contains(record.CarId)
select record.DriverId).Concat
(from var record in db.DriverShopManyToManyTable
where filter.ShopIds.Contains(record.ShopId)
select record.DriverId).Distinct()
Also using the groupby extension would give better performance than querying each driver Id.

Linq: Method has no supported translation to SQL - but how to dump to memory?

I have a Linq query that reads from a SQL table and 1 of the fields it returns are from a custom function (in C#).
Something like:
var q = from my in MyTable
select new
{
ID = my.ID,
Amount = GetAmount(ID)
};
If I do a q.Dump() in LinqPad, it shows the results, which tells me that it runs the custom function without trying to send it to SQL.
Now I want to union this to another query, with:
var q1 = (from p in AnotherQuery.Union(q)...
and the I get the error that Method has no supported translation to SQL.
So, my logic tells me that I need to dump q in memory and then try to union to that. I've tried doing that with ToList() and creating a secondary query that populates itself from the List, but that leads to a long list of different errors. Am I on the right track, by trying to get q in memory and union on that, or are there better ways of doing this?

You can't use any custom functions in a LINQ query that gets translated - only the functions supported by the given LINQ provider. If you want your query to happen on the server, you need to stick with the supported functions (even if it sometimes means having to inline code that would otherwise be reused).
The difference between your two queries boils down to when (and where) the projection happens. In your first case, the data from MyTable is returned from the DB - in your sample, just the ID. Then, the projection happens on top of this - the GetAmount method is called in your application for each of ID.
On the other hand, there's no such way for this to happen in your second query, since you're not using GetAmount in the final projection.
You either need to replace the custom function with inlined query the provider understands, or refactor all your queries to use the supported functions in addition with whatever you need to do in-memory. There's no point in giving you any sample code, since it depends entirely on your actual query, and what you're really trying to query for.

IEnumerable.Except wont work, so what do I do?

I have a linq to sql database. Very simplified we have 3 tables, Projects and Users. There is a joining table called User_Projects which joins them together.
I already have a working method of getting IEnumberable<Project> for a given user.
from up in User_Projects
select up.Project;
Now I want to get the projects the user isn't involved with. I figured the except method of IEnumerable would be pretty good here:
return db.Projects.Except(GetProjects());
That compiles, however I get a runtime error: "Local sequence cannot be used in LINQ to SQL implementation of query operators except the Contains() operator."
Is there any way to get around this?
Update:
A few views but no answers :p
I have tried this:
IEnumerable<Project> allProjects = db.Projects;
IEnumerable<Project> userProjects = GetProjects();
return allProjects.Except(GetProjects());
I know it's essentially the same as the original statement - but now i dont get a runtime error. Unfortunately, it doesn't really do the except part and just returns all the projects, for some reason

Linq to Sql doesn't understand how to work with an arbitrary in-memory sequence of objects. You need to express this in relational terms, which works on IDs:
var userProjectIds =
from project in GetProjects()
select project.ProjectId;
var nonUserProjects =
from project in db.Projects
where !userProjectIds.Contains(project.ProjectId)
select project;

You could try something simple like
User u = SomeUser;
from up in User_Projects
where up.User != u
select up.Project;

try this:
var userProjects = GetProjects();
return db.Projects.Except(userProjects.ToArray());
The ToArray should force evaluation of the sequence (if I'm understanding the issue right) and allow the operation to succeed.