I am doing a search on a database table using a wildcard(The contains extension) Here is the code
// This gets a list of the primary key IDs in a table that has 5000+ plus records in it
List<int> ids = context.Where(m => m.name.ToLower().Contains(searchTerm.ToLower())).Select(m => m.Id).ToList();
// Loop through all the ids and get the ones that match in a different table (So basically the FKs..)
foreach (int idin nameId)
{
total.AddRange(context2.Where(x => x.NameID == id).Select(m => m.Id).ToList());
}
In there anything I could change in the LINQ that would result in getting the IDs faster?
Thanks
I have not tested it, but you could do something along these lines:
var total =
from obj2 in context2
join obj1 in context1 on obj2.NameID equals obj1.Id
where obj1.name.ToLower().Contains(searchTerm.ToLower())
select obj2.Id
It joins the two tables, performing a cartesian product first and then limiting it to the pairs where the NameIds match (see this tutorial on the join clause). The where line does the actual filtering.
It should be faster because the whole matching is done in the database, and only the correct ids are returned.
If you had a Name property in the context2 item class that holds a reference to the context1 item, you could write it more readable:
var total = context2
.Where(x => x.Name.name.ToLower().Contains(searchTerm.toLower()))
.Select(x => x.ID);
In this case, Linq to SQL would do the join automatically for you.
In terms of performance you can see tests that show if you search 1 string in 1 000 000 entries its around 100 ms.
Here is the link with tests and implementation.
for (int y = 0; y < sf.Length; y++)
{
c[y] += ss.Where(o => o.Contains(sf[y])).Count();
}
Related
Good morning,
I'm having trouble with a EF query. This is what i am trying to do.
First i am pulling a list of ID's like so (List of IDs are found in the included x.MappingAccts entity):
Entities.DB1.Mapping mapping = null;
using (var db = new Entities.DB1.DB1Conn())
{
mapping = db.Mappings.Where(x => x.Code == code).Include(x => x.MappingAccts).FirstOrDefault();
}
Later, i'm trying to do a query on a different DB against the list of Id's i pulled above (essentially a IN clause):
using (var db = new Entities.DB2.DB2Conn())
{
var accounts = db.Accounts.Where(mapping.MappingAccts.Any(y => y.Id == ?????????)).ToList();
}
As you can see i only got part way with this.
Basically what i need to do is query the Accounts table against it's ID column and pull all records that match mapping.MappingAccts.Id column.
Most of the examples i am finding explain nicely how to do this against a single dimension array but i'm looking to compare specific columns.
Any assist would be awesome.
Nugs
An IN clause is generated using a IEnumerable.Contains.
From the first DB1 context, materialize the list of Id's
var idList = mapping.MappingAccts.Select(m => m.Id).ToList();
Then in the second context query against the materialized list of id's
var accounts = db.Accounts
.Where(a => idList.Contains(a.Id))
.ToList();
The only problem you may have is with the amount of id's you are getting in the first list. You may hit a limit with the SQL query.
This will give the list of Accounts which have the Ids contained by MappingAccts
using (var db = new Entities.DB2.DB2Conn())
{
var accounts = db.Accounts.Where(s => mapping.MappingAccts.Any(y => y.Id == s.Id)).ToList();
}
I have a database that contains 3 tables:
Phones
PhoneListings
PhoneConditions
PhoneListings has a FK from the Phones table(PhoneID), and a FK from the Phone Conditions table(conditionID)
I am working on a function that adds a Phone Listing to the user's cart, and returns all of the necessary information for the user. The phone make and model are contained in the PHONES table, and the details about the Condition are contained in the PhoneConditions table.
Currently I am using 3 queries to obtain all the neccesary information. Is there a way to combine all of this into one query?
public ActionResult phoneAdd(int listingID, int qty)
{
ShoppingBasket myBasket = new ShoppingBasket();
string BasketID = myBasket.GetBasketID(this.HttpContext);
var PhoneListingQuery = (from x in myDB.phoneListings
where x.phonelistingID == listingID
select x).Single();
var PhoneCondition = myDB.phoneConditions
.Where(x => x.conditionID == PhoneListingQuery.phonelistingID).Single();
var PhoneDataQuery = (from ph in myDB.Phones
where ph.PhoneID == PhoneListingQuery.phonePageID
select ph).SingleOrDefault();
}
You could project the result into an anonymous class, or a Tuple, or even a custom shaped entity in a single line, however the overall database performance might not be any better:
var phoneObjects = myDB.phoneListings
.Where(pl => pl.phonelistingID == listingID)
.Select(pl => new
{
PhoneListingQuery = pl,
PhoneCondition = myDB.phoneConditions
.Single(pc => pc.conditionID == pl.phonelistingID),
PhoneDataQuery = myDB.Phones
.SingleOrDefault(ph => ph.PhoneID == pl.phonePageID)
})
.Single();
// Access phoneObjects.PhoneListingQuery / PhoneCondition / PhoneDataQuery as needed
There are also slightly more compact overloads of the LINQ Single and SingleOrDefault extensions which take a predicate as a parameter, which will help reduce the code slightly.
Edit
As an alternative to multiple retrievals from the ORM DbContext, or doing explicit manual Joins, if you set up navigation relationships between entities in your model via the navigable join keys (usually the Foreign Keys in the underlying tables), you can specify the depth of fetch with an eager load, using Include:
var phoneListingWithAssociations = myDB.phoneListings
.Include(pl => pl.PhoneConditions)
.Include(pl => pl.Phones)
.Single(pl => pl.phonelistingID == listingID);
Which will return the entity graph in phoneListingWithAssociations
(Assuming foreign keys PhoneListing.phonePageID => Phones.phoneId and
PhoneCondition.conditionID => PhoneListing.phonelistingID)
You should be able to pull it all in one query with join, I think.
But as pointed out you might not achieve alot of speed from this, as you are just picking the first match and then moving on, not really doing any inner comparisons.
If you know there exist atleast one data point in each table then you might aswell pull all at the same time. if not then waiting with the "sub queries" is nice as done by StuartLC.
var Phone = (from a in myDB.phoneListings
join b in myDB.phoneConditions on a.phonelistingID equals b.conditionID
join c in ph in myDB.Phones on a.phonePageID equals c.PhoneID
where
a.phonelistingID == listingID
select new {
Listing = a,
Condition = b,
Data = c
}).FirstOrDefault();
FirstOrDefault because single throws error if there exists more than one element.
I have three tables, which two of them are in many to many relationship.
Picture:
This is the data in middle mm table:
Edit:
Got until here, I get proper 4 rows back, but they are all the same result(I know I need 4 rows back, but there are different results)
return this._mediaBugEntityDB.LotteryOffers
.Find(lotteryOfferId).LotteryDrawDates
.Join(this._mediaBugEntityDB.Lotteries, ldd => ldd.LotteryId, lot => lot.Id, (ldd, lot) =>
new Lottery
{
Name = lot.Name,
CreatedBy = lot.CreatedBy,
ModifiedOn = lot.ModifiedOn
}).AsQueryable();
My question is, how can I retrieve all the Lotteries via many to many table WHERE I have LotteryOfferId given only?
What I want to achieve is to get data from lottery table by LotteryDrawDateId.
First I use LotteryOfferId to get DrawDates from middle table, and by middle table I get drawDateIds to use them in LotteryDrawDate table. From that table I need to retreive Lottey table by LotteryId in LotteryDrawDate table.
I gain this by normal SQL(LotteryOffersLotteryDrawDates is middle table in DB, not seen in model):
select
Name, Lotteries.CreatedBy, Lotteries.ModifiedOn, count(Lotteries.Id)
as TotalDrawDates from Lotteries join LotteryDrawDates on Lotteries.Id
= LotteryDrawDates.LotteryId join LotteryOffersLotteryDrawDates on LotteryDrawDates.Id =
LotteryOffersLotteryDrawDates.LotteryDrawDate_Id
where LotteryOffersLotteryDrawDates.LotteryOffer_Id = 19 group by
Name, Lotteries.CreatedBy, Lotteries.ModifiedOn
But Linq is different story :P
I would like to do this with lambda expressions.
Thanks
db.LotteryOffer.Where(lo => lo.Id == <lotteryOfferId>)
.SelectMany(lo => lo.LotteryDrawDates)
.Select( ldd => ldd.Lottery )
.GroupBy( l => new { l.Name, l.CreatedBy, l.ModifiedOn } )
.Select( g => new
{
g.Key.Name,
g.Key.CreatedBy,
g.Key.ModifiedOn,
TotalDrawDates = g.Count()
} );
You can do this:
var query = from lo in this._mediaBugEntityDB.LotteryOffers
where lo.lotteryOfferId == lotteryOfferId
from ld in lo.LotteryDrawDates
group ld by ld.Lottery into grp
select grp.Key;
I do this in query syntax, because (in my opinion) it is easier to see what happens. The main point is the grouping by Lottery, because you get a number of LotteryDrawDates any of which can have the same Lottery.
If you want to display the counts of LotteryDrawDates per Lottery it's better to take a different approach:
from lot in this._mediaBugEntityDB.Lotteries.Include(x => x.LotteryDrawDates)
where lot.LotteryDrawDates
.Any(ld => ld.LotteryDrawDates
.Any(lo => lo.lotteryOfferId == lotteryOfferId))
select lot
Now you get Lottery objects with their LotteryDrawDates collections loaded, so afterwards you can access lottery.LotteryDrawDates.Count() without lazy loading exceptions.
In EF, if I have a list of primatives (List), "joining" that against a table is easy:
var ids = int[]{1,4,6}; //some random values
var rows = context.SomeTable.Where(r => ids.Contains(r.id))
This gets much more complicated the instant you want to join on multiple columns:
var keys = something.Select(s => new { s.Field1, s.Field2 })
var rows = context.SomeTable.Where(r => keys.Contains(r => new { s.Field1, s.Field2 })); // this won't work
I've found two ways to join it, but neither is great:
Suck in the entire table, and filtering it based on the other data. (this gets slow if the table is really large)
For each key, query the table (this gets slow if you have a decent number of rows to pull in)
Sometimes, the compromise I've been able to make is a modified #1: pulling in subset of the table based on a fairly unique key
var keys = something.Select(s => s.Field1)
var rows = context.SomeTable.Where(r => keys.Contains(s.Field1)).ToList();
foreach (var sRow in something)
{
var joinResult = rows.Where(r => r.Field1 == sRow.Field1 && r.Field2 == sRow.Field2);
//do stuff
}
But even this could pull back too much data.
I know there are ways to coax table valued parameters into ADO.Net, and ways I can build a series of .Where() clauses that are OR'd together. Does anyone have any magic bullets?
Instead of a .Contains(), how about you use an inner join and "filter" that way:
from s in context.SomeTable
join k in keys on new {k.Field1, k.Field2} equals new {s.Field1, s.Field2}
There may be a typo in the above, but you get the idea...
I got exactly the same problem, and the solutions I came up with were:
Naive: do a separate query for each local record
Smarter: Create 2 lists of unique Filed1 values and unique Fiels2 values, query using 2 contains expressions and then you will have to double filter result as they might be not that accurate.
Looks like this:
var unique1 = something.Select(x => x.Field1).Distinct().ToList();
var unique2 = something.Select(x => x.Field2).Distinct().ToList();
var priceData = rows.Where(x => unique1.Contains(x.Field1) && unique2.Contains(x.Field2));
Next one is my own solution which I called BulkSelect, the idea behind it is like this:
Create temp table using direct SQL command
Upload data for SELECT command to that temp table
Intercept and modify SQL which was generated by EF.
I did it for Postgres, but this may be ported to MSSQL is needed. This nicely described here and the source code is here
You can try flattening your keys and then using the same Contains pattern. This will probably not perform great on large queries, although you could use function indexes to store the flattened key in the database...
I have table Test with columns K1 int, K2 int, Name varchar(50)
var l = new List<Tuple<int, int>>();
l.Add(new Tuple<int, int>(1, 1));
l.Add(new Tuple<int, int>(1, 2));
var s = l.Select(k => k.Item1.ToString() + "," + k.Item2.ToString());
var q = Tests.Where(t => s.Contains(t.K1.ToString() + "," + t.K2.ToString()));
foreach (var y in q) {
Console.WriteLine(y.Name);
}
I've tested this in LinqPad with Linq to SQL
First attempt that didn't work:
I think the way to write it as a single query is something like this
var keys = something.Select(s => new { s.Field1, s.Field2 })
var rows = context.SomeTable.Where(r => keys.Any(k => r.Field1 == k.Field1 && r.Field2 == k.Field2));
Unfortunately I don't have EF on this laptop and can't even test if this is syntactically correct.
I've also no idea how performant it is if it works at all...
var rows =
from key in keys
join thingy in context.SomeTable
on 1 = 1
where thingy.Field1 == key && thingy.Field2 == key
select thingy
should work, and generate reasonable SQL
I have a following foreach statement:
foreach (var articleId in cleanArticlesIds)
{
var countArt = context.TrackingInformations.Where(x => x.ArticleId == articleId).Count();
articleDictionary.Add(articleId, countArt);
}
Database looks like this
TrackingInformation(Id, ArticleId --some stuff
Article(Id, --some stuff
what I want to do is to get all the article ids count from TrackingInformations Table.
For example:
ArticleId:1 Count:1
ArticleId:2 Count:8
ArticleId:3 Count:5
ArticleId:4 Count:0
so I can have a dictionary<articleId, count>
Context is the Entity Framework DbContext. The problem is that this solution works very slow (there are > 10k articles in db and they should rapidly grow)
Try next query to gather grouped data and them add missing information. You can try to skip Select clause, I don't know if EF can handle ToDictionary in good manner.
If you encounter Select n + 1 problem (huge amount of database requests), you can add ToList() step between Select and ToDictionary, so that all required information will be brought into memory.
This depends all your mapping configuration, environment, so in order to get good performance, you need to play a little bit with different queries. Main approach is to aggregate as much data as possible at database level with few queries.
var articleDictionary =
context.TrackingInformations.Where(trackInfo => cleanArticlesIds.Contains(trackInfo.ArticleId))
.GroupBy(trackInfo => trackInfo.ArticleId)
.Select(grp => new{grp.Key, Count = grp.Count()})
.ToDictionary(info => "ArticleId:" + info.Key,
info => info.Count);
foreach (var missingArticleId in cleanArticlesIds)
{
if(!articleDictionary.ContainsKey(missingArticleId))
articleDictionary.add(missingArticleId, 0);
}
If TrackingInformation is a navigatable property of Article, then you can do this:
var result=context.Article.Select(a=>new {a.id,Count=a.TrackingInformation.Count()});
Putting it into a dictionary is simple as well:
var result=context.Article
.Select(a=>new {a.id,Count=a.TrackingInformation.Count()})
.ToDictionary(a=>a.id,a=>a.Count);
If TrackingInforation isn't a navigatable property, then you can do:
var result=context.Article.GroupJoin(
context.TrackingInformation,
foo => foo.id,
bar => bar.id,
(x,y) => new { id = x.id, Count = y.Count() })
.ToDictionary(a=>a.id,a=>a.Count);