I have a object of MyFriendFollowStatus , for each friend I need information from 2 different databases, so I wrote something like this
db1Context FFdb = new db1Context();
db2Context EEdb = new db2Context();
foreach (fbFriendsFollowStatus a in fbids)
{
long ffID = FFdb.FFUsers.Where(x => x.FacebookID == a.fbid).FirstOrDefault().FFUserID;
a.ffID = ffID;
int status = EEdb.StatusTable(x => x.ffID == ffid).FirstOrDefault().Status;
a.Status = status;
}
this works, but it doesnt really seem right calling 2 databases - once each for each user , is there something built in LinqToSql that helps with something like this? or sometype of join I can use using 2 different databases?
Well, you can always limit your N+1 query problem to 3 queries - one to get users, one to get user's data form first database and one for the second database. Then connect all the results in memory - this will limit the connections to databases which should improve performance of your application.
I don't know if linq-to-sql or entity framework offers building model from different databases - this would pose some performance problems probably - like in includes or something, but I may simply not be aware of such features.
Sample code to do what you're trying to achieve would look something like that:
var facebookIds = fbFriendsFollowStatus.Select(a => a.fbid);
var FFUserIds= FFdb.FFUsers.Where(x => facebookIds.Contains(x.FacebookID)).Select(x => new { x.FacebookID, x.FFUserId)
var statuses = EEdb.StatusTable.Where(x => FFUserIds.Contains(x.ffID)).Select(x => new { x.ffID, x.Status})
And then some simple code to match results in memory - but that will be simple.
Please note that this code is sample - if I've mismatched some ids or something, but idea should be clear.
Related
This may be a silly question, but I have fluent linq query that is pulling data off a DB, but, of course, it processes the data in memory, not on the DB.
Is there a means of taking a query (Say table.Where(t => t. id > 22)) and having it run as a db query (i.e. SELECT * FROM TABLE WHERE ID > 22).
I want to use the fluent style because I am building up the query from various places. I know I can make this specific query into query syntax, but my actual queries are more complex.
I have tried standard searches, but I suspect it is not going to be possible, as nothing comes up.
Yes using EF Core.
The code is not necessarily clear:
var getTableData = table.Where(base.whereSelection);
whereSelection.ForEach(w => getTableData = getTableData.Where(w));
At the moment, all of the Where Selections are Funcs - they could be converted to something else (like Expressions) if I knew that would make them run on the DB. The rest of the code is also building up this set of details programatically:
var selectedData = getTableData.Select(Convert);
var sortedTableData = GetSort(selectedData, Ordering);
var sorted = PickDataFrom(sortedTableData);
return CheckAndTake(sorted, pageSize);
Everything seems to work. Just in the wrong place.
So I have program to deal with some sort of work in MySQL database. I'm connecting through SSH tunneling with putty (yes, I know launching programs on server itself would be much better but I don't have a choice here).
I have some problems with programs speed. I solved one by adding ".Include(table_name)" but I can't think about a way to do it here.
So purpose of this method is to clean database of unwanted, broken records. Simplified code looks like this:
using (var dbContext = new MyDatabase_dataEntities())
{
List<achievements> achiList = new List<achievements>();
var achievementsQuery = from data in dbContext.achievements
orderby data.playerID
select data;
achiList = achievementsQuery.Skip(counter * 5000).Take(5000).ToList();
foreach (achievements record in achiList)
{
var playerExists = from data in dbContext.players_data
where data.playerID == record.playerID
select data;
if(!playerExists.Any())
{
dbContext.achievements.Remove(record);
}
}
dbContext.SaveChanges();
counter++;
}
So this is built this way because I want to load achievements table then check if achievements have their player in player_data. If such player doesn't exist, remove his achievements.
It is all in do while, so I don't overload my memory by loading all data at once.
I know the problem is with checking in database in foreach steps but I can't figure out how to do it without it. Other things I tried generated errors either because EF couldn't translate it into SQL or exceptions were thrown when trying to access non-existing entity. Doing it in foreach bottlenecks whole program probably because of ping to the server.
I will need similar thing more often so I would be really gratefull if anyone could help me with making it so there is no need to call to database in "foreach". I know I could try to load whole players_data table and then check for Any(), but some tables I need it on are too big for that.
Oh, and turning off tracking changes doesn't help at this point because it is not what slows the program.
I would be gratefull for any help, thanks in advance!
EDIT: Mmmm, is there a way to get achievements which doesn't have player_data corresponding to them through one query using associations? Like adding to achievements query something like:
where !data.player_data.Exists()
Intellisense shows me that there is nothing like Exists or Any to use at this point. Is there any trick similar to this? It would definitely deal with the problem I have with speed there since database call in foreach wouldn't be needed.
If you want to delete achievements that don't have corresponding user records, then you can user a SQL query below:
DELETE a
FROM `achievements` a
LEFT JOIN `user` AS u
ON u.`playerID` = a.`playerID`
WHERE u.`playerID` IS NULL;
SQL query will be an order of magnitude faster than Entity Framework.
If you want to do that in the application, you can use the following code that uses LINQ to Entities and LINQ extensions methods. I assume you have a foreign key for player_data in achievements table so Entity Framework generates player_data lazy property for your achievements entity:
using (var dbContext = new MyDatabase_dataEntities())
{
var proceed = true;
while (proceed)
{
// Get net 1000 entities to delete
var entitiesToDelete = dbContext.achievements
.Where(x => x.players_data == null)
.Take(1000)
.ToList();
dbContext.achievements.RemoveRange(entitiesToDelete);
dbContext.SaveChanges();
// Proceed if deleted entities during this iteration
proceed = entitiesToDelete.Count() > 0;
}
}
If you prefer to use LINQ query syntax instead of extension methods, then your code will look like:
using (var dbContext = new MyDatabase_dataEntities())
{
var proceed = true;
while (proceed)
{
// Get net 1000 entities to delete
var query = from achievement in dbContext.achievements
where achievement.players_data == null
select achievement;
var entitiesToDelete = query.ToList();
dbContext.achievements.RemoveRange(entitiesToDelete);
dbContext.SaveChanges();
// Proceed if deleted entities during this iteration
proceed = entitiesToDelete.Count() > 0;
}
}
As the title says, I´m using entity framework 4.0 for a financial application. I have a winform where I list all the cheques (checks) I have. But in that form, the user can specify some filters.
If the user does not apply any filter, I just can make the query like this:
lista_cheques = db.Cheque.Include("Operacion").Include("Cliente").ToList();
datagridview.Datasource = lista_cheques;
That is simple. But when it applies filters, the problem gets bigger.
As you see, the user can use filter to see cheques (checks) of a specific client, dates, bank, CUIT number, check state, etc.
Now, my question is related to performance in the queries.
I was thinking on doing the filters separeted, like this:
lista_cheques = db.Cheque.Include("Operacion").Include("Cliente").Where(x => x.fecha_deposito == fecha).ToList();
lista_cheques = lista_cheques.Where(x => x.banco.id_banco = banco).ToList();
lista_cheques = lista_cheques.Where(x => x.Operacion.Cliente.id_cliente = id_cliente).ToList();
Translation:
fecha is date
Operacion is a group of checks
Cliente is client.
In this way, I´m doing a query, then, a query from that query result, then a new query from that new result and that goes on.
I think this way might have a big performance issues. I know that SQL server optimize queries. So, if I´m doing fragmented queries, the optimizer is not working properly.
The other way I thought about but it´s very tedious, is to create one big query to handle every possible filter selection.
For example, the other example would be like this:
lista_cheques = db.Cheque.Include("Operacion").Include("Cliente").Where(x => x.fecha_deposito == fecha && x.banco.id_banco = banco && x.Operacion.Cliente.id_cliente = id_cliente).ToList();
The big problem is that I will need lots of combinations to be able to handle all the filter posibilities.
Ok guys, now, will I have performace issues in the first code example? I doing there one big query to the database, and then I´m doing the query in the list of objects (which I think will be faster). I´m pretty new to this ORM, and this listing will have to handle a lot of registries..
I somebody can give me some advice? I made pretty much a mess explaining, hope you can understand..
lista_cheques = db.Cheque.Include("Operacion").Include("Cliente").Where(x => x.fecha_deposito == fecha).ToList();
lista_cheques = lista_cheques.Where(x => x.banco.id_banco = banco).ToList();
lista_cheques = lista_cheques.Where(x => x.Operacion.Cliente.id_cliente = id_cliente).ToList();
Nearly perfect. Kill all those ToList and it is good.
The ToList means SQL is evaluated, so if all 3 flters trigger, 2 and 3 are evaluated in memory.
Kick the ToList away, and the different Where clauses get combined on the database.
Standard LINQ 101. Works like a charm and is always nice to see.
Then add as LAST line:
lista_cheques = lista_cheques.ToList ();
I could use some help with a RIA services query that I have been struggling with for a bit now...
Person has a collection of DailyScore.
I want to be able to produce a query that is filtered on both Person and DailyScore.
DateTime start = ...
DateTime end = ...
var personQuery = context.GetPersonQuery().Skip.(10).Take(10)
var scoresQuery = from s in ctx.GetDailyScoresForPeriod(start, end)
where personQuery.Contains(s.Person)
select s;
...load the scoresQuery
What I would ideally want is the Person objects loaded with the appropriate DailyScores for the specified period.n We can assume all the 'includes' are in place too.
Is this possible in RIA Services? If so, can somebody give me the correct way of going about it, and if not are there any work arounds?
I feel like this is a simple request, so hopefully this is easy.
Thanks is advance,
Shane.
Also, can someone point me to a good online resource about the rules around using EntityQueries on the client side with RIA Services? I have struggled to find anything like this.
Actually, the query you are trying to do is not supported. You can't use Contains, Any, and similar constructs where you need to filter data in a collection.
In these scenarios you have two options:
Perform both queries separately, and then filter the data on the client. This approach has performance issues, because you are retrieving all the data from the server, and then filtering it out. In your case, you would something like this:
var personQuery = context.GetPersonQuery().Skip.(10).Take(10);
var scoresQuery = context.GetDailyScoresForPeriodQuery(start, end);
// Load both queries and then:
scores.Where(s => persons.Contains(s.Person))
Perform the query in the server. This approach is more performant, but you need to define a way to filter the persons in the same query. In your case, if you only want the persons whose ids fall in a specific range for example, you should add this method to your domain context (in the server):
IQueryable<DailyScore> GetDailyScoresForPeriodForPersons(DateTime start, DateTime end, int firstPerson, int lastPerson)
{
return context
.DailyScores
.Where(s => s.Start > start && s.End < end)
.Where(s => s.Person.ID >= firstPerson && s.Person.ID < lastPerson)
}
Then in your client, you would call it like this:
var scoresQuery = context.GetDailyScoresForPeriodForPersons(start, end, 10, 20)
Some resources:
Domain Context and Operations
Walkthrough: Adding Query Methods
RIA Services and relational data
I would like to query to different Tables, say apples and cars, which have no relationship so that active record goes only once to the database.
Example in pseudocode:
var q1 = new Query("select * form apple");
var q2 = new Query("select * from car");
var batchQuery = new BatchQuery(q1,q2);
var result = BatchQuery.Execute(); //only one trip to the database
var apples = result[0] as IEnumerable<Apple>;
var cars = result[1] as IEnumerable<Car>;
I have tried ActiveRecordMultiQuery, but there all Queries need to query the same table.
I don't believe there is a way to do this.
It seems like you might be going a bit overboard with optimization here: does it really make a noticeable difference to your application to make 2 separate queries? I think your time might be better spent looking for N+1 select queries elsewhere in your application instead.
If the cost of one extra query is in fact significant then you probably have an issue with the database server or the connection to it.