Query multiple tables c# linq efficiently - c#

I have a database with about 20 table, all of them have some columns which are same, eg, name, cost, year manufactured etc. I need to query those tables. what is best, most efficient way. tables and data have to stay as they are.
Here is what I am doing right now.
var table1 = (from a in _entities.ta1
join b in _entities.subTa on a.tid equals b.Id
select new
{
id = a.id,
name = a.name,
type = a.type,
}).ToList();
var table2 = (from a in _entities.ta2
join b in _entities.subTa2 on a.tid equals b.Id
select new
{
id = a.id,
name = a.name,
type = a.type,
}).ToList();
var abc = table1;
abc.AddRange(table2)
var list = new List<MyClass>();
foreach(var item in abc)
{
var classItem = new MyClass();
classItem.id = item.id;
classItem.name = item.name;
classItem.type = item.type;
list.Add(classItem);
}
return list;
This needs to be done to many table, which is not very efficient coding.
How can I improve this code?


You could use that, assuming that data types for all columns do match.
List<MyClass> list =
( //table1
from a in _entities.ta1
join b in _entities.subTa
on a.tid equals b.Id
select new MyClass()
{
id = a.id,
name = a.name,
type = a.type
})
.Concat(( //table2 (use .Union instead of .Concat if you wish to eliminate duplicate rows)
from a in _entities.ta2
join b in _entities.subTa2
on a.tid equals b.Id
select new MyClass()
{
id = a.id,
name = a.name,
type = a.type
}
)).ToList();
return list;
The reason why this could be more efficient is that it
groups all queries into one database query, thus causes less database traffic,
lets all the computation happen on the database server, therefore eliminating duplicate computation in the OP code, and
optimizes memory usage as it does NOT create a list of anonymous objects and then adding them to a new List<MyClass>.


This is the answer i was looking or. this is not ideal but it got the job done
var tables= new Dictionary<string, string>();
tables.Add("table1", "subTable1");
tables.Add("table2", "subTable2");
foreach (var table in tables)
{
var tableName = table.Key;
var subName= table.Value;
var data = _entities.Database.SqlQuery<MyClass>($#"select
a.Id,a.Name,b.subName from {tableName} a left join {subName} b on
a.subId=b.Id").ToList();
}
Thank you everyone for your contribution

Related

How to return nested object using Generic method and Dapper [duplicate]

I'm currently using Entity Framework for my db access but want to have a look at Dapper. I have classes like this:
public class Course{
public string Title{get;set;}
public IList<Location> Locations {get;set;}
...
}
public class Location{
public string Name {get;set;}
...
}
So one course can be taught at several locations. Entity Framework does the mapping for me so my Course object is populated with a list of locations. How would I go about this with Dapper, is it even possible or do I have to do it in several query steps?

Alternatively, you can use one query with a lookup:
var lookup = new Dictionary<int, Course>();
conn.Query<Course, Location, Course>(#"
SELECT c.*, l.*
FROM Course c
INNER JOIN Location l ON c.LocationId = l.Id
", (c, l) => {
Course course;
if (!lookup.TryGetValue(c.Id, out course))
lookup.Add(c.Id, course = c);
if (course.Locations == null)
course.Locations = new List<Location>();
course.Locations.Add(l); /* Add locations to course */
return course;
}).AsQueryable();
var resultList = lookup.Values;
See here https://www.tritac.com/blog/dappernet-by-example/

Dapper is not a full blown ORM it does not handle magic generation of queries and such.
For your particular example the following would probably work:
Grab the courses:
var courses = cnn.Query<Course>("select * from Courses where Category = 1 Order by CreationDate");
Grab the relevant mapping:
var mappings = cnn.Query<CourseLocation>(
"select * from CourseLocations where CourseId in #Ids",
new {Ids = courses.Select(c => c.Id).Distinct()});
Grab the relevant locations
var locations = cnn.Query<Location>(
"select * from Locations where Id in #Ids",
new {Ids = mappings.Select(m => m.LocationId).Distinct()}
);
Map it all up
Leaving this to the reader, you create a few maps and iterate through your courses populating with the locations.
Caveat the in trick will work if you have less than 2100 lookups (Sql Server), if you have more you probably want to amend the query to select * from CourseLocations where CourseId in (select Id from Courses ... ) if that is the case you may as well yank all the results in one go using QueryMultiple

No need for lookup Dictionary
var coursesWithLocations =
conn.Query<Course, Location, Course>(#"
SELECT c.*, l.*
FROM Course c
INNER JOIN Location l ON c.LocationId = l.Id
", (course, location) => {
course.Locations = course.Locations ?? new List<Location>();
course.Locations.Add(location);
return course;
}).AsQueryable();

I know I'm really late to this, but there is another option. You can use QueryMultiple here. Something like this:
var results = cnn.QueryMultiple(#"
SELECT *
FROM Courses
WHERE Category = 1
ORDER BY CreationDate
;
SELECT A.*
,B.CourseId
FROM Locations A
INNER JOIN CourseLocations B
ON A.LocationId = B.LocationId
INNER JOIN Course C
ON B.CourseId = B.CourseId
AND C.Category = 1
");
var courses = results.Read<Course>();
var locations = results.Read<Location>(); //(Location will have that extra CourseId on it for the next part)
foreach (var course in courses) {
course.Locations = locations.Where(a => a.CourseId == course.CourseId).ToList();
}

Sorry to be late to the party (like always). For me, it's easier to use a Dictionary, like Jeroen K did, in terms of performance and readability. Also, to avoid header multiplication across locations, I use Distinct() to remove potential dups:
string query = #"SELECT c.*, l.*
FROM Course c
INNER JOIN Location l ON c.LocationId = l.Id";
using (SqlConnection conn = DB.getConnection())
{
conn.Open();
var courseDictionary = new Dictionary<Guid, Course>();
var list = conn.Query<Course, Location, Course>(
query,
(course, location) =>
{
if (!courseDictionary.TryGetValue(course.Id, out Course courseEntry))
{
courseEntry = course;
courseEntry.Locations = courseEntry.Locations ?? new List<Location>();
courseDictionary.Add(courseEntry.Id, courseEntry);
}
courseEntry.Locations.Add(location);
return courseEntry;
},
splitOn: "Id")
.Distinct()
.ToList();
return list;
}

Something is missing. If you do not specify each field from Locations in the SQL query, the object Location cannot be filled. Take a look:
var lookup = new Dictionary<int, Course>()
conn.Query<Course, Location, Course>(#"
SELECT c.*, l.Name, l.otherField, l.secondField
FROM Course c
INNER JOIN Location l ON c.LocationId = l.Id
", (c, l) => {
Course course;
if (!lookup.TryGetValue(c.Id, out course)) {
lookup.Add(c.Id, course = c);
}
if (course.Locations == null)
course.Locations = new List<Location>();
course.Locations.Add(a);
return course;
},
).AsQueryable();
var resultList = lookup.Values;
Using l.* in the query, I had the list of locations but without data.

Not sure if anybody needs it, but I have dynamic version of it without Model for quick & flexible coding.
var lookup = new Dictionary<int, dynamic>();
conn.Query<dynamic, dynamic, dynamic>(#"
SELECT A.*, B.*
FROM Client A
INNER JOIN Instance B ON A.ClientID = B.ClientID
", (A, B) => {
// If dict has no key, allocate new obj
// with another level of array
if (!lookup.ContainsKey(A.ClientID)) {
lookup[A.ClientID] = new {
ClientID = A.ClientID,
ClientName = A.Name,
Instances = new List<dynamic>()
};
}
// Add each instance
lookup[A.ClientID].Instances.Add(new {
InstanceName = B.Name,
BaseURL = B.BaseURL,
WebAppPath = B.WebAppPath
});
return lookup[A.ClientID];
}, splitOn: "ClientID,InstanceID").AsQueryable();
var resultList = lookup.Values;
return resultList;

There is another approach using the JSON result. Even though the accepted answer and others are well explained, I just thought about an another approach to get the result.
Create a stored procedure or a select qry to return the result in json format. then Deserialize the the result object to required class format. please go through the sample code.
using (var db = connection.OpenConnection())
{
var results = await db.QueryAsync("your_sp_name",..);
var result = results.FirstOrDefault();
string Json = result?.your_result_json_row;
if (!string.IsNullOrEmpty(Json))
{
List<Course> Courses= JsonConvert.DeserializeObject<List<Course>>(Json);
}
//map to your custom class and dto then return the result
}
This is an another thought process. Please review the same.

Too much data in Contains (Linq): How to increase performace

I have a linq query like this:
from a in context.table_A
join b in
(from temp in context.table_B
where idlist.Contains(temp.id)
select temp)
on a.seq_id equals b.seq_id into c
where
idlist.Contains(a.id)
select new MyObject
{
...
}).ToList();
idlist is List
The problem I have is that the idlist has too many values (hundreds of thousands to several million records). It works fine with few records but when there are too many records, the contains function is error.
Error log is
The query plan cannot be created due to lack of internal resources of
the query processor. This is a rare event and only occurs for very
complex queries or queries that reference a very large number of
tables or partitions. Make the query easy.
I want to improve the performance of this section. Any ideas?

I would suggest to install extension linq2db.EntityFrameworkCore and use temporary tables with fast BulkCopy
public class IdItem
{
public int Id { get; set; }
}
...
var items = idList.Select(id => new IdItem { Id = id });
using var db = context.CreateLinqToDBConnection();
using var temp = db.CreateTempTable("#IdList", items);
var query =
from a in context.table_A
join id1 in temp on a.Id equals id1.Id
join b in context.table_B on a.seq_id equals b.seq_id
join id2 in temp on b.Id equals id2.Id
select new MyObject
{
...
};
// switch to alternative translator
var query = query.ToLinqToDB();
var result = query.ToList();

LINQ Join gives only first result multiple times

I have this Join :
var mycust= db.CUSTOMER.Where(x => x.NAME.Contains(nameid)).ToList();
var CancCustomer = (from cust in myCust
join ip in db.IPS on must.ID equals ip.CUSTOMER_ID
select new JoinObj {ID = cust.ID, NAME = cust.NAME, LASTNAME = cust.LASTNAME,
TYPE_ID = ip.TYPE_ID, TYPE2_ID = ip.TYPE2_ID,
SERVICE_ID = ip.SERVICE_ID , REASON = ip.REASON }).ToList();
This code returns the first linq result multiple times? What am I missing? Thanks.

Instead of Where, you should use SingleOrDefault/Single - these would indeed return a single row into your mycust variable.
SingleOrDefault would put a null into the variable if no such customers were found (and assuming a Customer is a reference type - if it were a value type, it would be the default value for the type). Single would throw an exception if no items were found or more than one were found, which could be very useful in finding errors in your data (such as duplicate customer records).
Additionally, it is likely your ip table has multiple matching records for a customer - which is why you would be seeing multiple records being returned from your select.

You have multiple Duplicate Record received Then Try For following quires
var mycust= db.CUSTOMER.Where(x => x.NAME.Contains(nameid)).ToList();
var CancCustomer = (from cust in myCust
join ip in db.IPS on cust.ID equals ip.CUSTOMER_ID
select new JoinObj {ID = cust.ID, NAME = cust.NAME, ASTNAME=cust.LASTNAME, TYPE_ID = ip.TYPE_ID, TYPE2_ID = ip.TYPE2_ID, SERVICE_ID = ip.SERVICE_ID , REASON = ip.REASON }).distinct().ToList();
Other Wise Multiple record then You Get One Record for You following Query
var mycust= db.CUSTOMER.Where(x => x.NAME.Contains(nameid)).ToList();
var CancCustomer = (from cust in myCust
join ip in db.IPS on must.ID equals ip.CUSTOMER_ID
select new JoinObj {ID = cust.ID, NAME = cust.NAME, LASTNAME = cust.LASTNAME, TYPE_ID = ip.TYPE_ID, TYPE2_ID = ip.TYPE2_ID, SERVICE_ID = ip.SERVICE_ID , REASON = ip.REASON }).distinct().FirstOrDefault();

How to assign values to a list by linq without for or foreach statements?

I want to compare two lists and assign 1st list to another in case of requirement.
var getdetail=_readonlyservice.getdetail().ToList();
foreach(var item in docdetail)
{
var temp=getdetail.firstordefualt(i=>i.Id=item.Id)
if(temp==null) continue;
item.code=temp.code;
}
I want to implement top statements in linq .any help ?

Think so..
var getdetail=_readonlyservice.getdetail().ToList();
var tempList = from dd in context.docdetail
join g in context.getdetail on dd.Id equals g.Id
select new // Your type
{
// Columns...
Code = g.Code
}

I believe you are trying to do like the way I did, although I was going to join table.
var result = (from e in DSE.employees
join d in DSE.departments on e.department_id equals d.department_id
join ws in DSE.workingshifts on e.shift_id equals ws.shift_id
select new
{
FirstName = e.FirstName,
LastName = e.LastName,
Gender = e.Gender,
Salary = e.Salary,
Department_id = e.department_id,
Department_Name = d.department_name,
Shift_id = ws.shift_id,
Duration = ws.duration,
}).ToList();
// TODO utilize the above result
I was using DTO method to do this. And then you return result(as this case is result).
You may view the whole question and solution here.
As this case, you are not required to put foreach loop, as the query said from every row in yourdatabase.table

Including objects from join in query result

Imagine a simple Entity Framework query with a context generated from the database such as:
var q = from a in context.Accounts
join c in context.Contracts
on a.Id equals c.AccountId
select new CustomAccount {
Id = a.Id,
Name = a.Name,
...
Contracts = //How do I easily populate the related contracts?
};
The query looks for accounts and joins to contracts. The relationship is not enforced in the database (I can't change the schema) so I can't use navigational properties. Is there an easy way that I can populate the related objects?

Just use a group by clause. Something like this (untested):
var q = from a in context.Accounts
join c in context.Contracts on a.Id equals c.AccountId
group a by new { a.Id, a.Name, a.Etc } into g
select new CustomAccount
{
Id = g.Key.Id,
Name = g.Key.Name,
Etc = g.Key.Etc,
Contracts = g.SelectMany(x => x.Contracts)
};

I'm not sure if I understand correctly but you can execute a query that return anonymous type objects
EDIT : you can create a custom class to hold the data member of your result and return in linq result.
EDIT : using group by on account name (e.g)
var q = from a in context.Accounts
join c in context.Contracts
on a.Id equals c.AccountId
group a by new { a.Name } into g
select new AccountContracts
{
AccountName = g.Key.Id, // Account name
Contracts = g.SelectMany(x => x.Contracts)
};

Categories

Resources