Develop Reference

Reconstituting nested objects with many to many join manually, is there a better way?

I have two objects: SettlementRole and SettlementType (see below for sqlite schema). There is a many to many relationship between the two, where each SettlmentRole has a collection of SettlementTypes, each with a weight specific to that pair, but each SettlementType may be in a number of SettlementRoles, each with different weights.
I'm not using an ORM for various reasons, and ORMs are not in scope here.
Question: Is there a cleaner, simpler way of doing the same process than the following? This is mostly a learning project and there are a bunch of other similar (but not identical) entity pairs/groups in the project, so I'd like to get one as clean as possible so I can follow the same process for the more complex ones.
The function to get all SettlementRoles looks like
public async Task<IList<SettlementRole>> GetAllRolesAsync()
{
using var connection = new SqliteConnection(dbName);
connection.Open();
var command = connection.CreateCommand();
command.CommandText = #"
SELECT sr.id, sr.name, sr.description, t.id, t.name, t.description, t.minSize, t.maxSize, map.weight
FROM settlementRole sr
JOIN settlementTypeRoleWeight map
ON map.roleId = sr.id
JOIN settlementType t
ON t.id = map.typeId
;";
var rawRoles = await command.ExecuteReaderAsync();
var roles = new List<SettlementRole>();
var map = new Dictionary<int, SettlementRole>();
while (rawRoles.Read())
{
var id = rawRoles.GetInt32(0);
var exists = map.TryGetValue(id, out var role);
if (!exists)
{
role = new SettlementRole() { Id = id, Name = rawRoles.GetString(1), Description = rawRoles.GetString(2) };
map.Add(id, role);
}
role!.AddType(new SettlementType()
{
Name = rawRoles.GetString(3),
Description = rawRoles.GetString(4),
MinSize = rawRoles.GetInt32(5),
MaxSize = rawRoles.GetInt32(6),
}, rawRoles.GetDouble(7));
}
return new List<SettlementRole>(map.Values);
Where settlementRole.AddType(SettlementType type, double weight) handles the internal weighting process of adding a type to the (default empty) collection.
and the schema for the three (including the join table) looks like
CREATE TABLE settlementRole(
id INTEGER NOT NULL PRIMARY KEY ASC AUTOINCREMENT,
name TEXT NOT NULL,
description TEXT NOT NULL DEFAULT ""
) ;
CREATE TABLE settlementType(
id INTEGER NOT NULL PRIMARY KEY ASC AUTOINCREMENT,
name TEXT NOT NULL,
description TEXT NOT NULL,
minSize INTEGER NOT NULL DEFAULT 1,
maxSize INTEGER,
professions TEXT NOT NULL DEFAULT ""
);
CREATE TABLE settlementTypeRoleWeight(
roleId INTEGER NOT NULL,
typeId INTEGER NOT NULL,
weight INTEGER NOT NULL DEFAULT 1,
PRIMARY KEY (roleId, typeId)
);
And the entities themselves (right now) are simple data objects (they'll have behavior, but that's out of scope here).

How can I convert SQL to lambda expressions

In my database, I created the tables structure as follows.
CREATE TABLE Course
(
Course_ID int IDENTITY(1,1) PRIMARY KEY,
Name varchar(255) NOT NULL,
);
CREATE TABLE Student
(
Stu_ID int IDENTITY(1,1) PRIMARY KEY,
Name varchar(255) NOT NULL,
Mobile varchar(255),
Age int,
Course_ID int,
FOREIGN KEY (Course_ID) REFERENCES Course(Course_ID)
);
CREATE TABLE Subject
(
Sub_ID int IDENTITY(1,1) PRIMARY KEY,
Name varchar(255) NOT NULL,
);
CREATE TABLE Teacher
(
Teach_ID int IDENTITY(1,1) PRIMARY KEY,
Name varchar(255) NOT NULL,
Mobile varchar(255)
);
CREATE TABLE Course_Subject
(
CouSub_ID int IDENTITY(1,1) PRIMARY KEY,
Course_ID int,
Sub_ID int,
FOREIGN KEY (Course_ID) REFERENCES Course(Course_ID),
FOREIGN KEY (Sub_ID) REFERENCES Subject(Sub_ID)
);
CREATE TABLE Teacher_Subject
(
TeachSub_ID int IDENTITY(1,1) PRIMARY KEY,
Teach_ID int,
Sub_ID int,
FOREIGN KEY (Teach_ID) REFERENCES Teacher(Teach_ID),
FOREIGN KEY (Sub_ID) REFERENCES Subject(Sub_ID)
);
Now my problem is I need to retrieve students data who learned from some teacher, which means need to retrieve some teacher's students who learned from his/her. To accomplish my requirement. I write this SQL query.
select
s.*
from
tbl_student s
inner join
Course_Subject tcs on s.Course_Id = tcs.Course_Id
inner join
Teacher_Subject tst on tst.Sub_ID = tcs.Sub_ID
inner join
Teacher t on t.Teach_ID = tst.Teach_ID
where
t.Teach_ID = #SomeTeacherId
Now I need to convert this query to a lambda expression or Linq. How can I do it? Please help me. Have any possible way to generate this using Visual Studio.

Well, you could use EF to generate object mapping to your tables. And use LINQ to rewrite your query with a slightly different syntax:
var result = from students in tbl_student
join subjects in Course_Subject on students.Course_Id == subjects.Course_Id
join ts in Teacher_Subject on subjects.Sub_ID == ts.Sub_ID
join teachers in Teacher on teachers.Teach_ID == ts.Teach_ID
where teachers.Teach_ID == "your_value"
select students;
Not sure it's an absolutely correct query, but I hope you'll get the main idea.

Have any possible way to generate this using Visual Studio.?
Yes, you can do this using Linq-to-SQL
for your query, this might be appropriated
var students = from student in db.Students
join tcs in db.CourseSubjects on student.CourseId equals tcs.CourseId
join tst in db.TeacherSubjects on tcs.SubId equals tst.SubId
join t in db.Teachers on tst.TeachId equals t.TeachId
where t.TeachId == someTeacherId
select student;

Lambda:
Students
.Where(x=> x.Course.Course_Subjects
.Any(y => y.Subject.Teacher_Subjects
.Any(z => z.Teach_ID == someTeacherId)
)
)
.Select(x => x)

Adding a large amount of records using LINQ

I have to import a hundreds records to database from Excel.
Each record has to be verified:
Against duplicate
Has to has foreign key in another table
I’m wondering how should I do this with the highest performance. I know that I shouldn’t use db.SaveChanges(); after each record so after verification - I’m adding each record to temporary list (var recordsToAdd), and I’m saving that list after all.
Please check my code below, is this good approach to do this?
using (var db = new DbEntities())
{
var recordsToAdd = new List<User>();
for (var row = 2; row <= lastRow; row++)
{
var newRecord = new User
{
Id = Int32.Parse(worksheet.Cells[idColumn + row].Value.ToNullSafeString()),
FirstName = worksheet.Cells[firstNameColumn + row].Value.ToNullSafeString(),
LastName = worksheet.Cells[lastNameColumn + row].Value.ToNullSafeString(),
SerialNumber = worksheet.Cells[serialNumber + row].Value.ToNullSafeString()
};
bool exists = db.User.Any(u => u.Id == newRecord.Id) || recordsToAdd.Any(u => u.Id == newRecord.Id);
if (!exists)
{
bool isSerialNumberExist = db.SerialNumbers.Any(u => u.SerialNumber == newRecord.SerialNumber);
if (isSerialNumberExist)
{
recordsToAdd.Add(newRecord);
}
else
{
resultMessages.Add(string.Format("SerialNumber doesn't exist"));
}
}
else
{
resultMessages.Add(string.Format("Record already exist"));
}
}
db.User.AddRange(recordsToAdd);
db.SaveChanges();
}

First of all let's separate the code into two parts. First part is creating a list of valid User records to be inserted. Second part is inserting those records to the database (last two lines of your code).
Assuming you are using EntityFramework as your ORM, second part may be optimized by bulk inserting the records. It has many existing solutions that can be easily found. (example)
There are some suggestions concerning the first part.
Load user ids in a HashSet or Dictionary. These data structures are optimized for searching. var userDbIds = new HashSet<int>(db.User.Select(x => x.Id));. You will quickly check if id exists without making a request to DB.
Do the same for serialNumber. var serialNumbers = new HashSet<string>(db.SerialNumber.Select(x => x.SerialNumber)); assuming that type of SerialNumber property is string.
Change the type of your recordToAdd variable to be Dictionary<int, User> for the same reason.
In the check would look like this:
bool exists = userDbIds.Contains(newRecord.Id) || recordsToAdd.ContainsKey(newRecord.Id);
if (!exists)
{
bool isSerialNumberExist = serialNumbers.Contains(newRecord.SerialNumber);
if (isSerialNumberExist)
{
recordsToAdd[newRecord.Id] = newRecord;
}
else
{
resultMessages.Add(string.Format("SerialNumber doesn't exist"));
}
}
else
{
resultMessages.Add(string.Format("Record already exist"));
}

One way to improve the performance is to minimize the db calls and linear searches by using a fast lookup data structures for performing the verification - HashSet<string> for Id and Dictionary<string, bool> for SerialNumber:
using (var db = new DbEntities())
{
var recordsToAdd = new List<User>();
var userIdSet = new HashSet<string>();
var serialNumberExistsInfo = new Dictionary<string, bool>();
for (var row = 2; row <= lastRow; row++)
{
var newRecord = new User
{
Id = Int32.Parse(worksheet.Cells[idColumn + row].Value.ToNullSafeString()),
FirstName = worksheet.Cells[firstNameColumn + row].Value.ToNullSafeString(),
LastName = worksheet.Cells[lastNameColumn + row].Value.ToNullSafeString(),
SerialNumber = worksheet.Cells[serialNumber + row].Value.ToNullSafeString()
};
bool exists = !userIdSet.Add(newRecord.Id) || db.User.Any(u => u.Id == newRecord.Id);
if (!exists)
{
bool isSerialNumberExist;
if (!serialNumberExistsInfo.TryGetValue(newRecord.SerialNumber, out isSerialNumberExist))
serialNumberExistsInfo.Add(newRecord.SerialNumber, isSerialNumberExist =
db.SerialNumbers.Any(u => u.SerialNumber == newRecord.SerialNumber));
if (isSerialNumberExist)
{
recordsToAdd.Add(newRecord);
}
else
{
resultMessages.Add(string.Format("SerialNumber doesn't exist"));
}
}
else
{
resultMessages.Add(string.Format("Record already exist"));
}
}
db.User.AddRange(recordsToAdd);
db.SaveChanges();
}

It would be most efficient to use a Table-Valued Parameter instead of LINQ. That way you can handle this in a set-based approach that is a single connection, single stored procedure execution, and single transaction. The basic setup is shown in the example code I provided in the following answer (here on S.O.):
How can I insert 10 million records in the shortest time possible?
The stored procedure can handle both validations:
don't insert duplicate records
make sure that SerialNumber exists
The User-Defined Table Type (UDTT) would be something like:
CREATE TYPE dbo.UserList AS TABLE
(
Id INT NOT NULL,
FirstName NVARCHAR(50) NOT NULL,
LastName NVARCHAR(50) NULL,
SerialNumber VARCHAR(50) NOT NULL
);
-- Uncomment the following if you get a permissions error:
-- GRANT EXECUTE ON TYPE::[dbo].[UserList] TO [ImportUser];
GO
The stored procedure (executed via SqlCommand.ExecuteNonQuery) would look something like:
CREATE PROCEDURE dbo.ImportUsers
(
#NewUserList dbo.UserList READONLY
)
AS
SET NOCOUNT ON;
INSERT INTO dbo.User (Id, FirstName, LastName, SerialNumber)
SELECT tmp.Id, tmp.FirstName, tmp.LastName, tmp.SerialNumber
FROM #NewUserList tmp
WHERE NOT EXISTS (SELECT *
FROM dbo.User usr
WHERE usr.Id = tmp.[Id])
AND EXISTS (SELECT *
FROM dbo.SerialNumbers sn
WHERE sn.SerialNumber = tmp.[SerialNumber]);
The stored procedure above simply ignores the invalid records. If you need notification of the "errors", you can use the following definition (executed via SqlCommand.ExecuteReader):
CREATE PROCEDURE dbo.ImportUsers
(
#NewUserList dbo.UserList READONLY
)
AS
SET NOCOUNT ON;
CREATE TABLE #TempUsers
(
Id INT NOT NULL,
FirstName NVARCHAR(50) NOT NULL,
LastName NVARCHAR(50) NULL,
SerialNumber VARCHAR(50) NOT NULL,
UserExists BIT NOT NULL DEFAULT (0),
InvalidSerialNumber BIT NOT NULL DEFAULT (0)
);
INSERT INTO #TempUsers (Id, FirstName, LastName, SerialNumber)
SELECT tmp.Id, tmp.FirstName, tmp.LastName, tmp.SerialNumber
FROM #NewUserList tmp;
-- Mark existing records
UPDATE tmp
SET tmp.UserExists = 1
FROM #TempUsers tmp
WHERE EXISTS (SELECT *
FROM dbo.User usr
WHERE usr.Id = tmp.[Id]);
-- Mark invalid SerialNumber records
UPDATE tmp
SET tmp.InvalidSerialNumber = 1
FROM #TempUsers tmp
WHERE tmp.UserExists = 0 -- no need to check already invalid records
AND NOT EXISTS (SELECT *
FROM dbo.SerialNumbers sn
WHERE sn.SerialNumber = tmp.[SerialNumber]);
-- Insert remaining valid records
INSERT INTO dbo.User (Id, FirstName, LastName, SerialNumber)
SELECT tmp.Id, tmp.FirstName, tmp.LastName, tmp.SerialNumber
FROM #TempUsers tmp
WHERE tmp.UserExists = 0
AND tmp.InvalidSerialNumber = 0;
-- return temp table to caller as it contains validation info
SELECT tmp.Id, tmp.FirstName, tmp.LastName, tmp.SerialNumber,
tmp.UserExists, tmp.InvalidSerialNumber
FROM #TempUsers tmp
-- optionally only return records that had a validation error
-- WHERE tmp.UserExists = 1
-- OR tmp.InvalidSerialNumber = 1;
When this version of the stored procedure completes, cycle through SqlDataReader.Read() to get the validation info.

sql view with hebrew column names errors

i use C# to display sql view in datagridview
i have the following view with hebrew for column names
CREATE VIEW [fixes_view]
AS
SELECT f.fixId AS N'מס תיקון',
f.receiveDate AS N'ת.קבלה',
c.clientName AS N'שם לקוח',
m.modelName AS N'דגם',
f.problem AS N'תיאור תקלה',
f.comments AS N'הערות',
e.employeeName AS N'שם עובד',
f.priceOffer AS N'מחיר ראשונ',
f.price AS N'מחיר סופי',
f.returned AS N'הוחזר'
FROM fixes f
INNER JOIN clients c
ON f.clientId = c.clientId
INNER JOIN modelist m
ON f.modelID = m.modelID
INNER JOIN employees e
ON f.employeeId = e.employeeId
when i want to use it in C# and call it with another condition using
public DataTable GetAllActiveFixTable()
{
return genericGetFixTable("WHERE returned=0;");
}
i tried to refer to the "AS" name 'הוחזר' but it gives me an error when i call it
error:
Invalid column name 'returned'.
private DataTable genericGetFixTable (string str)
{
return genericGetTable("select * FROM [fixes_view] " + str);
}
create table fixes (
fixId int IDENTITY(1,1) primary key,
receiveDate date not null,
clientId int not null,
modelID int not null,
problem nvarchar(100) not null,
comments nvarchar(50),
employeeId int not null,
priceOffer real,
price real,
returned bit,
foreign key (clientId)
references clients,
foreign key (modelID)
references modelist,
foreign key (employeeId)
references employees)

LINQ union with BIT column causing Specified cast is not valid error

I have a complicated join between a few tables but I have managed to replicate the error using linqpad and the small tables below. There are references between the COLNAME column and the YAXIS column and also between COLNAME and XAXIS that is not explicitly defined.
The error is "Specified cast is not valid", which originally I wasted time thinking the problem was converting the data returned to my object in VS 2010, but the error also happens in linqpad with no defined object. It seems insane that a bit column would cause this problem. If I change the column type to a VARCHAR it works fine. If I run the generated SQL from linqpad or sql profiler that also returns fine.
What is going on?
CREATE TABLE TEST_QUERY
([ID] INT IDENTITY(1,1) PRIMARY KEY,
blah VARCHAR(20))
CREATE TABLE TEST_QUERY_COLS
(QUERYID INT NOT NULL,
COLNAME VARCHAR(20) NOT NULL,
otherblah VARCHAR(20),
PRIMARY KEY(QUERYID,COLNAME))
CREATE TABLE TEST_CHART
(CHARTID INT IDENTITY(1,1) PRIMARY KEY,
QUERYID INT NOT NULL REFERENCES TEST_QUERY([ID]),
XAXIS VARCHAR(20) NOT NULL,
blahblah VARCHAR(20))
CREATE TABLE TEST_CHART_SERIES
(CHARTID INT NOT NULL REFERENCES TEST_CHART(CHARTID),
YAXIS VARCHAR(20) NOT NULL,
blahblahblah BIT NOT NULL,
PRIMARY KEY(CHARTID,YAXIS))
INSERT INTO TEST_QUERY(blah) VALUES('xxx')
INSERT INTO TEST_QUERY_COLS(QUERYID,COLNAME,otherblah) VALUES(1,'col1','xxx')
INSERT INTO TEST_QUERY_COLS(QUERYID,COLNAME,otherblah) VALUES(1,'col2','yyy')
INSERT INTO TEST_CHART(QUERYID,XAXIS,blahblah) VALUES(1,'col1','xxx')
INSERT INTO TEST_CHART_SERIES(CHARTID,YAXIS,blahblahblah) VALUES(1,'col2',1)
This is the linq statement:
((from ch in TEST_CHARTs
join a in TEST_CHART_SERIES on ch.CHARTID equals a.CHARTID into a_join
from cs in a_join.DefaultIfEmpty()
join ycols in TEST_QUERY_COLS on new { key1 = cs.YAXIS, key2 = ch.QUERYID } equals new { key1 = ycols.COLNAME, key2 = ycols.QUERYID }
where ch.CHARTID == 1
select new
{
ch.CHARTID,
POSITION = 0,
ycols.QUERYID,
ycols.Otherblah,
cs.Blahblahblah
})
.Union(from ch in TEST_CHARTs
join xcol in TEST_QUERY_COLS on new { key1 = ch.XAXIS, key2 = ch.QUERYID } equals new { key1 = xcol.COLNAME, key2 = xcol.QUERYID }
where ch.CHARTID == 1
select new
{
ch.CHARTID,
POSITION = 0,
xcol.QUERYID,
xcol.Otherblah,
Blahblahblah = false
})).Distinct()
Edit: I've filed a bug with microsoft here

The generated sql includes the following line, where #p4 corresponds to the Blahblahblah=false line in your projection:
DECLARE #p4 Int = 0
And the int that is returned from the query can't be converted to a bool. I don't know whether or not this is a linq to sql bug (seems like it), but there is a workaround. Basically you need to drop the Blahblahblah=false from the anonymous type projected, then .ToList() or .ToArray() the result, and finally add the bool field in a linq to objects projection:
var one =
(from ch in TEST_CHARTs
join a in TEST_CHART_SERIES on ch.CHARTID equals a.CHARTID into a_join
from cs in a_join.DefaultIfEmpty()
join ycols in TEST_QUERY_COLS on new { key1 = cs.YAXIS, key2 = ch.QUERYID } equals new { key1 = ycols.COLNAME, key2 = ycols.QUERYID }
where ch.CHARTID == 1
select new
{
ch.CHARTID,
POSITION = 0,
ycols.QUERYID,
ycols.Otherblah,
Blahblahblah = cs.Blahblahblah
}).ToList();
var two =
(from ch in TEST_CHARTs
join xcol in TEST_QUERY_COLS on new { key1 = ch.XAXIS, key2 = ch.QUERYID } equals new { key1 = xcol.COLNAME, key2 = xcol.QUERYID }
where ch.CHARTID == 1
select new
{
ch.CHARTID,
POSITION = 0,
xcol.QUERYID,
xcol.Otherblah
}).ToList();
var three =
from x in two
select new
{
x.CHARTID,
x.POSITION,
x.QUERYID,
x.Otherblah,
Blahblahblah = false
};
var four = one.Union(three).Distinct();
Note that this results in two sql queries, not one.
EDIT
Also, Distinct() can be left out, since union doesn't include duplicates. I should have actually read the code that I copied and pasted!