Develop Reference

C# - Linq to combine (or) join two datatables into one

I'm having problem in getting correct data from two datatables into one by using Linq in C#.
My datatables' data are coming from Excel file reading (not from DB).
I have tried below linq but return rows count is not what I want (my goal is to retrieve all data but for verification, I'm checking on row count so that I can know is it correct or not easily).
In dt1, I have 2645 records.
In dt2, I have 2600 records.
Return row count is 2600 (it looks like it is doing right join logic).
var v1 = from d1 in dt1.AsEnumerable()
from d2 in dt2.AsEnumerable()
.Where(x => x.Field<string>(X_ITEM_CODE) == d1.Field<string>(X_NO)
|| x.Field<string>(X_ITEM_KEY) == d1.Field<string>(X_NO))
select dt1.LoadDataRow(new object[]
{
// I use short cut way instead of Field<string> for testing purpose.
d1[X_NO],
d2[X_ITEM_CODE] == null ? "" : d2[X_ITEM_CODE] ,
d2[X_ITEM_KEY] == null ? "" : d2[X_ITEM_KEY],
d2[X_COSTS],
d2[X_DESC],
d2[X_QTY]== null ? 0 : dt[X_QTY]
}, false);
dt1 = v1.CopyToDataTable();
Console.WriteLine(dt1.Rows.Count);
I tried to use 'join' but my problem is the X_NO value can be either in X_ITEM_CODE or X_ITEM_KEY, so I can only put one condition in ON xxx equals yyy.
I would like to try 'join' if my above condition is suitable to use too. Please provide me some guide. Thanks.
[Additional Info]
I already tried foreach loop + dt1.Select(xxxx) + dt1.Rows.Add(xxx), it is working well but with around 2 minutes to complete the job.
I'm looking for a faster way and from above Linq code I tried, it seems faster than my foreach looping so I want to give Linq a chance.
For demo purpose, I only put a few columns in above example, my actual column count is 12 columns.
I afraid my post will become very long if I put on my foreach loop so I skip it when I post this question.
Anyway, below is the code and sample data. For those who can edit and think it is too long, kindly take out unnecessary/unrelated code or lines.
DataRow[] drs = null;
DataRow drO = null;
foreach (DataRow drY in dt2.Rows)
{
drs = null;
drs = dt1.Select(X_NO + "='" + drY[X_ITEM_KEY] + "' OR " + X_NO + "='" + drY[X_ITEM_CODE] + "'");
if (drs.Length >= 0)
{
// drs Leng will always 1 because no duplicate.
drs[0][X_ITEM_CODE] = drY[X_ITEM_CODE];
drs[0][X_ITEM_KEY] = drY[X_ITEM_KEY];
drs[0][X_COST] = clsD.GetInt(drY[X_COST]); // If null, return 0.
drs[0][X_DESC] = clsD.GetStr(drY[X_DESC]); // If null, return "".
drs[0][X_QTY] = clsD.GetInt(drY[X_QTY]);
}
else
{
// Not Found in ITEM CODE or KEY, add it.
drO = dtOutput.NewRow();
drO[X_ITEM_CODE] = drY[X_ITEM_CODE];
drO[X_ITEM_KEY] = drY[X_ITEM_KEY];
drO[X_COST] = clsD.GetInt(drY[X_COST]);
drO[X_DESC] = clsD.GetStr(drY[X_DESC]);
drO[X_QTY] = clsD.GetInt(drY[X_QTY]);
dt1.Rows.Add(drO);
}
}
// Note: For above else condition, I didn't put in my Linq testing yet.
// If without else condition, my dt1 will still have same record count.
[dt1 data]
X_NO,X_ITEM_CODE,X_ITEM_KEY,COST,DESC,QTY,....
AA060210A,,,,,,....
AB060220A,,,,,....
AC060230A,,,,,....
AD060240A,,,,,....
[dt2 data]
X_ITEM_CODE,X_ITEM_KEY,COST,DESC,QTY
AA060210A,AA060211A,100.00,PART1,10000
AB060221A,AB060220A,120.00,PART2,500
AC060232A,AC060230A,150.00,PART3,100
AD060240A,AD060243A,4.50,PART4,15250
[Update 2]
I tried below 'join' and it return nothing. So, can I assume join also will not help?
var vTemp1 = from d1 in dt1.AsEnumerable()
join d2 in dt2.AsEnumerable()
on 1 equals 1
where (d1[X_NO] == d2[X_ITEM_CODE] || d1[X_NO] == d2[X_ITEM_KEY])
select dt1.LoadDataRow(new object[]
{
d1[X_NO],
d2[X_ITEM_CODE] == null ? "" : d2[X_ITEM_CODE] ,
d2[X_ITEM_KEY] == null ? "" : d2[X_ITEM_KEY],
d2[X_COST],
d2[X_DESC],
d2[X_QTY]== null ? 0 : d2[X_QTY]
}, false);
Console.WriteLine(vTemp1.Count()); // return zero.

LINQ supports only equijoins, so apparently join operator cannot be used. But using LINQ query with Cartesian product and where will not give you any performance improvement.
What you really need (being LINQ or not) is a fast lookup by dt1[X_NO] field. Since as you said it is unique, you can build and use a dictionary for that:
var dr1ByXNo = dt1.AsEnumerable().ToDictionary(dr => dr.Field<string>(X_NO));
and then modify your process like this:
foreach (DataRow drY in dt2.Rows)
{
if (dr1ByXNo.TryGetValue(drY.Field<string>(X_ITEM_KEY), out dr0) ||
dr1ByXNo.TryGetValue(drY.Field<string>(X_ITEM_CODE), out dr0))
{
dr0[X_ITEM_CODE] = drY[X_ITEM_CODE];
dr0[X_ITEM_KEY] = drY[X_ITEM_KEY];
dr0[X_COST] = clsD.GetInt(drY[X_COST]); // If null, return 0.
dr0[X_DESC] = clsD.GetStr(drY[X_DESC]); // If null, return "".
dr0[X_QTY] = clsD.GetInt(drY[X_QTY]);
}
else
{
// Not Found in ITEM CODE or KEY, add it.
drO = dtOutput.NewRow();
drO[X_ITEM_CODE] = drY[X_ITEM_CODE];
drO[X_ITEM_KEY] = drY[X_ITEM_KEY];
drO[X_COST] = clsD.GetInt(drY[X_COST]);
drO[X_DESC] = clsD.GetStr(drY[X_DESC]);
drO[X_QTY] = clsD.GetInt(drY[X_QTY]);
dt1.Rows.Add(drO);
}
}
Since you are adding new records to the dt1 during the process, depending of your requirements you might need to add at the end of the else (after dt1.Rows.Add(drO); line) the following
dr1ByXNo.Add(dr0.Field<string>(X_NO), dr0);
I didn't include it because I don't see your code setting the new record X_NO field, so the above will produce duplicate key exception.

Linq2SQL grouping and ungrouping in the same query

Here's a stumper in LINQ to SQL:
string p = prefix ?? "";
string d = delimiter ?? "";
var filegroups = from b in folder.GetFiles(data)
where b.Uri.StartsWith(p) && b.Uri.CompareTo(marker ?? "") >= 0
group b by data.DataContext.GetFileFolder(p, d, b.Uri);
//var folders = from g in filegroups where g.Key.Length > 0 select g;
//var files = from g in filegroups where g.Key.Length == 0 select g;
var files = filegroups.SelectMany(g => g.Key.Length > 0
? from b in g.Take(1) select new FilePrefix { Name = g.Key }
: from b in g select new FilePrefix { Name = b.Uri, Original = b });
var retval = files.Take(maxresults);
folders cannot be nested (out of my control) but filenames can contain slashes and whatever so a deeper folder structure can be emulated
folder.GetFiles is a simple linq equiv (IOrderedQueryable) to select * from files where folderid=#folderid order by Uri
prefix is a filter saying return only those files that start with...delimiter is the path delimiter, such as '/'marker is for pagination - starts returning at a specified point
data.DataContext.GetFileFolder maps to a sql scalar function: return the whole string up to and including the next delimiter that occurs after the prefix string
RETURN substring(#uri, 0, charindex(#delimiter, #uri, len(#prefix)) + len(#delimiter)) That was for troubleshooting - original was a client-side where clause that did map correctly to TSQL. I had just hoped doing a function would change things in the final graph, but nope.
in the above, filegroups, and the commented out folders, and files, all work as expected
The goal is to hit the database just once. I'd like to, in a single return, show subfolders and files based upon interpretation of the FilePrefix object (folders have a null 'original' value)
The issue is with the final selectmany throwing "Could not format node 'ClientQuery' for execution as SQL."
I strongly suspect this would work perfectly if it weren't for the TSQL translation, but looking at this logically, why would it not do its database work and then select the FilePrefixes client side as a final step?
It's late ;) but tomorrow I'll revert to a double tap on the database by slipping a ToList() or something similar somewhere up there to cause that final step to be full client side (kludge). But if anyone has any insights on how to accomplish this with one database hit (short of writing a stored procedure), I'd love to hear it!!
The downside to the kludge is that the final Take(maxresults) could be expensive if the db hit results in a number of records that far exceeds that. And the subsequent Skip(maxresults).Take(1) that I didn't quote, for marking the next page, would hurt twice as much.
Thank you very much

Welp, it looks like 2 database hits are necessary. I started by noticing that the call graph converted the tertiary operator into an IIF which led me to think that IIF, on the sql side, probably doesn't like subqueries as parameters.
string p = prefix ?? "";
string d = delimiter ?? "";
var filegroups = from b in folder.GetFiles(data)
where b.Uri.StartsWith(p) && b.Uri.CompareTo(marker ?? "") >= 0
group b by data.DataContext.nx_GetFileFolder(p, d, b.Uri);
var folders = from g in filegroups where g.Key.Length > 0 select g.Key;
var files = from b in folder.GetFiles(data)
where b.Uri.StartsWith(p) && b.Uri.CompareTo(marker ?? "") >= 0
&& data.DataContext.nx_GetFileFolder(p, d, b.Uri).Length == 0
select b;
folders = folders.OrderBy(f => f).Take(maxresults + 1);
files = files.OrderBy(f => f.Uri).Take(maxresults + 1);
var retval = folders.AsEnumerable().Select(f => new FilePrefix { Name = f })
.Concat(files.AsEnumerable().Select(f => new FilePrefix { Name = f.Uri, Original = f }))
.OrderBy(b => b.Name).Take(maxresults + 1);
int count = 0;
foreach (var bp in retval)
{
if (count++ < maxresults)
yield return bp;
else
newmarker.Name = bp.Name;
}
yield break;
a bit less elegant... I left filegroups and folders, but rewrote the files query to get rid of the group (generated cleaner sql and probably more efficient).
Concat still gave me trouble in this new approach, so I kicked in the AsEnumerable calls, which is the point that breaks this into 2 hits to the database.
I kept maxresults in the sql to limit traffic, so worst case is twice as much data as I want going over the wire. The +1 is to get the next record so the user can be notified where to start on the next page. And I used the iterator pattern so I wouldn't have to loop again to get that next record.

XML to Linq C#, no results

I'm trying to grab the Filename from the below XML code where the partNum passed in by the end user matches the partNum of a JES.
My current code yields 0 results.
I will also be trying to grab the disrete attribute as well.
<JESs>
<JES partNum="116102440002" discrete="true">
<Filename>116-10244-0002_ILLK Collimator Cover Assy_Rev 3.docx</Filename>
</JES>
<JES partNum="116102440003" discrete="false">
<Filename>ILLK Collimator in Gimbal_Rev 4.docx</Filename>
</JES>
<JES partNum="116102440004" discrete="true">
<Filename>116-10244-0004_Collimator Cover Installation_Rev 1.docx</Filename>
</JES>
<JES partNum="116102440005" discrete="true">
<Filename>116-10244-0005_Collimator Lens Assembly_Rev 2.docx</Filename>
</JES>
</JESs>
C# Code
var FileName = (from n in xml.Descendants("JESs") where n.Element("JES").Attribute("partNum").Value == Convert.ToString(partNum) select n.Elements().Descendants().Elements()).ToList();

You can simplify (and fix) your query like this:
var partNumber = Convert.ToString(partNum);
var result = xml.Descendants("JES")
.FirstOrDefault(x => (string)x.Attribute("partNum") == partNumber);
if(result != null)
{
var fileName = (string)result.Element("Filename");
}

Shouldn't it be:
var FileNames = (from n in xml.Descendants("JESs")
where n.Element("JES").Attribute("partNum").Value == Convert.ToString(partNum)
select n.Element("JES").Element("Filename")).ToList();

Try this
var fileName = (from x in xml.Elements()
where x.Attribute("partNum").Value == "xxxx"
select x.Element("Filename").Value).FirstOrDefault();
//replace xxxx with actual value
If no elements found for given partNum, fileName will be null

Comparing two counts in LINQ

hi have a simple query that compares all sent out requests to those that have been downloaded.
i have a tsql that works perfectly like so:
select case when (select COUNT(*) from FileRecipient where File_ID = 3 and downloaded = 'Y')
= (select COUNT(*) from FileRecipient where File_ID = 3) then 'Y' else 'N' end
but would like to do it in LINQ.
any and all help would be appreciated. i tried the following, but obviously i'm a long way away from getting it right. this just gives me the first of the two counts. do i need to do two separate statements to get the second?
public static bool DownloadedByAll(int fsID)
{
using (SEntities ctx = CommonS.GetSContext())
{
var result = (from fr in ctx.FileRecipients
where fr.File_ID == fsID && fr.downloaded == "Y"
select fr.Recipient_ID).Count();
}
}
or would it just be simpler for me to use tsql within the entity framework for this?

This should get you the result you need
var result = ctx.FileRecipients
.Count(f=>f.File_ID == 3
&& f.downloaded == 'Y') == ctxt.FileRecipients
.Count(f=>f.File_ID == 3);

Using Intersect I'm getting a Local sequence cannot be used in LINQ to SQL implementations of query operators except the Contains operator

I'm using a Linq to SQL query to provide a list of search term matches against a database field. The search terms are an in memory string array. Specifically, I'm using an "intersect" within the Linq query, comparing the search terms with a database field "Description". In the below code, the description field is iss.description. The description field is separated into an array within the Linq query and the intersect is used to compare the search terms and description term to keep all of the comparing and conditions within the Linq query so that the database is not taxed. In my research, trying o overcome the problem, I have found that the use of an in-memory, or "local" sequence is not supported. I have also tried a few suggestions during my research, like using "AsEnumerable" or "AsQueryable" without success.
searchText = searchText.ToUpper();
var searchTerms = searchText.Split(' ');
var issuesList1 = (
from iss in DatabaseConnection.CustomerIssues
let desc = iss.Description.ToUpper().Split(' ')
let count = desc.Intersect(searchTerms).Count()
where desc.Intersect(searchTerms).Count() > 0
join stoi in DatabaseConnection.SolutionToIssues on iss.IssueID equals stoi.IssueID into stoiToiss
from stTois in stoiToiss.DefaultIfEmpty()
join solJoin in DatabaseConnection.Solutions on stTois.SolutionID equals solJoin.SolutionID into solutionJoin
from solution in solutionJoin.DefaultIfEmpty()
select new IssuesAndSolutions
{
IssueID = iss.IssueID,
IssueDesc = iss.Description,
SearchHits = count,
SolutionDesc = (solution.Description == null)? "No Solutions":solution.Description,
SolutionID = (solution.SolutionID == null) ? 0 : solution.SolutionID,
SolutionToIssueID = (stTois.SolutionToIssueID == null) ? 0 : stTois.SolutionToIssueID,
Successful = (stTois.Successful == null)? false : stTois.Successful
}).ToList();
...
The only way I have been successful is to create two queries and calling a method as shown below, but this requires the Linq Query to return all of the matching results (with the number of hits for search terms in the description) including the non-matched records and provide an in-memory List<> and then use another Linq Query to filter out the non-matched records.
public static int CountHits(string[] searchTerms, string Description)
{
int hits = 0;
foreach (string item in searchTerms)
{
if (Description.ToUpper().Contains(item.Trim().ToUpper())) hits++;
}
return hits;
}
public static List<IssuesAndSolutions> SearchIssuesAndSolutions(string searchText)
{
using (BYCNCDatabaseDataContext DatabaseConnection = new BYCNCDatabaseDataContext())
{
searchText = searchText.ToUpper();
var searchTerms = searchText.Split(' ');
var issuesList1 = (
from iss in DatabaseConnection.CustomerIssues
join stoi in DatabaseConnection.SolutionToIssues on iss.IssueID equals stoi.IssueID into stoiToiss
from stTois in stoiToiss.DefaultIfEmpty()
join solJoin in DatabaseConnection.Solutions on stTois.SolutionID equals solJoin.SolutionID into solutionJoin
from solution in solutionJoin.DefaultIfEmpty()
select new IssuesAndSolutions
{
IssueID = iss.IssueID,
IssueDesc = iss.Description,
SearchHits = CountHits(searchTerms, iss.Description),
SolutionDesc = (solution.Description == null)? "No Solutions":solution.Description,
SolutionID = (solution.SolutionID == null) ? 0 : solution.SolutionID,
SolutionToIssueID = (stTois.SolutionToIssueID == null) ? 0 : stTois.SolutionToIssueID,
Successful = (stTois.Successful == null)? false : stTois.Successful
}).ToList();
var issuesList = (
from iss in issuesList1
where iss.SearchHits > 0
select iss).ToList();
...
I would be comfortable with two Linq Queries, but with the first Linq Query only returning the matched records and then maybe using a second, maybe lambda expression to order them, but my trials have not been successful.
Any help would be most appreciated.

Ok, so after more searching more techniques, and trying user1010609's technique, I managed to get it working after an almost complete rewrite. The following code first provides a flat record query with all of the information I am searching, then a new list is formed with the filtered information compared against the search terms (counting the hits of each search term for ordering by relevance). I was careful not to return a list of the flat file so there would be some efficiency in the final database retrieval (during the formation of the filtered List<>). I am positive this is not even close to being an efficient method, but it works. I am eager to see more and unique techniques to solving this type of problem. Thanks!
searchText = searchText.ToUpper();
List<string> searchTerms = searchText.Split(' ').ToList();
var allIssues =
from iss in DatabaseConnection.CustomerIssues
join stoi in DatabaseConnection.SolutionToIssues on iss.IssueID equals stoi.IssueID into stoiToiss
from stTois in stoiToiss.DefaultIfEmpty()
join solJoin in DatabaseConnection.Solutions on stTois.SolutionID equals solJoin.SolutionID into solutionJoin
from solution in solutionJoin.DefaultIfEmpty()
select new IssuesAndSolutions
{
IssueID = iss.IssueID,
IssueDesc = iss.Description,
SolutionDesc = (solution.Description == null) ? "No Solutions" : solution.Description,
SolutionID = (solution.SolutionID == null) ? 0 : solution.SolutionID,
SolutionToIssueID = (stTois.SolutionToIssueID == null) ? 0 : stTois.SolutionToIssueID,
Successful = (stTois.Successful == null) ? false : stTois.Successful
};
List<IssuesAndSolutions> filteredIssues = new List<IssuesAndSolutions>();
foreach (var issue in allIssues)
{
int hits = 0;
foreach (var term in searchTerms)
{
if (issue.IssueDesc.ToUpper().Contains(term.Trim())) hits++;
}
if (hits > 0)
{
IssuesAndSolutions matchedIssue = new IssuesAndSolutions();
matchedIssue.IssueID = issue.IssueID;
matchedIssue.IssueDesc = issue.IssueDesc;
matchedIssue.SearchHits = hits;
matchedIssue.CustomerID = issue.CustomerID;
matchedIssue.AssemblyID = issue.AssemblyID;
matchedIssue.DateOfIssue = issue.DateOfIssue;
matchedIssue.DateOfResolution = issue.DateOfResolution;
matchedIssue.CostOFIssue = issue.CostOFIssue;
matchedIssue.ProductID = issue.ProductID;
filteredIssues.Add(matchedIssue);
}
}