Develop Reference

Seeking for help for SQL server query to search multiple columns of a table using a combination of keywords

I am having a nightmare in building an efficient search query. I am using LINQ. The requirement is as follows:
In the application there is one text fox field which is used as a Quick Search.
The table I will be searching holds three fields Make, Model and Extension.
A typical keyword user can enter is like Honda Accord XL
Based on the keywords database should return me the matching rows and here the problem starts. There is no restriction on the order which keywords will be entered to prepare the phrase, i.e. one can enter Accord XL Honda or it could be like XL Accord or could just be Honda. In the example Honda is Make, Accord is the Model and XL is the extension.
Ideally the search result should only pull up perfect matches like if Honda Accord is entered it will not bring up other models from Honda. But the major problem is I don't know what they will enter and I have to look into three different columns of the table using Contains operator.
Here is what I tried:
I spit the phrase into words and place them in an array
var arr = keyWord.Split(new [] {' '}). Next step I build the query inside a loop of those array elements:
foreach (var k in arr)
{
var item = new Vehicle();
var arrayItem = k;
var query = DataContext.Vehicles.Where(v =>v.RealName.Contains(arrayItem)
|| v.Model.Contains(arrayItem)
|| v.Style.Contains(arrayItem)).ToList();
foreach (var v in query)
{
if(!result.Contains(v))
result.Add(v);
}
}
return result;
Now when the loop is executing and matching records for Make it already fills the list with say 250 items. But how can I remove unwanted items like when a record has CT as Model or TYL as Extension? If I knew the order of the words in which the keyword was created then I will have the option to remove unmatched Make, Model or Extension from the list by using one line of code for each and return the final result. But in this case if I have to do it I again have to use loop and remove unmatched items, even that will not probably give me the correct data. And definitely this is not the efficient way to do this.

--- Try to use Below way your code first retrieve respective make to database ------then filter gettting previous result.
List<Vehicle> lsvehicle= new List<Vehicle()>;
foreach (var k in arr)
{
var arrayItem = k;
lsvehicle = DataContext.Vehicles.Where(v =>v.RealName.Contains(arrayItem) ).ToList();
}
foreach (var k in arr)
{
lsvehicle = lsvehicle.Where(v =>v.Model.Contains(arrayItem) || v.Style.Contains(arrayItem)).tolist();
}
return lsvehicle ;

This can be achieved by concatenating the Make + Model + Extension string and then comparing that the whole array is contained by this string
var query = DataContext.Vehicles;
foreach (var k in arr)
{
var item = new Vehicle();
var arrayItem = k;
query = query.Where(v => (v.RealName + v.Model + v.Style).Contains(arrayItem)).ToList();
}
return query;
NOTE: Logical answer, may require syntax error correction if any

I would suggest following approach, assuming you have the option to create view in database
and you are searching through three columns
1)Create a view With all combination
2)Use linq to get the record from the view
sql
create view [Vw_VehicleSearch]
AS
Select
M+V+E [MVE],
M+E+V [MEV],
V+M+E [VME],
v+E+M [VEM],
E+M+V [EMV],
E+V+M [EVM]
from
vehicletable
c#
public List<string> Search(string quickSearchText)
{
using(var ctx=new model()))
{
var result=ctx
.Vw_VehicleSearch
.Where(v=>v.MVE.Contains(quickSearchText)
|| v=>v.MEV.Contains(quickSearchText)
.|| v=>v.VME.Contains(quickSearchText)
.|| v=>v.VEM.Contains(quickSearchText)
.|| v=>v.EMV.Contains(quickSearchText)
.|| v=>v.EVM.Contains(quickSearchText)
.ToList();
return result.Select(r=>r.MVE).ToList();
}
}

What you want is: all vehicles of which either RealName, or Model, or Style contains all keywords. This can be achieved by:
var query = DataContext.Vehicles.Where(v =>
arr.All(s => v.RealName.Contains(s))
|| arr.All(s => v.Model.Contains(s))
|| arr.All(s => v.Style.Contains(s)))
.ToList();
Entity Framework is able to translate this query into SQL because it has a trick to convert the array arr into a table (of sorts) that can be used in the SQL statement (you should take a look at the generated SQL).
This is not the most efficient way to run a query. It will become considerably slower when the number of keywords becomes "large". I don't think that will be an issue here though.

Querying a chain of list of lists with LINQ

I am working with an XML standard called SDMX. It's fairly complicated but I'll make it as short as possible. I am receiving an object called CategoryScheme. This object can contain a number of Category, and each Category can contain more Category, and so on, the chain can be infinite. Every Category has an unique ID.
Usually each Category contains a lot of Categories. Together with this object I am receiving an Array, that contains the list of IDs that indicates where a specific Category is nested, and then I am receiving the ID of that category.
What I need to do is to create an object that maintains the hierarchy of the Category objects, but each Category must have only one child and that child has to be the one of the tree that leads to the specific Category.
So I had an idea, but in order to do this I should generate LINQ queries inside a cycle, and I have no clue how to do this. More information of what I wanted to try is commented inside the code
Let's go to the code:
public void RemoveCategory(ArtefactIdentity ArtIdentity, string CategoryID, string CategoryTree)
{
try
{
WSModel wsModel = new WSModel();
// Prepare Art Identity and Array
ArtIdentity.Version = ArtIdentity.Version.Replace("_", ".");
var CatTree = JArray.Parse(CategoryTree).Reverse();
// Get Category Scheme
ISdmxObjects SdmxObj = wsModel.GetCategoryScheme(ArtIdentity, false, false);
ICategorySchemeMutableObject CatSchemeObj = SdmxObj.CategorySchemes.FirstOrDefault().MutableInstance;
foreach (var Cat in CatTree)
{
// The cycle should work like this.
// At every iteration it must delete all the elements except the correct one
// and on the next iteration it must delete all the elements of the previously selected element
// At the end, I need to have the CatSchemeObj full of the all chains of categories.
// Iteration 1...
//CatSchemeObj.Items.ToList().RemoveAll(x => x.Id != Cat.ToString());
// Iteration 2...
//CatSchemeObj.Items.ToList().SingleOrDefault().Items.ToList().RemoveAll(x => x.Id != Cat.ToString());
// Iteration 3...
//CatSchemeObj.Items.ToList().SingleOrDefault().Items.ToList().SingleOrDefault().Items.ToList().RemoveAll(x => x.Id != Cat.ToString());
// Etc...
}
}
catch (Exception ex)
{
throw ex;
}
}
Thank you for your help.

So, as i already said in my comment, building a recursive function should fix the issue. If you're new to it, you can find some basic information about recursion in C# here.
The method could look something like this:
private void DeleteRecursively(int currentRecursionLevel, string[] catTree, ICategorySchemeMutableObject catSchemeObj)
{
catSchemeObj.Items.ToList().RemoveAll(x => x.Id != catTree[currentRecursionLevel].ToString());
var leftoverObject = catSchemeObj.Items.ToList().SingleOrDefault();
if(leftoverObject != null) DeleteRecursively(++currentRecursionLevel, catTree, leftoverObject);
}
Afterwards you can call this method in your main method, instead of the loop:
DeleteRecursively(0, CatTree, CatSchemeObject);
But as i also said, keep in mind, that calling the method in the loop, seems senseless to me, because you already cleared the tree, besides the one leftover path, so calling the method with the same tree, but another category, will result in an empty tree (in CatSchemeObject).
CAUTION! Another thing to mention i noticed right now: Calling to list on your Items property and afterwards deleting entries, will NOT affect your source object, as ToList is generating a new object. It IS keeping the referenced original objects, but a deletion only affects the list. So you must write back the resulting list to your Items property, or find a way to directly delete in the Items object. (Assuming it's an IEnumerable and not a concrete collection type you should write it back).
Just try it out with this simple example, and you will see that the original list is not modified.
IEnumerable<int> test = new List<int>() { 1, 2, 3, 4 , 1 };
test.ToList().RemoveAll(a => a != 1);

Edited:
So here is another possible way of going after the discussion below.
Not sure what do you really need so just try it out.
int counter = 0;
var list = CatSchemeObj.Items.ToList();
//check before you call it or you will get an error
if(!list.Equals(default(list)))
{
while(true)
{
var temp = list.Where(x => CatTree[counter++] == x.Id); // or != ? play with it .
list = temp.Items.ToList().SingleOrDefault();
if(list.Equals(default(list))
{
break;
}
}
}
I just translated you problem to 2 solutions, but I am not sure if you won't lose data because of the SingleOrDefault call. It means 'Grab the first item regardless of everything'. I know you said you have only 1 Item that is ok, but still... :)
Let me know in comment if this worked for you or not.
//solution 1
// inside of this loop check each child list if empty or not
foreach (var Cat in CatTree)
{
var list = CatSchemeObj.Items.ToList();
//check before you call it or you will get an error
if(!list.Equals(default(list)))
{
while(true)
{
list.RemoveAll(x => x.Id != Cat.ToString());
list = list.ToList().SingleOrDefault();
if(list.Equals(default(list))
{
break;
}
}
}
}
//solution 2
foreach (var Cat in CatTree)
{
var list = CatSchemeObj.Items.ToList();
//check before you call it or you will get an error
if(!list.Equals(default(list)))
{
CleanTheCat(cat, list);
}
}
//use this recursive function outside of loop because it will cat itself
void CleanTheCat(string cat, List<typeof(ICategorySchemeMutableObject.Items) /*Place here whatever type you have*/> CatSchemeObj)
{
CatSchemeObj.RemoveAll(x => x.Id != cat);
var catObj = CatSchemeObj.Items.ToList().SingleOrDefault();
if (!catObj.Equals(default(catObj)){
CleanTheCat(cat, catObj);
}
}

Thank you to whoever tried to help but I solved it by myself in a much easier way.
I just sent the full CategoryScheme object to the method that converted it in the XML format, then just one line did the trick:
XmlDocument.Descendants("Category").Where(x => !CatList.Contains(x.Attribute("id").Value)).RemoveIfExists();

What is the most efficient way to find elements in a list that do not exist in another list and vice versa?

Consider you have two lists in C#, first list contains elements of TypeOne and second list contains elements of TypeTwo:
TypeOne
{
int foo;
int bar;
}
TypeTwo
{
int baz;
int qux;
}
Now I need to find elements ( with some property value ) in the first list that don't exist in the second list, and similarly I want to find elements in the second list that don't exist in the first list. (There are only zero or one occurences in either lists.)
What I tried so far is to iterate both lists like this:
foreach (var item in firstList)
{
if (!secondList.Any(a=> a.baz == item.foo)
{
// Item is in the first list but not in second list.
}
}
and again:
foreach (var item in secondList)
{
if (!firstList.Any(a=> a.foo == item.baz)
{
// Item is in the second list but not in first list.
}
}
I hardly think this is a good way to do what I want. I'm iterating my lists two times and use Any in each of them which also iterates the list. So too many iterations.
What is the most efficient way to achieve this?

I am afraid there is no prebuild solution for this, so the best we can do is optimize as much as possible. We only have to iterate the first list, because everything that is in second will be compared already
// First we need copies to operate on
var firstCopy = new List<TypeOne>(firstList);
var secondCopy = new List<TypeTwo>(secondList);
// Now we iterate the first list once complete
foreach (var typeOne in firstList)
{
var match = secondCopy.FirstOrDefault(s => s.baz == typeOne.foo);
if (match == null)
{
// Item in first but not in second
}
else
{
// Match is duplicate and shall be removed from both
firstCopy.Remove(typeOne);
secondCopy.Remove(match);
}
}
After running this both copies will only contain the values which are unique in this instance. This not only reduces it to half the number of iterations but also constantly improves because the second copy shrinks with each match.

Use this LINQ Query.
var result1 = secondList.Where(p2 => !firstList.Any(p1 => p1.foo == p2.baz));
var result2=firstList.Where(p1=> !secondList.Any(p2=> p2.foo == p1.baz);

Searching List<T>

I have a List<string> containing file paths. How can I check if there is a particular file in my list e.g. file.txt? Preferably a method returning not a bool value but list element or element's index.
I've read the List<T> documentation but it only confused me as I'm a beginning programmer and my English isn't very good.

Use Where to get a list of values:
var found = fileList.
Where((f)=>System.IO.Path.GetFileName(f)
.Equals(SEARCH_VALUE,
StringComparison.InvariantCultureIgnoreCase));
or use FirstOrDefault for single element or null in case it's not found

If your list contains the full path (like c:\windows\system.ini") I would use System.IO.Path.GetFileName and also keep in mind to search case intenstive
var result = from f in files
where Path.GetFileName(f).Equals("file.txt",
StringComparison.InvariantCultureIgnoreCase)
select f;
bool found = result.Any();

The IndexOf method is what you need, if you want to find the path that exactly watches what you are looking for.
However, if you what to find paths in your list that end with a certain file name, you can do,
var matches = paths.Select((path, i) => new { Path = path, Index = i })
.Where(item => Path.GetFileName(item.Path).Equals(
"file.txt",
StringComparison.OrdinalIgnoreCase));
However, note that matches will be a sequence of 0 or more matches. So, you can do,
if (matches.Any())
{
// I found something.
foreach (var match in matches)
{
var matchIndex = match.Index;
var matchPath = match.Path;
}
}
else
{
// Oops, no matches.
}
or, if you only want the first.
var firstMatchPath = matches.First().Path;
would do.

If you just want the first value if there is one then you can do this.
var value = mylist.FirstOrDefault(x=>x.EndsWith("file.txt"));
or if you want to do something with each matching string.
foreach (string value in mylist.Where(x=>x.EndsWith("file.txt")) )
{
// Do whatever you intend with value.
}
or if you want a list of the indices of the values, then you could try this.
var indexValues = new List<int>();
foreach (string value in mylist.Where(x=>x.EndsWith("file.txt")) )
{
indexValues.Add(mylist.IndexOf(value));
}

Use LINQ (assuming you havethe paths as strings):
var found = from f in fileList where f.equals("file.txt") select f;

Considering that you have path and file name is located at the end of path:
//List of elements
List<string> foundElements = myInitialList.Where(s => s.EndsWith("file.txt")).ToList();
//List of Indexes (base on lift of elements here above)
List<int> indexList = new List<int>();
foundElements.ForEach(f => indexList.Add(myInitialList.IndexOf(f)));

It's not clear from your question, but it seems that the list will contain file paths, but what you are looking for is a filename.
The following code will give you the index of the first occurrence of a file called "file.txt" in the list of path names, or -1 if it isn't in the list.
Note how this uses Path.GetFileName(). This is so that it will match "c:\dir1\dir2\file.txt" and not "c:\dir1\dir2\wrongfile.txt".
int index = filenames.FindIndex(filename => Path.GetFileName(filename).Equals("file.txt", StringComparison.OrdinalIgnoreCase));
However, if you are searching for an entire path then you can do it like this:
int index = filenames.FindIndex(filename => filename.Equals("file.txt", StringComparison.OrdinalIgnoreCase));
Note how we do a comparison of the entire filename rather than using "EndsWith".
If the filenames are already all lowercase and you are comparing entire paths then you can do a simpler search:
int index = filenames.IndexOf("file.txt");
Or if they are all uppercase you'd have to do:
int index = filenames.IndexOf("FILE.TXT");

Looping on IEnumerator<T>, Any Suggestions

I am having a situation where looping through the result of LINQ is getting on my nerves. Well here is my scenario:
I have a DataTable, that comes from database, from which I am taking data as:
var results = from d in dtAllData.AsEnumerable()
select new MyType
{
ID = d.Field<Decimal>("ID"),
Name = d.Field<string>("Name")
}
After doing the order by depending on the sort order as:
if(orderBy != "")
{
string[] ord = orderBy.Split(' ');
if (ord != null && ord.Length == 2 && ord[0] != "")
{
if (ord[1].ToLower() != "desc")
{
results = from sorted in results
orderby GetPropertyValue(sorted, ord[0])
select sorted;
}
else
{
results = from sorted in results
orderby GetPropertyValue(sorted, ord[0]) descending
select sorted;
}
}
}
The GetPropertyValue method is as:
private object GetPropertyValue(object obj, string property)
{
System.Reflection.PropertyInfo propertyInfo = obj.GetType().GetProperty(property);
return propertyInfo.GetValue(obj, null);
}
After this I am taking out 25 records for first page like:
results = from sorted in results
.Skip(0)
.Take(25)
select sorted;
So far things are going good, Now I have to pass this results to a method which is going to do some manipulation on the data and return me the desired data, here in this method when I want to loop these 25 records its taking a good enough time. My method definition is:
public MyTypeCollection GetMyTypes(IEnumerable<MyType> myData, String dateFormat, String offset)
I have tried foreach and it takes like 8-10 secs on my machine, it is taking time at this line:
foreach(var _data in myData)
I tried while loop and is doing same thing, I used it like:
var enumerator = myData.GetEnumerator();
while(enumerator.MoveNext())
{
int n = enumerator.Current;
Console.WriteLine(n);
}
This piece of code is taking time at MoveNext
Than I went for for loop like:
int length = myData.Count();
for (int i = 0; i < 25;i++ )
{
var temp = myData.ElementAt(i);
}
This code is taking time at ElementAt
Can anyone please guide me, what I am doing wrong. I am using Framework 3.5 in VS 2008.
Thanks in advance

EDIT: I suspect the problem is in how you're ordering. You're using reflection to first fetch and then invoke a property for every record. Even though you only want the first 25 records, it has to call GetPropertyValue on all the records first, in order to order them.
It would be much better if you could do this without reflection at all... but if you do need to use reflection, at least call Type.GetProperty() once instead of for every record.
(In some ways this is more to do with helping you diagnose the problem more easily than a full answer as such...)
As Henk said, this is very odd:
results = from sorted in results
.Skip(0)
.Take(25)
select sorted;
You almost certainly really just want:
results = results.Take(25);
(Skip(0) is pointless.)
It may not actually help, but it will make the code simpler to debug.
The next problem is that we can't actually see all your code. You've written:
After doing the order by depending on the sort order
... but you haven't shown how you're performing the ordering.
You should show us a complete example going from DataTable to its use.
Changing how you iterate over the sequence will not help - it's going to do the same thing either way, really - although it's surprising that in your last attempt, Count() apparently works quickly. Stick to the foreach - but work out exactly what that's going to be doing. LINQ uses a lot of lazy evaluation, and if you've done something which makes that very heavy going, that could be the problem. It's hard to know without seeing the whole pipeline.

The problem is that your "results" IEnumerable isn't actually being evaluated until it is passed into your method and enumerated. That means that the whole operation, getting all the data from dtAllData, selecting out the new type (which is happening on the whole enumerable, not just the first 25), and then finally the take 25 operation, are all happening on the first enumeration of the IEnumerable (foreach, while, whatever).
That's why your method is taking so long. It's actually doing some of the work defined elsewhere inside the method. If you want that to happen before your method, you could do a "ToList()" prior to the method.

You might find it easier to adopt a hybrid approach;
In order:
1) Sort your datatable in-situ. It's probably best to do this at the database level, but, if you can't, then DataTable.DefaultView.Sort is pretty efficient:
dtAllData.DefaultView.Sort = ord[0] + " " + ord[1];
This assumes that ord[0] is the column name, and ord[1] is either ASC or DESC
2) Page through the DefaultView by index:
int pageStart = 0;
List<DataRowView> pageRows = new List<DataRowView>();
for (int i = pageStart; i < dtAllData.DefaultView.Count; i++ )
{
if(pageStart + 25 > i || i == dtAllData.DefaultView.Count - 1) { break; //Exit if more than the number of pages or at the end of the rows }
pageRows.Add(dtAllData.DefaultView[i]);
}
...and create your objects from this much smaller list... (I've assumed the columns are called Id and Name, as well as the types)
List<MyType> myObjects = new List<MyType>();
foreach(DataRowView pageRow in pageRows)
{
myObjects.Add(new MyObject() { Id = Convert.ToInt32(pageRow["Id"]), Name = Convert.ToString(pageRow["Name"])});
}
You can then proceed with the rest of what you were doing.