I am trying to convert my search functionality to allow for fuzzy searches involving multiple words. My existing search code looks like:
// Split the search into seperate queries per word, and combine them into one major query
var finalQuery = new BooleanQuery();
string[] terms = searchString.Split(new[] { " " }, StringSplitOptions.RemoveEmptyEntries);
foreach (string term in terms)
{
// Setup the fields to search
string[] searchfields = new string[]
{
// Various strings denoting the document fields available
};
var parser = new MultiFieldQueryParser(Lucene.Net.Util.Version.LUCENE_29, searchfields, new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_29));
finalQuery.Add(parser.Parse(term), BooleanClause.Occur.MUST);
}
// Perform the search
var directory = FSDirectory.Open(new DirectoryInfo(LuceneIndexBaseDirectory));
var searcher = new IndexSearcher(directory, true);
var hits = searcher.Search(finalQuery, MAX_RESULTS);
This works correctly, and if I have an entity with the name field of "My name is Andrew", and I perform a search for "Andrew Name", Lucene correctly finds the correct document. Now I want to enable fuzzy searching, so that "Anderw Name" is found correctly. I changed my method to use the following code:
const int MAX_RESULTS = 10000;
const float MIN_SIMILARITY = 0.5f;
const int PREFIX_LENGTH = 3;
if (string.IsNullOrWhiteSpace(searchString))
throw new ArgumentException("Provided search string is empty");
// Split the search into seperate queries per word, and combine them into one major query
var finalQuery = new BooleanQuery();
string[] terms = searchString.Split(new[] { " " }, StringSplitOptions.RemoveEmptyEntries);
foreach (string term in terms)
{
// Setup the fields to search
string[] searchfields = new string[]
{
// Strings denoting document field names here
};
// Create a subquery where the term must match at least one of the fields
var subquery = new BooleanQuery();
foreach (string field in searchfields)
{
var queryTerm = new Term(field, term);
var fuzzyQuery = new FuzzyQuery(queryTerm, MIN_SIMILARITY, PREFIX_LENGTH);
subquery.Add(fuzzyQuery, BooleanClause.Occur.SHOULD);
}
// Add the subquery to the final query, but make at least one subquery match must be found
finalQuery.Add(subquery, BooleanClause.Occur.MUST);
}
// Perform the search
var directory = FSDirectory.Open(new DirectoryInfo(LuceneIndexBaseDirectory));
var searcher = new IndexSearcher(directory, true);
var hits = searcher.Search(finalQuery, MAX_RESULTS);
Unfortunately, with this code if I submit the search query "Andrew Name" (same as before) I get zero results back.
The core idea is that all terms must be found in at least one document field, but each term can reside in different fields. Does anyone have any idea why my rewritten query fails?
Final Edit: Ok it turns out I was over complicating this by a LOT, and there was no need to change from my first approach. After reverting back to the first code snippet, I enabled fuzzy searching by changing
finalQuery.Add(parser.Parse(term), BooleanClause.Occur.MUST);
to
finalQuery.Add(parser.Parse(term.Replace("~", "") + "~"), BooleanClause.Occur.MUST);
Your code works for me if I rewrite the searchString to lower-case. I'm assuming that you're using the StandardAnalyzer when indexing, and it will generate lower-case terms.
You need to 1) pass your tokens through the same analyzer (to enable identical processing), 2) apply the same logic as the analyzer or 3) use an analyzer which matches the processing you do (WhitespaceAnalyzer).
You want this line:
var queryTerm = new Term(term);
to look like this:
var queryTerm = new Term(field, term);
Right now you're searching field term (which probably doesn't exist) for the empty string (which will never be found).
Related
Using .NET 6.0 and Lucene.NET-4.8.0-beta00016 from NuGet
I am having an issue implementing the quickstart example from the website. When using TextField in a document, the field is not indexed. The search later in the BuildIndex method retrieves no results. If TextField is changed to StringField, the example works and the search returns a valid result.
Why does StringField work and TextField doesn't? I read that StringField is not analyzed but TextField is, so perhaps it's something to do with the StandardAnalyzer?
public class LuceneFullTextSearchService {
private readonly IndexWriter _writer;
private readonly Analyzer _standardAnalyzer;
public LuceneFullTextSearchService(string indexName)
{
// Compatibility version
const LuceneVersion luceneVersion = LuceneVersion.LUCENE_48;
string indexPath = Path.Combine(Environment.CurrentDirectory, indexName);
Directory indexDir = FSDirectory.Open(indexPath);
// Create an analyzer to process the text
_standardAnalyzer = new StandardAnalyzer(luceneVersion);
// Create an index writer
IndexWriterConfig indexConfig = new IndexWriterConfig(luceneVersion, _standardAnalyzer)
{
OpenMode = OpenMode.CREATE_OR_APPEND,
};
_writer = new IndexWriter(indexDir, indexConfig);
}
public void BuildIndex(string searchPath)
{
Document doc = new Document();
TextField docText = new TextField("title", "Apache", Field.Store.YES);
doc.Add(docText);
_writer.AddDocument(doc);
//Flush and commit the index data to the directory
_writer.Commit();
// Parse the user's query text
Query query = new TermQuery(new Term("title", "Apache"));
// Search
using DirectoryReader reader = _writer.GetReader(applyAllDeletes: true);
IndexSearcher searcher = new IndexSearcher(reader);
TopDocs topDocs = searcher.Search(query, n: 2);
// Show results
Document resultDoc = searcher.Doc(topDocs.ScoreDocs[0].Doc);
string title = resultDoc.Get("title");
}
}
StandardAnalyzer includes a LowerCaseFilter, so your text is stored in the index as lower-case.
However, when you build your query, the text you use is "Apache" rather than "apache", so it doesn't produce any hits.
// Parse the user's query text
Query query = new TermQuery(new Term("title", "Apache"));
Option 1
Lowercase your search term.
// Parse the user's query text
Query query = new TermQuery(new Term("title", "Apache".ToLowerInvariant()));
Option 2
Use a QueryParser with the same analyzer you use to build the index.
QueryParser parser = new QueryParser(luceneVersion, "title", _standardAnalyzer);
Query query = parser.Parse("Apache");
The Lucene.Net.QueryParser package contains several implementations (the above example uses the Lucene.Net.QueryParsers.Classic.QueryParser).
I have lucene documents with below structure
{
name : "A",
id :1
},
{
name : "B",
id :1
},
{
name : "C",
id :3
}
Now I have a collection like List which contains A, B. I wanted to select documents where name is A or B . So as per above lucene documents I should have documents A and B . I wanted to fetch these 2 documents with a single lucene call instead of multiple lucene calls for each document.
i tried with BooleanQuery and adding my search query in a loop but the search query did not return anything. if I hit lucene with single document it works and returns a single document.
Could anyone please suggest How I can retrieve all matching documents with a single query ?
I tried something like below
List<string> terms = new List<string>(){'A', 'B'};
var mainQuery = new BooleanQuery);
var parser = new QueryParser(Lucene.Net.Util.Version.LUCENE_30, "name", analyzer);
foreach (var term in terms)
{
var query = parser.Parse(term);
mainQuery.Add(query, Occur.MUST_NOT);
}
var hits = _searcher.Search(mainQuery, 1000);
Above query did not work and returns 0 result .
I am able to resolve this by my own.It's just a simple OR clause which is Occur.SHOULD
var booleanQuery = new BooleanQuery();
foreach (var term in terms)
{
var termQuery = new TermQuery(new Term("name", term ));
booleanQuery.Add(termQuery, Occur.SHOULD);
}
I have a table of Somethings. Those Somethings have a Name field. A User of my website can enter a forward or backward combination of that name and I need to return the correct something.
Example:
User enters: "name the".
I have a Something.Name of "the name".
I return that Something.
Now, the SQL would look like this:
SELECT * FROM Somethings
Where Name LIKE '%name the%'
OR Name LIKE '%the name%'
As you can see, I need to get all permutations of the string the user enters and then search for each one. I tried doing this in a loop, using Linq to Entities, like so:
var string[] arr = {"name the", "the name"};
foreach (var str in arr)
{
var matchingSomethings = db.Somethings
.Where(e => e.Name.IndexOf(str , StringComparison.InvariantCultureIgnoreCase) >= 0);
}
This worked really well, when the user entered two words. But, with five words (120 permutations), it became cumbersome and the query timed out for the user.
I then tried:
var matchingSomethings = db.Somethings
.Where(e => arr.Contains(e.Name));
You can clearly see the obvious flaw there, being that the a Something that had a Name of "the name of the Something" would not be matched in this query, because that name is not included in the snippet the user entered of "the name".
So how would I write that Linq to Entities query to get all possible Somethings where any permutation of the snippet the user entered is contained in the Name of the Something?
Look for each word individually and return the item that had all matching words. This should do it:
public void MatchSomethings(string input)
{
var data = new List<something>();
var wordsInName = input.Split(new[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
var match = data.Where(s => wordsInName.All(word => s.Name.Contains(word)));
}
I m using lucene.Net version 3.0.3. I want to do regular expression search. I tried the following code:
// code
String SearchExpression = "[DM]ouglas";
const int hitsLimit = 1000000;
//state the file location of the index
string indexFileLocation = IndexLocation;
Lucene.Net.Store.Directory dir = Lucene.Net.Store.FSDirectory.Open(indexFileLocation);
//create an index searcher that will perform the search
Lucene.Net.Search.IndexSearcher searcher = new Lucene.Net.Search.IndexSearcher(dir);
var analyzer = new WhitespaceAnalyzer();
var parser = new MultiFieldQueryParser(Lucene.Net.Util.Version.LUCENE_30, new[] {
Field_Content, }, analyzer);
Term t = new Term(Field_Content, SearchExpression);
RegexQuery scriptQuery = new RegexQuery(t);
string s = string.Format("{0}", SearchExpression);
var query = parser.Parse(s);
BooleanQuery booleanQuery = new BooleanQuery();
booleanQuery.Add(query, Occur.MUST);
var hits = searcher.Search(booleanQuery, null, hitsLimit, Sort.RELEVANCE).ScoreDocs;
foreach (var hit in hits)
{
var hitDocument = searcher.Doc(hit.Doc);
string contentValue = hitDocument.Get(Field_Content);
}
// end of code
When I try to search with patten "Do*uglas", I get the results.
But if I search with the pattern "[DM]ouglas]" it is giving me the following error:
"Cannot parse '[DM]ouglas': Encountered " "]" "] "" at line 1, column 3. Was expecting one of: "TO" ... <RANGEIN_QUOTED> ... <RANGEIN_GOOP> ...".
I also tried doing simple search pattern like ".ouglas" which should give me results, as I have "Douglas" in my text content.
Does anyone know how to do regular expression search using lucene.Net version 3.0.3?
The StandardQueryParser does not support regular expressions at all. It is, instead, attempting to interpret that portion of the query as a range query.
I you wish to use regexes to search, you will need to construct a RegexQuery manually. Note, that RegexQuery performance tends to be poor. You might be able to improve it by switching from JavaUtilRegexCapabilities to JakartaRegexpCapabilities.
I'm working on an EPiServer website using a Lucene.net based search engine.
I have a query for finding only pages with a certain pageTypeId. Now I want to do the opposite, I want to only find pages that is NOT a certain pageTypeId. Is that possible?
This is the code for creating a query to search only for pages with pageTypeId 1, 2 or 3:
public BooleanClause GetClause()
{
var booleanQuery = new BooleanQuery();
var typeIds = new List<string>();
typeIds.Add("1");
typeIds.Add("2");
typeIds.Add("3");
foreach (var id in this.typeIds)
{
var termQuery = new TermQuery(
new Term(IndexFieldNames.PageTypeId, id));
var clause = new BooleanClause(termQuery,
BooleanClause.Occur.SHOULD);
booleanQuery.Add(clause);
}
return new BooleanClause(booleanQuery,
BooleanClause.Occur.MUST);
}
I want instead to create a query where I search for pages that have a pageTypeId that is NOT "4".
I tried simply replacing "SHOULD" and "MUST" with "MUST_NOT", but that didn't work.
Thanks to #goalie7960 for replying so quickly. Here is my revised code for searching for anything except some selected page types. This search includes all documents except those with pageTypeId "1", "2" or "3":
public BooleanClause GetClause()
{
var booleanQuery = new BooleanQuery();
booleanQuery.Add(new MatchAllDocsQuery(),
BooleanClause.Occur.MUST);
var typeIds = new List<string>();
typeIds.Add("1");
typeIds.Add("2");
typeIds.Add("3");
foreach (var typeId in this.typeIds)
{
booleanQuery.Add(new TermQuery(
new Term(IndexFieldNames.PageTypeId, typeId)),
BooleanClause.Occur.MUST_NOT);
}
return new BooleanClause(booleanQuery,
BooleanClause.Occur.MUST);
}
Assuming all your docs have a pageTypeId you can try using a MatchAllDocsQuery and then a MUST_NOT to remove all the docs you want to skip. Something like this would work I think:
BooleanQuery subQuery = new BooleanQuery();
subQuery.Add(new MatchAllDocsQuery(), BooleanClause.Occur.MUST);
subQuery.Add(new TermQuery(new Term(IndexFieldNames.PageTypeId, "4")), BooleanClause.Occur.MUST_NOT);
return subQuery;