I m using lucene.Net version 3.0.3. I want to do regular expression search. I tried the following code:
// code
String SearchExpression = "[DM]ouglas";
const int hitsLimit = 1000000;
//state the file location of the index
string indexFileLocation = IndexLocation;
Lucene.Net.Store.Directory dir = Lucene.Net.Store.FSDirectory.Open(indexFileLocation);
//create an index searcher that will perform the search
Lucene.Net.Search.IndexSearcher searcher = new Lucene.Net.Search.IndexSearcher(dir);
var analyzer = new WhitespaceAnalyzer();
var parser = new MultiFieldQueryParser(Lucene.Net.Util.Version.LUCENE_30, new[] {
Field_Content, }, analyzer);
Term t = new Term(Field_Content, SearchExpression);
RegexQuery scriptQuery = new RegexQuery(t);
string s = string.Format("{0}", SearchExpression);
var query = parser.Parse(s);
BooleanQuery booleanQuery = new BooleanQuery();
booleanQuery.Add(query, Occur.MUST);
var hits = searcher.Search(booleanQuery, null, hitsLimit, Sort.RELEVANCE).ScoreDocs;
foreach (var hit in hits)
{
var hitDocument = searcher.Doc(hit.Doc);
string contentValue = hitDocument.Get(Field_Content);
}
// end of code
When I try to search with patten "Do*uglas", I get the results.
But if I search with the pattern "[DM]ouglas]" it is giving me the following error:
"Cannot parse '[DM]ouglas': Encountered " "]" "] "" at line 1, column 3. Was expecting one of: "TO" ... <RANGEIN_QUOTED> ... <RANGEIN_GOOP> ...".
I also tried doing simple search pattern like ".ouglas" which should give me results, as I have "Douglas" in my text content.
Does anyone know how to do regular expression search using lucene.Net version 3.0.3?
The StandardQueryParser does not support regular expressions at all. It is, instead, attempting to interpret that portion of the query as a range query.
I you wish to use regexes to search, you will need to construct a RegexQuery manually. Note, that RegexQuery performance tends to be poor. You might be able to improve it by switching from JavaUtilRegexCapabilities to JakartaRegexpCapabilities.
Related
Using .NET 6.0 and Lucene.NET-4.8.0-beta00016 from NuGet
I am having an issue implementing the quickstart example from the website. When using TextField in a document, the field is not indexed. The search later in the BuildIndex method retrieves no results. If TextField is changed to StringField, the example works and the search returns a valid result.
Why does StringField work and TextField doesn't? I read that StringField is not analyzed but TextField is, so perhaps it's something to do with the StandardAnalyzer?
public class LuceneFullTextSearchService {
private readonly IndexWriter _writer;
private readonly Analyzer _standardAnalyzer;
public LuceneFullTextSearchService(string indexName)
{
// Compatibility version
const LuceneVersion luceneVersion = LuceneVersion.LUCENE_48;
string indexPath = Path.Combine(Environment.CurrentDirectory, indexName);
Directory indexDir = FSDirectory.Open(indexPath);
// Create an analyzer to process the text
_standardAnalyzer = new StandardAnalyzer(luceneVersion);
// Create an index writer
IndexWriterConfig indexConfig = new IndexWriterConfig(luceneVersion, _standardAnalyzer)
{
OpenMode = OpenMode.CREATE_OR_APPEND,
};
_writer = new IndexWriter(indexDir, indexConfig);
}
public void BuildIndex(string searchPath)
{
Document doc = new Document();
TextField docText = new TextField("title", "Apache", Field.Store.YES);
doc.Add(docText);
_writer.AddDocument(doc);
//Flush and commit the index data to the directory
_writer.Commit();
// Parse the user's query text
Query query = new TermQuery(new Term("title", "Apache"));
// Search
using DirectoryReader reader = _writer.GetReader(applyAllDeletes: true);
IndexSearcher searcher = new IndexSearcher(reader);
TopDocs topDocs = searcher.Search(query, n: 2);
// Show results
Document resultDoc = searcher.Doc(topDocs.ScoreDocs[0].Doc);
string title = resultDoc.Get("title");
}
}
StandardAnalyzer includes a LowerCaseFilter, so your text is stored in the index as lower-case.
However, when you build your query, the text you use is "Apache" rather than "apache", so it doesn't produce any hits.
// Parse the user's query text
Query query = new TermQuery(new Term("title", "Apache"));
Option 1
Lowercase your search term.
// Parse the user's query text
Query query = new TermQuery(new Term("title", "Apache".ToLowerInvariant()));
Option 2
Use a QueryParser with the same analyzer you use to build the index.
QueryParser parser = new QueryParser(luceneVersion, "title", _standardAnalyzer);
Query query = parser.Parse("Apache");
The Lucene.Net.QueryParser package contains several implementations (the above example uses the Lucene.Net.QueryParsers.Classic.QueryParser).
I use lucene for searching.
For each doc in index I have some field called "uniqueIdentifier" with type string.
When I want to find all items with "uniqueIdentifier" == "haaglanden", I use the next code:
var searcher = Examine.ExamineManager.Instance.SearchProviderCollection["RegionsSearcher"];
var searchCriteria = searcher.CreateSearchCriteria(BooleanOperation.And);
var temp = searchCriteria.RawQuery("+uniqueIdentifier:" + uniqueIdentifier);
In temp I see :
LuceneQuery: {+(+uniqueIdentifier:haagland)}
But "haagland" != "haaglanden".
And I can not find my docs.
How can I build query with "haaglanden"?
The cause was in analyzer.
Swiched Lucene.Net.Analysis.Nl.DutchAnalyzer to Lucene.Net.Analysis.Standard.StandardAnalyzer.
I am passing a search query to the Lucene QueryParser.Parse(string query) method, and then passing the result to Searcher.Search(Query query, int n).
A string of:
"system cleaner"
returns 1 hit.
A string of:
"system clean*"
or:
"system clean\*"
returns 0 hits.
How can I provide a search query that uses both a quoted phrase and a wildcard?
The QueryParser doesn't support that. You can construct such a query using the SpanQuery API:
SpanQuery firstwordQuery = new SpanTermQuery(new Term("myField", "system"));
//Unfortunately, Lucene.Net doesn't have SpanMultiTermQueryWrapper...
SpanQuery secondwordQuery = new SpanRegexQuery(new Term("myField", "clean.*"));
SpanQuery[] spanClauses = new SpanQuery[] {firstwordQuery, secondwordQuery};
Query finalQuery = new SpanNearQuery(spanClauses, 0, true);
I'm working on an EPiServer website using a Lucene.net based search engine.
I have a query for finding only pages with a certain pageTypeId. Now I want to do the opposite, I want to only find pages that is NOT a certain pageTypeId. Is that possible?
This is the code for creating a query to search only for pages with pageTypeId 1, 2 or 3:
public BooleanClause GetClause()
{
var booleanQuery = new BooleanQuery();
var typeIds = new List<string>();
typeIds.Add("1");
typeIds.Add("2");
typeIds.Add("3");
foreach (var id in this.typeIds)
{
var termQuery = new TermQuery(
new Term(IndexFieldNames.PageTypeId, id));
var clause = new BooleanClause(termQuery,
BooleanClause.Occur.SHOULD);
booleanQuery.Add(clause);
}
return new BooleanClause(booleanQuery,
BooleanClause.Occur.MUST);
}
I want instead to create a query where I search for pages that have a pageTypeId that is NOT "4".
I tried simply replacing "SHOULD" and "MUST" with "MUST_NOT", but that didn't work.
Thanks to #goalie7960 for replying so quickly. Here is my revised code for searching for anything except some selected page types. This search includes all documents except those with pageTypeId "1", "2" or "3":
public BooleanClause GetClause()
{
var booleanQuery = new BooleanQuery();
booleanQuery.Add(new MatchAllDocsQuery(),
BooleanClause.Occur.MUST);
var typeIds = new List<string>();
typeIds.Add("1");
typeIds.Add("2");
typeIds.Add("3");
foreach (var typeId in this.typeIds)
{
booleanQuery.Add(new TermQuery(
new Term(IndexFieldNames.PageTypeId, typeId)),
BooleanClause.Occur.MUST_NOT);
}
return new BooleanClause(booleanQuery,
BooleanClause.Occur.MUST);
}
Assuming all your docs have a pageTypeId you can try using a MatchAllDocsQuery and then a MUST_NOT to remove all the docs you want to skip. Something like this would work I think:
BooleanQuery subQuery = new BooleanQuery();
subQuery.Add(new MatchAllDocsQuery(), BooleanClause.Occur.MUST);
subQuery.Add(new TermQuery(new Term(IndexFieldNames.PageTypeId, "4")), BooleanClause.Occur.MUST_NOT);
return subQuery;
I am trying to convert my search functionality to allow for fuzzy searches involving multiple words. My existing search code looks like:
// Split the search into seperate queries per word, and combine them into one major query
var finalQuery = new BooleanQuery();
string[] terms = searchString.Split(new[] { " " }, StringSplitOptions.RemoveEmptyEntries);
foreach (string term in terms)
{
// Setup the fields to search
string[] searchfields = new string[]
{
// Various strings denoting the document fields available
};
var parser = new MultiFieldQueryParser(Lucene.Net.Util.Version.LUCENE_29, searchfields, new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_29));
finalQuery.Add(parser.Parse(term), BooleanClause.Occur.MUST);
}
// Perform the search
var directory = FSDirectory.Open(new DirectoryInfo(LuceneIndexBaseDirectory));
var searcher = new IndexSearcher(directory, true);
var hits = searcher.Search(finalQuery, MAX_RESULTS);
This works correctly, and if I have an entity with the name field of "My name is Andrew", and I perform a search for "Andrew Name", Lucene correctly finds the correct document. Now I want to enable fuzzy searching, so that "Anderw Name" is found correctly. I changed my method to use the following code:
const int MAX_RESULTS = 10000;
const float MIN_SIMILARITY = 0.5f;
const int PREFIX_LENGTH = 3;
if (string.IsNullOrWhiteSpace(searchString))
throw new ArgumentException("Provided search string is empty");
// Split the search into seperate queries per word, and combine them into one major query
var finalQuery = new BooleanQuery();
string[] terms = searchString.Split(new[] { " " }, StringSplitOptions.RemoveEmptyEntries);
foreach (string term in terms)
{
// Setup the fields to search
string[] searchfields = new string[]
{
// Strings denoting document field names here
};
// Create a subquery where the term must match at least one of the fields
var subquery = new BooleanQuery();
foreach (string field in searchfields)
{
var queryTerm = new Term(field, term);
var fuzzyQuery = new FuzzyQuery(queryTerm, MIN_SIMILARITY, PREFIX_LENGTH);
subquery.Add(fuzzyQuery, BooleanClause.Occur.SHOULD);
}
// Add the subquery to the final query, but make at least one subquery match must be found
finalQuery.Add(subquery, BooleanClause.Occur.MUST);
}
// Perform the search
var directory = FSDirectory.Open(new DirectoryInfo(LuceneIndexBaseDirectory));
var searcher = new IndexSearcher(directory, true);
var hits = searcher.Search(finalQuery, MAX_RESULTS);
Unfortunately, with this code if I submit the search query "Andrew Name" (same as before) I get zero results back.
The core idea is that all terms must be found in at least one document field, but each term can reside in different fields. Does anyone have any idea why my rewritten query fails?
Final Edit: Ok it turns out I was over complicating this by a LOT, and there was no need to change from my first approach. After reverting back to the first code snippet, I enabled fuzzy searching by changing
finalQuery.Add(parser.Parse(term), BooleanClause.Occur.MUST);
to
finalQuery.Add(parser.Parse(term.Replace("~", "") + "~"), BooleanClause.Occur.MUST);
Your code works for me if I rewrite the searchString to lower-case. I'm assuming that you're using the StandardAnalyzer when indexing, and it will generate lower-case terms.
You need to 1) pass your tokens through the same analyzer (to enable identical processing), 2) apply the same logic as the analyzer or 3) use an analyzer which matches the processing you do (WhitespaceAnalyzer).
You want this line:
var queryTerm = new Term(term);
to look like this:
var queryTerm = new Term(field, term);
Right now you're searching field term (which probably doesn't exist) for the empty string (which will never be found).