Develop Reference

lucene.net match if search term has no space

I'm using lucene.net to perform searchs in posts in my c# asp.net application, This is a sample document in my indexes:
var doc = new Document();
var title = new Field("Title", "the album hardwired to self-destruct released", Field.Store.YES, Field.Index.ANALYZED, Field.TermVector.WITH_POSITIONS_OFFSETS);
title.Boost = 5;
doc.Add(title);
var ns_title = new Field("NoSpace_Title", "thealbumhardwiredtoselfdesctructreleased", Field.Store.NO, Field.Index.ANALYZED, Field.TermVector.WITH_POSITIONS_OFFSETS);
ns_title.Boost = 5;
doc.Add(ns_title);
doc.Add(new Field("Body", "the body text of the post", Field.Store.YES, Field.Index.ANALYZED, Field.TermVector.WITH_POSITIONS_OFFSETS));
doc.Add(new Field("Id", "1", Field.Store.YES, Field.Index.NOT_ANALYZED));
writer.AddDocument(doc);
Problem:
if I search for self or destruct or self destruct I get hit.
if I search for selfdestruct I dont get a hit.
The search method:
var searchWords = s.Split(' ').ToList();
var directory = GetDirectory();
var reader = IndexReader.Open(directory, true);
var searcher = new IndexSearcher(reader);
var analyzer = new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30);
var parser = new MultiFieldQueryParser(Lucene.Net.Util.Version.LUCENE_30, "Title,NoSpace_Title,Body".Split(','), analyzer);
var booleanQuery = new BooleanQuery();
// Title:selfdestruct*NoSpace_Title:selfdestruct*Body:selfdestruct*
s = string.Join(" ", searchWords.Select(x => x.Contains("*") ? x : x + "*"));
Query query = parser.Parse(QueryParser.Escape(s));
query.Boost = 5;
booleanQuery.Add(query, Occur.SHOULD);
// Title:*selfdestruct*,NoSpace_Title:*selfdestruct*,Body:*selfdestruct*
// (I suppose this should work and get hit but it doesn't)
s = "*" + string.Join("", searchWords) + "*";
Query query2 = parser.Parse(QueryParser.Escape(s));
query2.Boost = 3;
booleanQuery.Add(query2, Occur.SHOULD);
// Title:selfdestruct~0.85 (fuzzy search)
s = string.Join(" ", searchWords.Select(x => x.Contains("~") ? x : x + "~0.85"));
Query query3 = parser.Parse(QueryParser.Escape(s));
booleanQuery.Add(query3, Occur.SHOULD);
var collector = TopScoreDocCollector.Create(1000, true);
searcher.Search(booleanQuery, collector);
var hits = collector.TopDocs().ScoreDocs;
var docs = hits.Select(x => searcher.Doc(x.Doc)).ToList();

You can support this by adding a ShingleFilter into your analyzer.
ShingleFilter will combine adjacent tokens into single tokens to facilitate searching for them without a space. By default, it will output Unigrams as well (that is, it will also maintain the single tokens). So, when you index "self-destruct", it will index the tokens "self", "destruct", and "selfdestruct".
An easy way to do this without creating your own custom analyzer, is to use ShingleAnalyzerWrapper:
var analyzer = new ShingleAnalyzerWrapper(
new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30),
2);

Lucene.net does not find matches correctly

I am new in lucene.net , in some searches that i had , i found that i can use lucene in my project ,
now i can not fix the bugs in my code .
Let me I explain in Code
First of all i create indexes like these
var strIndexDir = path;
Directory indexDir = FSDirectory.Open(new DirectoryInfo(strIndexDir));
Analyzer std = new StandardAnalyzer(global::Lucene.Net.Util.Version.LUCENE_30)
foreach (var res in resturant)
{
var doc = new Document();
restaurantName = new Field("Name",
res.Name, Field.Store.YES,
Field.Index.ANALYZED, Field.TermVector.YES);
var restaurantId = new Field("Id",
res.RestaurantId.ToString(), Field.Store.YES,
Field.Index.NO, Field.TermVector.NO);
var restaurantSlug = new Field("Slug",
res.Slug, Field.Store.YES,
Field.Index.NO, Field.TermVector.NO);
var restaurantAddress = new Field("Address",
res.Address ?? "empty", Field.Store.YES,
Field.Index.NOT_ANALYZED, Field.TermVector.YES);
var resturantType = new Field("Type",
"restaurant", Field.Store.YES,
Field.Index.NO, Field.TermVector.NO);
doc.Add(restaurantName);
doc.Add(restaurantId);
doc.Add(restaurantSlug);
doc.Add(restaurantAddress);
doc.Add(resturantType);
idxw.AddDocument(doc);
}
idxw.Optimize();
idxw.Close();
I Think with my self the indexing is ok , becuase i want just find the restaurant name and addresses
also for search query i use this way
string strIndexDir = path;
Analyzer std = new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30);
var indexReader = IndexReader.Open(FSDirectory.Open(path), readOnly: true);
var parserName =
new Lucene.Net.QueryParsers.QueryParser(Lucene.Net.Util.Version.LUCENE_30, "Name", std);
var parserAddress =
new Lucene.Net.QueryParsers.QueryParser(Lucene.Net.Util.Version.LUCENE_30, "Address", std);
var parserSlug =
new Lucene.Net.QueryParsers.QueryParser(Lucene.Net.Util.Version.LUCENE_30, "Slug", std);
var parserTitle =
new Lucene.Net.QueryParsers.QueryParser(Lucene.Net.Util.Version.LUCENE_30, "Title", std);
var searcher = new IndexSearcher(FSDirectory.Open(path));
using (var srchr = new IndexSearcher(IndexReader.Open(directory,true)))
{
var qryName = parserName.Parse(q);
var qryAddress = parserAddress.Parse(q);
var qrySlug = parserSlug.Parse(q);
var qrytitle = parserTitle.Parse(q);
var cllctr = TopScoreDocCollector.Create(10, true);
searcher.Search(qryName, cllctr);
searcher.Search(qryAddress, cllctr);
searcher.Search(qrySlug, cllctr);
searcher.Search(qrytitle, cllctr);
var hits = cllctr.TopDocs().ScoreDocs;
Now let me say where is the problem .
for example i search this key word (q="box") want to find the restaurant name that name is boxshaharkgharb and want use "box"
the problem is that hot is always 0 but when i type boxshaharkgharb for example (q="boxshaharkgharb") the the result is ok .
how can handel that

By using wildcard * you can force Lucene to search by fragment.
If you need to do this for all queries - you need to review your choice - as Lucene best performs using whole term searches. Reason for that is that by default wildcards turn into constant score queries, while term search uses relevancy to rank results.

How to Read Lucene.net returned query results

I can't figure out how to read the results returned from a Lucene.net query.
I have this code:
Initialization
var test = new Document();
test.Add(new Field("id", "1", Field.Store.YES, Field.Index.NOT_ANALYZED));
test.Add(new Field("title", "the title", Field.Store.YES, Field.Index.ANALYZED));
test.Add(new Field("body", "the body of the question", Field.Store.YES, Field.Index.ANALYZED));
string path = HttpRuntime.AppDomainAppPath + "\\LuceneIndex";
Lucene.Net.Store.Directory directory = FSDirectory.Open(new DirectoryInfo(path));
Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_30);
var writer = new IndexWriter(directory, analyzer, true, IndexWriter.MaxFieldLength.LIMITED);
writer.AddDocument(test);
writer.Optimize();
writer.Flush(true, true, true);
writer.Dispose();
directory.Dispose();
analyzer.Dispose();
Reading the data
Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_30);
QueryParser parser = new QueryParser(Version.LUCENE_30, "ti", analyzer);
string path = HttpRuntime.AppDomainAppPath + "\\LuceneIndex";
Lucene.Net.Store.Directory directory = FSDirectory.Open(new DirectoryInfo(path));
IndexSearcher searcher = new IndexSearcher(directory);
Query query = new TermQuery(new Term("body", "body"));
TopDocs docs = searcher.Search(query,5);
analyzer.Dispose();
searcher.Dispose();
I inspected the data in docs, but it doesn't contain the Id of the matched search results.

You can geht the results as follows:
IndexSearcher searcher = new IndexSearcher(directory);
Query query = new TermQuery(new Term("body", "body"));
TopDocs docs = searcher.Search(query,5);
// Get id and score for the top docs
ScoreDocs[] results = docs.ScoreDocs
foreach (ScoreDoc item in results)
{
// Get lucene docID
int luceneID = item.Doc
// Get actual document for the docID from index
Document doc = searcher.Doc(luceneID);
}
Lucene(.Net) has its own unique docIDs for the indexed documents, which is independent from the ID stored in your ID field. You can access the actual Document and its stored fields by calling searcher.Doc(int docID) or for an IndexReader you can call reader.Doc(int docID)

Lucene query not returning hit on standard analyzer

I have a filename thatfeelwhen.pdf that when I search for using words like "that" or "feel", I don't get a hit, when I do if I type "when" or the entire filename. I'm using a standard analyzer. How can I get the searcher for Lucene to match everything? My search queries seem to be matching on the content within the file but not in the filename.
public partial class _Default : Page
{
Directory finalDirectory = null;
Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_30);
Code below in other methods:
private static void AddTextToIndex(string filename, string pdfBody, IndexWriter writer)
{
Document doc = new Document();
doc.Add(new Field("fileName", filename.ToString(), Field.Store.YES, Field.Index.ANALYZED));
doc.Add(new Field("pdfBody", pdfBody.ToString(), Field.Store.NO, Field.Index.ANALYZED));
writer.AddDocument(doc);
}
private static Directory buildIndex(Analyzer analyzer)
{
string[] syllabusFiles = System.IO.Directory.GetFiles(#"C:\mywebsite\files\forms");
Directory directory = FSDirectory.Open(new DirectoryInfo(#"C:\mywebsite\files\LuceneIndex"));
var writer = new IndexWriter(directory, analyzer, true, IndexWriter.MaxFieldLength.LIMITED);
int j = 0;
while (j < syllabusFiles.Length)
{
string pdfTextExtracted = pdfText(syllabusFiles[j]);
string fileNameOnly = syllabusFiles[j].Replace("C:\\website\\files\\forms", "");
AddTextToIndex(fileNameOnly, pdfTextExtracted, writer);
j++;
}
writer.Optimize();
writer.Dispose();
return directory;
}
protected void txtBoxSearchPDF_Click(object sender, EventArgs e)
{
if (txtBoxSearchString.Text == "")
{
lblNoSearchString.Visible = true;
}
else if (txtBoxSearchString.Text == "build_index")
{
this.finalDirectory = buildIndex(this.analyzer);
}
else
{
//searching PDF text
lblNoSearchString.Visible = false;
StringBuilder sb = new StringBuilder();
this.finalDirectory = FSDirectory.Open(new DirectoryInfo(#"C:\mywebsite\files\LuceneIndex"));
IndexReader indexReader = IndexReader.Open(this.finalDirectory, true);
Searcher indexSearch = new IndexSearcher(indexReader);
string searchQuery = txtBoxSearchString.Text;
var fields = new[] { "fileName", "pdfBody" };
var queryParser = new MultiFieldQueryParser(Version.LUCENE_30, fields, this.analyzer);
Query query;
try
{
query = queryParser.Parse(searchQuery.Trim());
}
catch (ParseException)
{
query = queryParser.Parse(QueryParser.Escape(searchQuery.Trim()));
}
TopDocs resultDocs = indexSearch.Search(query, indexReader.MaxDoc);
var hits = resultDocs.ScoreDocs;
foreach (var hit in hits)
{
var documentFromSearcher = indexSearch.Doc(hit.Doc);
string getResult = documentFromSearcher.Get("fileName");
string formattedResult = getResult.Replace(" ", "%20");
sb.AppendLine(#"" + getResult+"");
sb.AppendLine("<br>");
}

I chose to use Analyzer analyzer = new SingleCharTokenAnalyzer(); and am getting much better results.
I tried Simple, Standard, Whitespace, and Keyword Analyzers and none were really suiting my needs without having to resort with creating extra work to customize them.

Lucene .NET search results

I'm using this code to index:
public void IndexEmployees(IEnumerable<Employee> employees)
{
var indexPath = GetIndexPath();
var directory = FSDirectory.Open(indexPath);
var indexWriter = new IndexWriter(directory, new StandardAnalyzer(Version.LUCENE_29), true, IndexWriter.MaxFieldLength.UNLIMITED);
foreach (var employee in employees)
{
var document = new Document();
document.Add(new Field("EmployeeId", employee.EmployeeId.ToString(), Field.Store.YES, Field.Index.NO, Field.TermVector.NO));
document.Add(new Field("Name", employee.FirstName + " " + employee.LastName, Field.Store.YES, Field.Index.ANALYZED, Field.TermVector.NO));
document.Add(new Field("OfficeName", employee.OfficeName, Field.Store.YES, Field.Index.ANALYZED, Field.TermVector.NO));
document.Add(new Field("CompetenceRatings", string.Join(" ", employee.CompetenceRatings.Select(cr => cr.Name)), Field.Store.YES, Field.Index.ANALYZED, Field.TermVector.NO));
indexWriter.AddDocument(document);
}
indexWriter.Optimize();
indexWriter.Close();
var indexReader = IndexReader.Open(directory, true);
var spell = new SpellChecker.Net.Search.Spell.SpellChecker(directory);
spell.ClearIndex();
spell.IndexDictionary(new LuceneDictionary(indexReader, "Name"));
spell.IndexDictionary(new LuceneDictionary(indexReader, "OfficeName"));
spell.IndexDictionary(new LuceneDictionary(indexReader, "CompetenceRatings"));
}
public DirectoryInfo GetIndexPath()
{
return new DirectoryInfo(HttpContext.Current.Server.MapPath("/App_Data/EmployeeIndex/"));
}
And this code to find results (as well as suggestions):
public SearchResult Search(DirectoryInfo indexPath, string[] searchFields, string searchQuery)
{
var directory = FSDirectory.Open(indexPath);
var standardAnalyzer = new StandardAnalyzer(Version.LUCENE_29);
var indexReader = IndexReader.Open(directory, true);
var indexSearcher = new IndexSearcher(indexReader);
var parser = new MultiFieldQueryParser(Version.LUCENE_29, searchFields, standardAnalyzer);
//parser.SetDefaultOperator(QueryParser.Operator.OR);
var query = parser.Parse(searchQuery);
var hits = indexSearcher.Search(query, null, 5000);
return new SearchResult
{
Suggestions = FindSuggestions(indexPath, searchQuery),
LuceneDocuments = hits
.scoreDocs
.Select(scoreDoc => indexSearcher.Doc(scoreDoc.doc))
.ToArray()
};
}
public string[] FindSuggestions(DirectoryInfo indexPath, string searchQuery)
{
var directory = FSDirectory.Open(indexPath);
var spell = new SpellChecker.Net.Search.Spell.SpellChecker(directory);
var similarWords = spell.SuggestSimilar(searchQuery, 20);
return similarWords;
}
var searchResult = Search(GetIndexPath(), new[] { "Name", "OfficeName", "CompetenceRatings" }, "admin*");
Simple queries like: admin or admin* doesnt give me any results. I know that there is an employee with that name. I want to be able to find James Jameson if I search for James.
Thanks!

First thing. You have to commit the changes to the index.
indexWriter.Optimize();
indexWriter.Commit(); //Add This
indexWriter.Close();
Edit#2
Also, keep it simple until you get something that works.
Comment this stuff out.
//var indexReader = IndexReader.Open(directory, true);
//var spell = new SpellChecker.Net.Search.Spell.SpellChecker(directory);
//spell.ClearIndex();
//spell.IndexDictionary(new LuceneDictionary(indexReader, "Name"));
//spell.IndexDictionary(new LuceneDictionary(indexReader, "OfficeName"));
//spell.IndexDictionary(new LuceneDictionary(indexReader, "CompetenceRatings"));
Edit#3
The fields you are searching are probably not going to change often. I would include them in your search function.
string[] fields = new string[] { "Name", "OfficeName", "CompetenceRatings" };
The biggest reason I suggest this is that Fields are case-sensitive and sometimes you wont get any results and it's because you search the "name" field (which doesn't exist) instead of the "Name" field. Easier to spot the mistake this way.

In my (limited) experience working with Lucene, I've found that you have to build up your own query in order to get "google" like behavior. Here is what I do, YMMV, but it generates expected results in my application. The basic idea is you combine a term query (exact match), a prefix query (anything that begins with the term), and a fuzzy query for each term in the search string. The code below won't compile, but gives you the idea
Query GetQuery(string querystring)
{
Search.Search.BooleanQuery query = new Search.Search.BooleanQuery();
Search.Analysis.TokenStream tk = StandardAnalyzerInstance.TokenStream(null, new StringReader(querystring));
Search.Analysis.Tokenattributes.TermAttribute ta = tk.GetAttribute(typeof(Search.Analysis.Tokenattributes.TermAttribute)) as Search.Analysis.Tokenattributes.TermAttribute;
while (tk.IncrementToken())
{
string term = ta.Term();
Search.Search.BooleanQuery bq = new Search.Search.BooleanQuery();
bq.Add(new Search.Search.TermQuery(new Search.Index.Term("fieldToQuery", term)), Search.Search.BooleanClause.Occur.SHOULD);
bq.Add(new Search.Search.PrefixQuery(new Search.Index.Term("fieldToQuery", term)), Search.Search.BooleanClause.Occur.SHOULD);
bq.Add(new Search.Search.FuzzyQuery(new Search.Index.Term("fieldToQuery", term)), Search.Search.BooleanClause.Occur.SHOULD);
query.Add(bq, Search.Search.BooleanClause.Occur.MUST);
}
return query;
}

That Parse() method is inherited. Have you tried utilizing the static methods that returns a Query object?
Parse(Version matchVersion, String[] queries, String[] fields, Analyzer analyzer)