Develop Reference

Lucene 4.8 facets usage

I have difficulties understanding this example on how to use facets :
https://lucenenet.apache.org/docs/4.8.0-beta00008/api/Lucene.Net.Demo/Lucene.Net.Demo.Facet.SimpleFacetsExample.html
My goal is to create an index in which each document field have a facet, so that at search time i can choose which facets use to navigate data.
What i am confused about is setup of facets in index creation, to
summarize my question : is index with facets compatibile with
ReferenceManager?
Need DirectoryTaxonomyWriter to be actually written and persisted
on disk or it will embedded into the index itself and is just
temporary? I mean given the code
indexWriter.AddDocument(config.Build(taxoWriter, doc)); of the
example i expect it's temporary and will be embedded into the index (but then the example also show you need the Taxonomy to drill down facet). So can the Taxonomy be tangled in some way with the index so that the are handled althogeter with ReferenceManager?
If is not may i just use the same folder i use for storing index?
Here is a more detailed list of point that confuse me :
In my scenario i am indexing the document asyncrhonously (background process) and then fetching the indext ASAP throught ReferenceManager in ASP.NET application. I hope this way to fetch the index is compatibile with DirectoryTaxonomyWriter needed by facets.
Then i modified the code i write introducing the taxonomy writer as indicated in the example, but i am a bit confused, seems like i can't store DirectoryTaxonomyWriter into the same folder of index because the folder is locked, need i to persist it or it will be embedded into the index (so a RAMDirectory is enougth)? if i need to persist it in a different direcotry, can i safely persist it into subdirectory?
Here the code i am actually using :
private static void BuildIndex (IndexEntry entry)
{
string targetFolder = ConfigurationManager.AppSettings["IndexFolder"] ?? string.Empty;
//** LOG
if (System.IO.Directory.Exists(targetFolder) == false)
{
string message = #"Index folder not found";
_fileLogger.Error(message);
_consoleLogger.Error(message);
return;
}
var metadata = JsonConvert.DeserializeObject<IndexMetadata>(File.ReadAllText(entry.MetdataPath) ?? "{}");
string[] header = new string[0];
List<dynamic> csvRecords = new List<dynamic>();
using (var reader = new StreamReader(entry.DataPath))
{
CsvConfiguration csvConfiguration = new CsvConfiguration(CultureInfo.InvariantCulture);
csvConfiguration.AllowComments = false;
csvConfiguration.CountBytes = false;
csvConfiguration.Delimiter = ",";
csvConfiguration.DetectColumnCountChanges = false;
csvConfiguration.Encoding = Encoding.UTF8;
csvConfiguration.HasHeaderRecord = true;
csvConfiguration.IgnoreBlankLines = true;
csvConfiguration.HeaderValidated = null;
csvConfiguration.MissingFieldFound = null;
csvConfiguration.TrimOptions = CsvHelper.Configuration.TrimOptions.None;
csvConfiguration.BadDataFound = null;
using (var csvReader = new CsvReader(reader, csvConfiguration))
{
csvReader.Read();
csvReader.ReadHeader();
csvReader.Read();
header = csvReader.HeaderRecord;
csvRecords = csvReader.GetRecords<dynamic>().ToList();
}
}
string targetDirectory = Path.Combine(targetFolder, "Index__" + metadata.Boundle + "__" + DateTime.Now.ToString("yyyyMMdd_HHmmss") + "__" + Path.GetRandomFileName().Substring(0, 6));
System.IO.Directory.CreateDirectory(targetDirectory);
//** LOG
{
string message = #"..creating index : {0}";
_fileLogger.Information(message, targetDirectory);
_consoleLogger.Information(message, targetDirectory);
}
using (var dir = FSDirectory.Open(targetDirectory))
{
using (DirectoryTaxonomyWriter taxoWriter = new DirectoryTaxonomyWriter(dir))
{
Analyzer analyzer = metadata.GetAnalyzer();
var indexConfig = new IndexWriterConfig(LuceneVersion.LUCENE_48, analyzer);
using (IndexWriter writer = new IndexWriter(dir, indexConfig))
{
long entryNumber = csvRecords.Count();
long index = 0;
long lastPercentage = 0;
foreach (dynamic csvEntry in csvRecords)
{
Document doc = new Document();
IDictionary<string, object> dynamicCsvEntry = (IDictionary<string, object>)csvEntry;
var indexedMetadataFiled = metadata.IdexedFields;
foreach (string headField in header)
{
if (indexedMetadataFiled.ContainsKey(headField) == false || (indexedMetadataFiled[headField].NeedToBeIndexed == false && indexedMetadataFiled[headField].NeedToBeStored == false))
continue;
var field = new Field(headField,
((string)dynamicCsvEntry[headField] ?? string.Empty).ToLower(),
indexedMetadataFiled[headField].NeedToBeStored ? Field.Store.YES : Field.Store.NO,
indexedMetadataFiled[headField].NeedToBeIndexed ? Field.Index.ANALYZED : Field.Index.NO
);
doc.Add(field);
var facetField = new FacetField(headField, (string)dynamicCsvEntry[headField]);
doc.Add(facetField);
}
long percentage = (long)(((decimal)index / (decimal)entryNumber) * 100m);
if (percentage > lastPercentage && percentage % 10 == 0)
{
_consoleLogger.Information($"..indexing {percentage}%..");
lastPercentage = percentage;
}
writer.AddDocument(doc);
index++;
}
writer.Commit();
}
}
}
//** LOG
{
string message = #"Index Created : {0}";
_fileLogger.Information(message, targetDirectory);
_consoleLogger.Information(message, targetDirectory);
}
}

Trying to query AD users, but getting less than expected results

I am trying to query AD to get the list of users
The query is success, and I am getting 90 records as results, the problem is that I am expecting more over than 15000 records.
Before I go to the IT and check the right domain path, it would be great if I am sure that my code is not the issue.
Thank you for every much
Already tried changing the search page size
DataTable resultSet = new DataTable();
resultSet.Columns.Add(new DataColumn("mail", typeof (string)));
resultSet.Columns.Add(new DataColumn("username", typeof (string)));
resultSet.Columns.Add(new DataColumn("displayname", typeof (string)));
try
{
string DomainPath = "LDAP://MyDomain";
DirectoryEntry searchRoot = new DirectoryEntry(DomainPath);
DirectorySearcher search = new DirectorySearcher(searchRoot);
search.Filter = "(&(objectClass=user)(objectCategory=person))";
search.PropertiesToLoad.Add("samaccountname");
search.PropertiesToLoad.Add("mail");
search.PageSize = 1001;
search.PropertiesToLoad.Add("displayname");//first name
SearchResult result;
SearchResultCollection resultCol = search.FindAll();
if (resultCol != null)
{
for (int counter = 0; counter < resultCol.Count; counter++)
{
result = resultCol[counter];
if (result.Properties.Contains("samaccountname") && result.Properties.Contains("mail") && result.Properties.Contains("displayname"))
{
var row = resultSet.NewRow();
resultSet.Rows.Add(row);
var Email = (String) result.Properties["mail"][0];
row["mail"] = Email;
var UserName = (String) result.Properties["samaccountname"][0];
row["username"] = UserName;
var DisplayName = (String) result.Properties["displayname"][0];
row["displayname"] = DisplayName;
row.AcceptChanges();
}
}
resultCol.Dispose();
}
}
catch (Exception ex)
{
App.Log.Write("Exception while read user from AD: " + ex.ToString());
}
return resultSet;
Expected over 15000 records getting 90 only

C# Solr query - how to only get future dates?

I have a search method that queries Solr for event items. I need to modify it to only get events where the date has not already passed (i.e. Where(x => x.EventDate.Date >= DateTime.Now.Date), but I'm not sure how to add this because I'm not very familiar with Solr. Here's my search function:
public SearchQueryResults Search(string keywords, int page,int perPage, List<Guid> contentTypeFilters, List<Guid> otherFilters, ISortBuilder<SearchResultItem> sortBuilder)
{
var searchFilters = new List<IPredicateBuilder<SearchResultItem>>()
{
new IsSearchablePredicateBuilder()
};
if (contentTypeFilters.Any())
{
var contentTypePredicateBuilder = new ContentTypePredicateBuilder();
contentTypePredicateBuilder.ContentTypes = contentTypeFilters;
searchFilters.Add(contentTypePredicateBuilder);
}
if (otherFilters.Any())
{
var tagFilterBuilder = new TagsAndPredicateBuilder(otherFilters,_sitecoreContext);
searchFilters.Add(tagFilterBuilder);
}
if (string.IsNullOrWhiteSpace(keywords))
{
keywords = "";
}
SearchRequest searchRequest = new SearchRequest();
var queryParams = new Dictionary<string, string>() { };
queryParams.Add("q", keywords);
searchRequest.QueryParameters = queryParams;
searchRequest.SortBy = "";
searchRequest.SortOrder = "";
SearchQuery<SearchResultItem> queryArguments = new SearchQuery<SearchResultItem>();
queryArguments.FilterBuilders = searchFilters;
queryArguments.Page = page;
queryArguments.PerPage = perPage;
queryArguments.FacetsBuilder = new SearchFacetBuilder<SearchResultItem>();
queryArguments.SearchRequest = searchRequest;
queryArguments.IndexName = _indexName;
if (string.IsNullOrWhiteSpace(keywords))
{
queryArguments.QueryBuilders =new List<IPredicateBuilder<SearchResultItem>>();
}
else
{
queryArguments.QueryBuilders = new[] { new KeywordPredicateBuilder<SearchResultItem>(new[] { keywords }) };
}
queryArguments.SortBuilder = sortBuilder;
try
{
var results = _searchManager.GetResults<SearchResultItem>(queryArguments);
SearchQueryResults queryResults = new SearchQueryResults();
queryResults.ResultItems = results.Results;
queryResults.CurrentPage = page;
queryResults.TotalResults = Int32.Parse(results.TotalResults.ToString());
queryResults.TotalPages = (queryResults.TotalResults + perPage - 1) / perPage; ;
return queryResults;
}
catch (Exception exc)
{
Sitecore.Diagnostics.Log.Error("Error with FilteredSearch, could be a loss of connection to the SOLR server: " + exc.Message, this);
return null;
}
}
and here is how it's being called:
Results = _searchService.Search(searchTerm, CurrentPage - 1, 10, contentTypes, searchFilters,
new GenericSortBuilder<SearchResultItem>(q => q.OrderByDescending(r => r.SearchDate)));
How do I add in date filtering so that it only returns items where the date is in the future?

I would add filter query to the list of existing ones filtering the date field. On the documentation page, I was able to find information about fluent API, which could help here
Query.Field("date").From(DateTime.Now)
I'm not C# developer, that this code could have some mistakes, but I think the main idea is clear what needs to be done.

Conditional search with lucene.net

the below way i did the search with lucene.net. this routine search multiple word against all index field called "Title", "Description", "Url", "Country" .
i need to know how can i give a condition like where country='UK' or country='US'
i want that multiple word should be search like below but i want to add one more clause that when country is UK. so please guide me what to add in my code.
if (!string.IsNullOrEmpty(multiWordPhrase))
{
string[] fieldList = { "Title", "Description", "Url" };
List<BooleanClause.Occur> occurs = new List<BooleanClause.Occur>();
foreach (string field in fieldList)
{
occurs.Add(BooleanClause.Occur.SHOULD);
}
searcher = new IndexSearcher(_directory, false);
Query qry = MultiFieldQueryParser.Parse(Version.LUCENE_29, multiWordPhrase, fieldList, occurs.ToArray(), new StandardAnalyzer(Version.LUCENE_29));
TopDocs topDocs = searcher.Search(qry, null, ((PageIndex + 1) * PageSize), Sort.RELEVANCE);
ScoreDoc[] scoreDocs = topDocs.ScoreDocs;
int resultsCount = topDocs.TotalHits;
list.HasData = resultsCount;
StartRecPos = (PageIndex * PageSize) + 1;
if (topDocs != null)
{
for (int i = (PageIndex * PageSize); i <= (((PageIndex + 1) * PageSize)-1) && i < topDocs.ScoreDocs.Length; i++)
{
Document doc = searcher.Doc(topDocs.ScoreDocs[i].doc);
oSr = new Result();
oSr.ID = doc.Get("ID");
oSr.Title = doc.Get("Title");
oSr.Description = doc.Get("Description");
//oSr.WordCount = AllExtension.WordCount(oSr.Description, WordExist(oSr.Title, multiWordPhrase));
string preview =
oSr.Description = BBAReman.AllExtension.HighlightKeywords(oSr.Description, multiWordPhrase); //sr.Description;
oSr.Url = doc.Get("Url");
TmpEndRecpos++;
list.Result.Add(oSr);
}
}
thanks

Look up BooleanQuery
if (!string.IsNullOrEmpty(multiWordPhrase))
{
BooleanQuery bq = new BooleanQuery();
string[] fieldList = { "Title", "Description", "Url" };
List<BooleanClause.Occur> occurs = new List<BooleanClause.Occur>();
foreach (string field in fieldList)
{
occurs.Add(BooleanClause.Occur.SHOULD);
}
Query qry = MultiFieldQueryParser.Parse(Version.LUCENE_29, multiWordPhrase, fieldList, occurs.ToArray(), new StandardAnalyzer(Version.LUCENE_29));
bq.Add(qry,BooleanClause.Occur.Must);
//this is the country query (modify the Country field name to whatever you have)
string country = "UK";
Query q2 = new QueryParser(Version.LUCENE_CURRENT, "Country", analyzer).parse(country);
bq.Add(q2,BooleanClause.Occur.Must);
searcher = new IndexSearcher(_directory, false);
TopDocs topDocs = searcher.Search(bq, null, ((PageIndex + 1) * PageSize), Sort.RELEVANCE);
ScoreDoc[] scoreDocs = topDocs.ScoreDocs;
int resultsCount = topDocs.TotalHits;
list.HasData = resultsCount;
StartRecPos = (PageIndex * PageSize) + 1;
if (topDocs != null)
{
//loop through your results
}

Not getting the value of the hit field?

I am doing a project on search using Lucene.Net. We have created an index which contains 100 000 documents with 5 fields. But while searching I'm unable to track my correct record. Can anybody help me? Why is that so?
My code looks like this
List<int> ids = new List<int>();
List<Hits> hitList = new List<Hits>();
List<Document> results = new List<Document>();
int startPage = (pageIndex.Value - 1) * pageSize.Value;
string indexFileLocation = #"c:\\ResourceIndex\\"; //Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData), "ResourceIndex");
var fsDirectory = FSDirectory.Open(new DirectoryInfo(indexFileLocation));
Analyzer analyzer = new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_29);
IndexReader indexReader = IndexReader.Open(fsDirectory, true);
Searcher indexSearch = new IndexSearcher(indexReader);
//ids.AddRange(this.SearchPredicates(indexSearch, startPage, pageSize, query));
/*Searching From the ResourceIndex*/
Query resourceQuery = MultiFieldQueryParser.Parse(Lucene.Net.Util.Version.LUCENE_29,
new string[] { productId.ToString(), languagelds, query },
new string[] { "productId", "resourceLanguageIds", "externalIdentifier" },
analyzer);
TermQuery descriptionQuery = new TermQuery(new Term("description", '"'+query+'"'));
//TermQuery identifierQuery = new TermQuery(new Term("externalIdentifier", query));
BooleanQuery filterQuery = new BooleanQuery();
filterQuery.Add(descriptionQuery, BooleanClause.Occur.MUST);
//filterQuery.Add(identifierQuery,BooleanClause.Occur.MUST_NOT);
Filter filter = new CachingWrapperFilter(new QueryWrapperFilter(filterQuery));
TopScoreDocCollector collector = TopScoreDocCollector.create(100, true);
//Hits resourceHit = indexSearch.Search(resourceQuery, filter);
indexSearch.Search(resourceQuery, filter, collector);
ScoreDoc[] hits = collector.TopDocs().scoreDocs;
//for (int i = startPage; i <= pageSize && i < resourceHit.Length(); i++)
//{
// ids.Add(Convert.ToInt32(resourceHit.Doc(i).GetField("id")));
//}
for (int i = 0; i < hits.Length; i++)
{
int docId = hits[i].doc;
float score = hits[i].score;
Lucene.Net.Documents.Document doc = indexSearch.Doc(docId);
string result = "Score: " + score.ToString() +
" Field: " + doc.Get("id");
}

You're calling Document.Get("id"), which returns the value of a stored field. It wont work without Field.Store.YES when indexing.
You could use the FieldCache if you've got the field indexed without analyzing (Field.Index.NOT_ANALYZED) or using the KeywordAnalyzer. (Meaning one term per field and document.)
You'll need to use the innermost reader for the FieldCache to work optimally. Here's a code paste from FieldCache with frequently updated index which uses the FieldCache in a proper way, reading an integer value from the id field.
// Demo values, use existing code somewhere here.
var directory = FSDirectory.Open(new DirectoryInfo("index"));
var reader = IndexReader.Open(directory, readOnly: true);
var documentId = 1337;
// Grab all subreaders.
var subReaders = new List<IndexReader>();
ReaderUtil.GatherSubReaders(subReaders, reader);
// Loop through all subreaders. While subReaderId is higher than the
// maximum document id in the subreader, go to next.
var subReaderId = documentId;
var subReader = subReaders.First(sub => {
if (sub.MaxDoc() < subReaderId) {
subReaderId -= sub.MaxDoc();
return false;
}
return true;
});
var values = FieldCache_Fields.DEFAULT.GetInts(subReader, "id");
var value = values[subReaderId];