Develop Reference

How to highlight only results of PrefixQuery in Lucene and not whole words?

I'm fairly new to Lucene and perhaps doing something really wrong, so please correct me if it is the case. Being searching for the answer for a few days now and not sure where to go from here.
The goal is to use Lucene.NET to search for user names with partial search (like StartsWith) and highlight only the found parts. For instance if I search for abc in a list of ['a', 'ab', 'abc', 'abcd', 'abcde'] it should return just the last three in a form of ['<b>abc</b>', '<b>abc</b>d', '<b>abc</b>de']
Here is how I approached this.
First the index creation:
using var indexDir = FSDirectory.Open(Path.Combine(IndexDirectory, IndexName));
using var standardAnalyzer = new StandardAnalyzer(CurrentVersion);
var indexConfig = new IndexWriterConfig(CurrentVersion, standardAnalyzer);
indexConfig.OpenMode = OpenMode.CREATE_OR_APPEND;
using var indexWriter = new IndexWriter(indexDir, indexConfig);
if (indexWriter.NumDocs == 0)
{
//fill the index with Documents
}
The documents are created like this:
static Document BuildClientDocument(int id, string surname, string name)
{
var document = new Document()
{
new StringField("Id", id.ToString(), Field.Store.YES),
new TextField("Surname", surname, Field.Store.YES),
new TextField("Surname_sort", surname.ToLower(), Field.Store.NO),
new TextField("Name", name, Field.Store.YES),
new TextField("Name_sort", name.ToLower(), Field.Store.NO),
};
return document;
}
The search is done like this:
using var multiReader = new MultiReader(indexWriter.GetReader(true)); //the plan was to use multiple indexes per entity types
var indexSearcher = new IndexSearcher(multiReader);
var queryString = "abc"; //just as a sample
var queryWords = queryString.SplitWords();
var query = new BooleanQuery();
queryWords
.Process((word, index) =>
{
var boolean = new BooleanQuery()
{
{ new PrefixQuery(new Term("Surname", word)) { Boost = 100 }, Occur.SHOULD }, //surnames are most important to match
{ new PrefixQuery(new Term("Name", word)) { Boost = 50 }, Occur.SHOULD }, //names are less important
};
boolean.Boost = (queryWords.Count() - index); //first words in a search query are more important than others
query.Add(boolean, Occur.MUST);
})
;
var topDocs = indexSearcher.Search(query, 50, new Sort( //sort by relevance and then in lexicographical order
SortField.FIELD_SCORE,
new SortField("Surname_sort", SortFieldType.STRING),
new SortField("Name_sort", SortFieldType.STRING)
));
And highlighting:
var htmlFormatter = new SimpleHTMLFormatter();
var queryScorer = new QueryScorer(query);
var highlighter = new Highlighter(htmlFormatter, queryScorer);
foreach (var found in topDocs.ScoreDocs)
{
var document = indexSearcher.Doc(found.Doc);
var surname = document.Get("Surname"); //just for simplicity
var surnameFragment = highlighter.GetBestFragment(standardAnalyzer, "Surname", surname);
Console.WriteLine(surnameFragment);
}
The problem is that the highlighter returns results like this:
<b>abc</b>
<b>abcd</b>
<b>abcde</b>
<b>abcdef</b>
So it "highlights" entire words even though I was searching for partials.
Explain returned NON-MATCH all the way so not sure if it's helpful here.
Is it possible to highlight only the parts which were searched for? Like in my example.

While searching a bit more on this I came to a conclusion that to make such highlighting work one needs to tweak index generation methods and split indices by parts so offsets would be properly calculated. Or else highlighting will highlight only surrounding words (fragments) entirely.
So based on this I've managed to build a simple highlighter of my own.
public class Highlighter
{
private const string TempStartToken = "\x02";
private const string TempEndToken = "\x03";
private const string SearchPatternTemplate = $"[{TempStartToken}{TempEndToken}]*{{0}}";
private const string ReplacePattern = $"{TempStartToken}$&{TempEndToken}";
private readonly ConcurrentDictionary<HighlightKey, Regex> _regexPatternsCache = new();
private static string GetHighlightTypeTemplate(HighlightType highlightType) =>
highlightType switch
{
HighlightType.Starts => "^{0}",
HighlightType.Contains => "{0}",
HighlightType.Ends => "{0}$",
HighlightType.Equals => "^{0}$",
_ => throw new ArgumentException($"Unsupported {nameof(HighlightType)}: '{highlightType}'", nameof(highlightType)),
};
public string Highlight(string text, IReadOnlySet<string> words, string startToken, string endToken, HighlightType highlightType)
{
foreach (var word in words)
{
var key = new HighlightKey
{
Word = word,
HighlightType = highlightType,
};
var regex = _regexPatternsCache.GetOrAdd(key, _ =>
{
var parts = word.Select(w => string.Format(SearchPatternTemplate, Regex.Escape(w.ToString())));
var pattern = string.Concat(parts);
var highlightPattern = string.Format(GetHighlightTypeTemplate(highlightType), pattern);
return new Regex(highlightPattern, RegexOptions.IgnoreCase | RegexOptions.CultureInvariant | RegexOptions.Compiled);
});
text = regex.Replace(text, ReplacePattern);
}
return text
.Replace(TempStartToken, startToken)
.Replace(TempEndToken, endToken)
;
}
private record HighlightKey
{
public string Word { get; init; }
public HighlightType HighlightType { get; init; }
}
}
public enum HighlightType
{
Starts,
Contains,
Ends,
Equals,
}
Use it like this:
var queries = new[] { "abc" }.ToHashSet();
var search = "a ab abc abcd abcde";
var highlighter = new Highlighter();
var outputs = search
.Split((string[])null, StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)
.Select(w => highlighter.Highlight(w, queries, "<b>", "</b>", HighlightType.Starts))
;
var result = string.Join(" ", outputs).Dump();
Util.RawHtml(result).Dump();
Output looks like this:
a ab <b>abc</b> <b>abc</b>d <b>abc</b>de
a ab abc abcd abcde
I'm open to any other better solutions.

C# Solr query - how to only get future dates?

I have a search method that queries Solr for event items. I need to modify it to only get events where the date has not already passed (i.e. Where(x => x.EventDate.Date >= DateTime.Now.Date), but I'm not sure how to add this because I'm not very familiar with Solr. Here's my search function:
public SearchQueryResults Search(string keywords, int page,int perPage, List<Guid> contentTypeFilters, List<Guid> otherFilters, ISortBuilder<SearchResultItem> sortBuilder)
{
var searchFilters = new List<IPredicateBuilder<SearchResultItem>>()
{
new IsSearchablePredicateBuilder()
};
if (contentTypeFilters.Any())
{
var contentTypePredicateBuilder = new ContentTypePredicateBuilder();
contentTypePredicateBuilder.ContentTypes = contentTypeFilters;
searchFilters.Add(contentTypePredicateBuilder);
}
if (otherFilters.Any())
{
var tagFilterBuilder = new TagsAndPredicateBuilder(otherFilters,_sitecoreContext);
searchFilters.Add(tagFilterBuilder);
}
if (string.IsNullOrWhiteSpace(keywords))
{
keywords = "";
}
SearchRequest searchRequest = new SearchRequest();
var queryParams = new Dictionary<string, string>() { };
queryParams.Add("q", keywords);
searchRequest.QueryParameters = queryParams;
searchRequest.SortBy = "";
searchRequest.SortOrder = "";
SearchQuery<SearchResultItem> queryArguments = new SearchQuery<SearchResultItem>();
queryArguments.FilterBuilders = searchFilters;
queryArguments.Page = page;
queryArguments.PerPage = perPage;
queryArguments.FacetsBuilder = new SearchFacetBuilder<SearchResultItem>();
queryArguments.SearchRequest = searchRequest;
queryArguments.IndexName = _indexName;
if (string.IsNullOrWhiteSpace(keywords))
{
queryArguments.QueryBuilders =new List<IPredicateBuilder<SearchResultItem>>();
}
else
{
queryArguments.QueryBuilders = new[] { new KeywordPredicateBuilder<SearchResultItem>(new[] { keywords }) };
}
queryArguments.SortBuilder = sortBuilder;
try
{
var results = _searchManager.GetResults<SearchResultItem>(queryArguments);
SearchQueryResults queryResults = new SearchQueryResults();
queryResults.ResultItems = results.Results;
queryResults.CurrentPage = page;
queryResults.TotalResults = Int32.Parse(results.TotalResults.ToString());
queryResults.TotalPages = (queryResults.TotalResults + perPage - 1) / perPage; ;
return queryResults;
}
catch (Exception exc)
{
Sitecore.Diagnostics.Log.Error("Error with FilteredSearch, could be a loss of connection to the SOLR server: " + exc.Message, this);
return null;
}
}
and here is how it's being called:
Results = _searchService.Search(searchTerm, CurrentPage - 1, 10, contentTypes, searchFilters,
new GenericSortBuilder<SearchResultItem>(q => q.OrderByDescending(r => r.SearchDate)));
How do I add in date filtering so that it only returns items where the date is in the future?

I would add filter query to the list of existing ones filtering the date field. On the documentation page, I was able to find information about fluent API, which could help here
Query.Field("date").From(DateTime.Now)
I'm not C# developer, that this code could have some mistakes, but I think the main idea is clear what needs to be done.

How to replace "http:" with "https:" in a string with C#?

I've stored all URLs in my application with "http://" - I now need to go through and replace all of them with "https:". Right now I have:
foreach (var link in links)
{
if (link.Contains("http:"))
{
/// do something, slice or replace or what?
}
}
I'm just not sure what the best way to update the string would be. How can this be done?

If you're dealing with uris, you probably want to use UriBuilder since doing a string replace on structured data like URIs is not a good idea.
var builder = new UriBuilder(link);
builder.Scheme = "https";
Uri modified = builder.Uri;
It's not clear what the type of links is, but you can create a new collection with the modified uris using linq:
IEnumerable<string> updated = links.Select(link => {
var builder = new UriBuilder(link);
builder.Scheme = "https";
return builder.ToString();
});

The problem is your strings are in a collection, and since strings are immutable you can't change them directly. Since you didn't specify the type of links (List? Array?) the right answer will change slightly. The easiest way is to create a new list:
links = links.Select(link => link.Replace("http://","https://")).ToList();
However if you want to minimize the number of changes and can access the string by index you can just loop through the collection:
for(int i = 0; i < links.Length; i++ )
{
links[i] = links[i].Replace("http://","https://");
}

based on your current code, link will not be replace to anything you want because it is read only (see here: Why can't I modify the loop variable in a foreach?). instead use for
for(int a = 0; a < links.Length; a++ )
{
links[a] = links[a].Replace("http:/","https:/")
}

http://myserver.xom/login.aspx?returnurl=http%3a%2f%2fmyserver.xom%2fmyaccount.aspx&q1=a%20b%20c&q2=c%2b%2b
What about the urls having also url in the querystring part? I think we should also replace them. And because of the url encoding-escaping this is the hard part of the job.
private void BlaBla()
{
// call the replacing function
Uri myNewUrl = ConvertHttpToHttps(myOriginalUrl);
}
private Uri ConvertHttpToHttps(Uri originalUri)
{
Uri result = null;
int httpsPort = 443;// if needed assign your own value or implement it as parametric
string resultQuery = string.Empty;
NameValueCollection urlParameters = HttpUtility.ParseQueryString(originalUri.Query);
if (urlParameters != null && urlParameters.Count > 0)
{
StringBuilder sb = new StringBuilder();
foreach (string key in urlParameters)
{
if (sb.Length > 0)
sb.Append("&");
string value = urlParameters[key].Replace("http://", "https://");
string valuEscaped = Uri.EscapeDataString(value);// this is important
sb.Append(string.Concat(key, "=", valuEscaped));
}
resultQuery = sb.ToString();
}
UriBuilder resultBuilder = new UriBuilder("https", originalUri.Host, httpsPort, originalUri.AbsolutePath);
resultBuilder.Query = resultQuery;
result = resultBuilder.Uri;
return result;
}

Use string.Replace and some LINQ:
var httpsLinks = links.Select(l=>l.Replace("http://", "https://");

Multiple complexFilter in Magento's api v2

Currently I’m having some difficulties with using new Magento's soap v2 from c# interface.
With php i was able to do something like this:
$params["created_at"]["from"] = date("Y-m-d H:i:s",Functions::convert_time($dataDa));
$params["created_at"]["to"] = date("Y-m-d H:i:s",Functions::convert_time($dataA));
MageInterface::getSingleton()->shipmentList($params);
In this mode i was able to find list of orders which were created from $dataDa to $dataA without problems. With c# however it seems that only the last one of the selectors work.
My code:
var cpf = new complexFilter[2];
cpf[0] = new complexFilter
{
key = "created_at",
value = new associativeEntity
{
key = "to",
value = uxDataA.DateTime.ToString("yy-MM-dd HH:mm:ss")
}
});
cpf[1] = new complexFilter
{
key = "created_at",
value = new associativeEntity
{
key = "from",
value = uxDataDa.DateTime.ToString("yy-MM-dd HH:mm:ss")
}
});
var filters = new filters();
filters.complex_filter = cpf;
var risultato = mage.salesOrderList(sessionKey, filters);
In this mode only created_at->from criteria is taken in consideration (it's like second complex filter override previous one with the same key). Ideas?
Thanks in advance.

This works for me :
private filters addFilter(filters filtresIn, string key, string op, string value)
{
filters filtres = filtresIn;
if (filtres == null)
filtres = new filters();
complexFilter compfiltres = new complexFilter();
compfiltres.key = key;
associativeEntity ass = new associativeEntity();
ass.key = op;
ass.value = value;
compfiltres.value = ass;
List<complexFilter> tmpLst;
if (filtres.complex_filter!=null)
tmpLst = filtres.complex_filter.ToList();
else tmpLst = new List<complexFilter>();
tmpLst.Add(compfiltres);
filtres.complex_filter = tmpLst.ToArray();
return filtres;
}
and call
{
Mage_Api_Model_Server_V2_HandlerPortTypeClient clientSoap = new Mage_Api_Model_Server_V2_HandlerPortTypeClient();
string sessionId = clientSoap.login(LOG, PASS);
filters filtres = new filters();
filtres = addFilter(filtres, "status", "eq", "processing");
filtres = addFilter(filtres, "created_at", "from", "2014-09-07 08:00:00");
filtres = addFilter(filtres, "created_at", "to", "2014-09-07 00:00:00");
salesOrderEntity[] lst = clientSoap.salesOrderList(sessionId, filtres);
}

Solved, there was a bug (or the feature?) in mage\sales\order\api\v2.php
See more info in this thread: http://www.magentocommerce.com/boards/viewthread/70368/

Get URL parameters from a string in .NET

I've got a string in .NET which is actually a URL. I want an easy way to get the value from a particular parameter.
Normally, I'd just use Request.Params["theThingIWant"], but this string isn't from the request. I can create a new Uri item like so:
Uri myUri = new Uri(TheStringUrlIWantMyValueFrom);
I can use myUri.Query to get the query string...but then I apparently have to find some regexy way of splitting it up.
Am I missing something obvious, or is there no built in way to do this short of creating a regex of some kind, etc?

Use static ParseQueryString method of System.Web.HttpUtility class that returns NameValueCollection.
Uri myUri = new Uri("http://www.example.com?param1=good&param2=bad");
string param1 = HttpUtility.ParseQueryString(myUri.Query).Get("param1");
Check documentation at http://msdn.microsoft.com/en-us/library/ms150046.aspx

This is probably what you want
var uri = new Uri("http://domain.test/Default.aspx?var1=true&var2=test&var3=3");
var query = HttpUtility.ParseQueryString(uri.Query);
var var2 = query.Get("var2");

Here's another alternative if, for any reason, you can't or don't want to use HttpUtility.ParseQueryString().
This is built to be somewhat tolerant to "malformed" query strings, i.e. http://test/test.html?empty= becomes a parameter with an empty value. The caller can verify the parameters if needed.
public static class UriHelper
{
public static Dictionary<string, string> DecodeQueryParameters(this Uri uri)
{
if (uri == null)
throw new ArgumentNullException("uri");
if (uri.Query.Length == 0)
return new Dictionary<string, string>();
return uri.Query.TrimStart('?')
.Split(new[] { '&', ';' }, StringSplitOptions.RemoveEmptyEntries)
.Select(parameter => parameter.Split(new[] { '=' }, StringSplitOptions.RemoveEmptyEntries))
.GroupBy(parts => parts[0],
parts => parts.Length > 2 ? string.Join("=", parts, 1, parts.Length - 1) : (parts.Length > 1 ? parts[1] : ""))
.ToDictionary(grouping => grouping.Key,
grouping => string.Join(",", grouping));
}
}
Test
[TestClass]
public class UriHelperTest
{
[TestMethod]
public void DecodeQueryParameters()
{
DecodeQueryParametersTest("http://test/test.html", new Dictionary<string, string>());
DecodeQueryParametersTest("http://test/test.html?", new Dictionary<string, string>());
DecodeQueryParametersTest("http://test/test.html?key=bla/blub.xml", new Dictionary<string, string> { { "key", "bla/blub.xml" } });
DecodeQueryParametersTest("http://test/test.html?eins=1&zwei=2", new Dictionary<string, string> { { "eins", "1" }, { "zwei", "2" } });
DecodeQueryParametersTest("http://test/test.html?empty", new Dictionary<string, string> { { "empty", "" } });
DecodeQueryParametersTest("http://test/test.html?empty=", new Dictionary<string, string> { { "empty", "" } });
DecodeQueryParametersTest("http://test/test.html?key=1&", new Dictionary<string, string> { { "key", "1" } });
DecodeQueryParametersTest("http://test/test.html?key=value?&b=c", new Dictionary<string, string> { { "key", "value?" }, { "b", "c" } });
DecodeQueryParametersTest("http://test/test.html?key=value=what", new Dictionary<string, string> { { "key", "value=what" } });
DecodeQueryParametersTest("http://www.google.com/search?q=energy+edge&rls=com.microsoft:en-au&ie=UTF-8&oe=UTF-8&startIndex=&startPage=1%22",
new Dictionary<string, string>
{
{ "q", "energy+edge" },
{ "rls", "com.microsoft:en-au" },
{ "ie", "UTF-8" },
{ "oe", "UTF-8" },
{ "startIndex", "" },
{ "startPage", "1%22" },
});
DecodeQueryParametersTest("http://test/test.html?key=value;key=anotherValue", new Dictionary<string, string> { { "key", "value,anotherValue" } });
}
private static void DecodeQueryParametersTest(string uri, Dictionary<string, string> expected)
{
Dictionary<string, string> parameters = new Uri(uri).DecodeQueryParameters();
Assert.AreEqual(expected.Count, parameters.Count, "Wrong parameter count. Uri: {0}", uri);
foreach (var key in expected.Keys)
{
Assert.IsTrue(parameters.ContainsKey(key), "Missing parameter key {0}. Uri: {1}", key, uri);
Assert.AreEqual(expected[key], parameters[key], "Wrong parameter value for {0}. Uri: {1}", parameters[key], uri);
}
}
}

Looks like you should loop over the values of myUri.Query and parse it from there.
string desiredValue;
foreach(string item in myUri.Query.Split('&'))
{
string[] parts = item.Replace("?", "").Split('=');
if(parts[0] == "desiredKey")
{
desiredValue = parts[1];
break;
}
}
I wouldn't use this code without testing it on a bunch of malformed URLs however. It might break on some/all of these:
hello.html?
hello.html?valuelesskey
hello.html?key=value=hi
hello.html?hi=value?&b=c
etc

#Andrew and #CZFox
I had the same bug and found the cause to be that parameter one is in fact: http://www.example.com?param1 and not param1 which is what one would expect.
By removing all characters before and including the question mark fixes this problem. So in essence the HttpUtility.ParseQueryString function only requires a valid query string parameter containing only characters after the question mark as in:
HttpUtility.ParseQueryString ( "param1=good&param2=bad" )
My workaround:
string RawUrl = "http://www.example.com?param1=good&param2=bad";
int index = RawUrl.IndexOf ( "?" );
if ( index > 0 )
RawUrl = RawUrl.Substring ( index ).Remove ( 0, 1 );
Uri myUri = new Uri( RawUrl, UriKind.RelativeOrAbsolute);
string param1 = HttpUtility.ParseQueryString( myUri.Query ).Get( "param1" );`

You can just use the Uri to get the list of the query strings or find a specific parameter.
Uri myUri = new Uri("http://www.example.com?param1=good&param2=bad");
var params = myUri.ParseQueryString();
var specific = myUri.ParseQueryString().Get("param1");
var paramByIndex = myUri.ParseQueryString().Get(2);
You can find more from here: https://learn.microsoft.com/en-us/dotnet/api/system.uri?view=net-5.0

You can use the following workaround for it to work with the first parameter too:
var param1 =
HttpUtility.ParseQueryString(url.Substring(
new []{0, url.IndexOf('?')}.Max()
)).Get("param1");

Or if you don't know the URL (so as to avoid hardcoding, use the AbsoluteUri
Example ...
//get the full URL
Uri myUri = new Uri(Request.Url.AbsoluteUri);
//get any parameters
string strStatus = HttpUtility.ParseQueryString(myUri.Query).Get("status");
string strMsg = HttpUtility.ParseQueryString(myUri.Query).Get("message");
switch (strStatus.ToUpper())
{
case "OK":
webMessageBox.Show("EMAILS SENT!");
break;
case "ER":
webMessageBox.Show("EMAILS SENT, BUT ... " + strMsg);
break;
}

Use .NET Reflector to view the FillFromString method of System.Web.HttpValueCollection. That gives you the code that ASP.NET is using to fill the Request.QueryString collection.

Single line LINQ solution:
Dictionary<string, string> ParseQueryString(string query)
{
return query.Replace("?", "").Split('&').ToDictionary(pair => pair.Split('=').First(), pair => pair.Split('=').Last());
}

if you want in get your QueryString on Default page .Default page means your current page url .
you can try this code :
string paramIl = HttpUtility.ParseQueryString(this.ClientQueryString).Get("city");

This is actually very simple, and that worked for me :)
if (id == "DK")
{
string longurl = "selectServer.aspx?country=";
var uriBuilder = new UriBuilder(longurl);
var query = HttpUtility.ParseQueryString(uriBuilder.Query);
query["country"] = "DK";
uriBuilder.Query = query.ToString();
longurl = uriBuilder.ToString();
}

For anyone who wants to loop through all query strings from a string
foreach (var item in new Uri(urlString).Query.TrimStart('?').Split('&'))
{
var subStrings = item.Split('=');
var key = subStrings[0];
var value = subStrings[1];
// do something with values
}

Here is a sample that mentions what dll to include
var testUrl = "https://www.google.com/?q=foo";
var data = new Uri(testUrl);
// Add a reference to System.Web.dll
var args = System.Web.HttpUtility.ParseQueryString(data.Query);
args.Set("q", "my search term");
var nextUrl = $"{data.Scheme}://{data.Host}{data.LocalPath}?{args.ToString()}";

Easiest way how to get value of know the param name:
using System.Linq;
string loc = "https://localhost:5000/path?desiredparam=that_value&anotherParam=whatever";
var c = loc.Split("desiredparam=").Last().Split("&").First();//that_value

HttpContext.Current.Request.QueryString.Get("id");

I used it and it run perfectly
<%=Request.QueryString["id"] %>