Develop Reference

Lucene 4.8 facets usage

I have difficulties understanding this example on how to use facets :
https://lucenenet.apache.org/docs/4.8.0-beta00008/api/Lucene.Net.Demo/Lucene.Net.Demo.Facet.SimpleFacetsExample.html
My goal is to create an index in which each document field have a facet, so that at search time i can choose which facets use to navigate data.
What i am confused about is setup of facets in index creation, to
summarize my question : is index with facets compatibile with
ReferenceManager?
Need DirectoryTaxonomyWriter to be actually written and persisted
on disk or it will embedded into the index itself and is just
temporary? I mean given the code
indexWriter.AddDocument(config.Build(taxoWriter, doc)); of the
example i expect it's temporary and will be embedded into the index (but then the example also show you need the Taxonomy to drill down facet). So can the Taxonomy be tangled in some way with the index so that the are handled althogeter with ReferenceManager?
If is not may i just use the same folder i use for storing index?
Here is a more detailed list of point that confuse me :
In my scenario i am indexing the document asyncrhonously (background process) and then fetching the indext ASAP throught ReferenceManager in ASP.NET application. I hope this way to fetch the index is compatibile with DirectoryTaxonomyWriter needed by facets.
Then i modified the code i write introducing the taxonomy writer as indicated in the example, but i am a bit confused, seems like i can't store DirectoryTaxonomyWriter into the same folder of index because the folder is locked, need i to persist it or it will be embedded into the index (so a RAMDirectory is enougth)? if i need to persist it in a different direcotry, can i safely persist it into subdirectory?
Here the code i am actually using :
private static void BuildIndex (IndexEntry entry)
{
string targetFolder = ConfigurationManager.AppSettings["IndexFolder"] ?? string.Empty;
//** LOG
if (System.IO.Directory.Exists(targetFolder) == false)
{
string message = #"Index folder not found";
_fileLogger.Error(message);
_consoleLogger.Error(message);
return;
}
var metadata = JsonConvert.DeserializeObject<IndexMetadata>(File.ReadAllText(entry.MetdataPath) ?? "{}");
string[] header = new string[0];
List<dynamic> csvRecords = new List<dynamic>();
using (var reader = new StreamReader(entry.DataPath))
{
CsvConfiguration csvConfiguration = new CsvConfiguration(CultureInfo.InvariantCulture);
csvConfiguration.AllowComments = false;
csvConfiguration.CountBytes = false;
csvConfiguration.Delimiter = ",";
csvConfiguration.DetectColumnCountChanges = false;
csvConfiguration.Encoding = Encoding.UTF8;
csvConfiguration.HasHeaderRecord = true;
csvConfiguration.IgnoreBlankLines = true;
csvConfiguration.HeaderValidated = null;
csvConfiguration.MissingFieldFound = null;
csvConfiguration.TrimOptions = CsvHelper.Configuration.TrimOptions.None;
csvConfiguration.BadDataFound = null;
using (var csvReader = new CsvReader(reader, csvConfiguration))
{
csvReader.Read();
csvReader.ReadHeader();
csvReader.Read();
header = csvReader.HeaderRecord;
csvRecords = csvReader.GetRecords<dynamic>().ToList();
}
}
string targetDirectory = Path.Combine(targetFolder, "Index__" + metadata.Boundle + "__" + DateTime.Now.ToString("yyyyMMdd_HHmmss") + "__" + Path.GetRandomFileName().Substring(0, 6));
System.IO.Directory.CreateDirectory(targetDirectory);
//** LOG
{
string message = #"..creating index : {0}";
_fileLogger.Information(message, targetDirectory);
_consoleLogger.Information(message, targetDirectory);
}
using (var dir = FSDirectory.Open(targetDirectory))
{
using (DirectoryTaxonomyWriter taxoWriter = new DirectoryTaxonomyWriter(dir))
{
Analyzer analyzer = metadata.GetAnalyzer();
var indexConfig = new IndexWriterConfig(LuceneVersion.LUCENE_48, analyzer);
using (IndexWriter writer = new IndexWriter(dir, indexConfig))
{
long entryNumber = csvRecords.Count();
long index = 0;
long lastPercentage = 0;
foreach (dynamic csvEntry in csvRecords)
{
Document doc = new Document();
IDictionary<string, object> dynamicCsvEntry = (IDictionary<string, object>)csvEntry;
var indexedMetadataFiled = metadata.IdexedFields;
foreach (string headField in header)
{
if (indexedMetadataFiled.ContainsKey(headField) == false || (indexedMetadataFiled[headField].NeedToBeIndexed == false && indexedMetadataFiled[headField].NeedToBeStored == false))
continue;
var field = new Field(headField,
((string)dynamicCsvEntry[headField] ?? string.Empty).ToLower(),
indexedMetadataFiled[headField].NeedToBeStored ? Field.Store.YES : Field.Store.NO,
indexedMetadataFiled[headField].NeedToBeIndexed ? Field.Index.ANALYZED : Field.Index.NO
);
doc.Add(field);
var facetField = new FacetField(headField, (string)dynamicCsvEntry[headField]);
doc.Add(facetField);
}
long percentage = (long)(((decimal)index / (decimal)entryNumber) * 100m);
if (percentage > lastPercentage && percentage % 10 == 0)
{
_consoleLogger.Information($"..indexing {percentage}%..");
lastPercentage = percentage;
}
writer.AddDocument(doc);
index++;
}
writer.Commit();
}
}
}
//** LOG
{
string message = #"Index Created : {0}";
_fileLogger.Information(message, targetDirectory);
_consoleLogger.Information(message, targetDirectory);
}
}

How to send multiple images to cloudinary?

Follow code:
string[] files
= System.IO.Directory.GetFiles(#"C:\Users\Matheus Miranda\Pictures", "*.jpg");
foreach (var file in files)
{
var uploadParams = new ImageUploadParams()
{
File = new FileDescription(file),
PublicId = "my_folder/images",
EagerAsync = true
};
var uploadResult = cloudinary.Upload(uploadParams);
}
It is not working, it is always recording the previous file.
I'm trying to save multiple images to cloudinary and nothing is success.
Only one image is saved. I use a libray Cloudinary.
Any solution ?

When I tested it out, it works as expected; however, I would adjust a couple of things. The first thing is that you do not need the parameter eager_async as no eager transformation is being applied to the assets. An eager transformation lets you create a modified version of the original asynchronously after the asset has been uploaded. Secondly, if you wish to see the upload response, you can use property JsonObj and display it in the console. I have modified your sample here:
string[] files
= System.IO.Directory.GetFiles(#"C:\Users\Matheus Miranda\Pictures", "*.jpg");
foreach (var file in files)
{
var uploadParams = new ImageUploadParams()
{
File = new FileDescription(file),
UseFilename = true
};
var uploadResult = cloudinary.Upload(uploadParams);
Console.WriteLine(uploadResult.JsonObj);
}

I found the solution!
string[] files =
System.IO.Directory.GetFiles(#"C:\Users\Matheus Miranda\Pictures\teste", "*.jpg");
for (int i = 0; i < files.Length; i++)
{
var uploadParams = new ImageUploadParams()
{
File = new FileDescription(files[i]),
PublicId = $"my_folder/images/{System.IO.Path.GetFileName(files[i])}"
};
var uploadResult = cloudinary.Upload(uploadParams);
}

Creating a blank document in a document library using client object model

I'm creating a function where you can provide a content type name and target either a list or document library and create a default item. Im using the client object model for office 2013
public void MyFunction()
{
//clientContext must be authenticated already on your sharepoint site
var listName = "Default Document Set";
var docSetContentTypeName = "Document";
var newDocSetName = string.Format("Item {0}", Guid.NewGuid());
Web web = clientContext.Web;
List list = clientContext.Web.Lists.GetByTitle(listName);
clientContext.Load(clientContext.Site);
ContentTypeCollection listContentTypes = list.ContentTypes;
clientContext.Load(listContentTypes, types => types.Include
(type => type.Id, type => type.Name,
type => type.Parent));
var result = clientContext.LoadQuery(listContentTypes.Where
(c => c.Name == docSetContentTypeName));
clientContext.ExecuteQuery();
ContentType targetDocumentSetContentType = result.FirstOrDefault();
ListItemCreationInformation newItemInfo = new ListItemCreationInformation();
newItemInfo.UnderlyingObjectType = FileSystemObjectType.Folder;
newItemInfo.LeafName = newDocSetName;
ListItem newListItem = list.AddItem(newItemInfo);
newListItem["ContentTypeId"] = targetDocumentSetContentType.Id.ToString();
newListItem["Title"] = newDocSetName;
newListItem.Update();
clientContext.Load(list);
clientContext.ExecuteQuery();
}
The function works fine on ContentTypes like Item and Document Set but when I use Document, it creates an item with a content type "Document" but it has an icon of folder and acts like a folder.
Is there something I need to add?
Thanks in advance.

FileSystemObjectType.Folder is used for creating a Folder object and therefore could not be specified for creating File object. At the same time List.AddItem Method could not be used for creating a File object.
You could consider the following example that demonstrates how to create(upload) a file into Documents library:
public static void UploadFile(List list,string filePath,IDictionary<string,object> itemProperties)
{
var ctx = list.Context;
var fileInfo = new FileCreationInformation();
fileInfo.Url = Path.GetFileName(filePath);
fileInfo.Overwrite = true;
fileInfo.Content = System.IO.File.ReadAllBytes(filePath);
var file = list.RootFolder.Files.Add(fileInfo);
var listItem = file.ListItemAllFields;
foreach (var p in itemProperties)
{
listItem[p.Key] = p.Value;
}
listItem.Update();
ctx.ExecuteQuery();
}
Using Open XML SDK 2.0 for Microsoft Office you could create an empty document and upload it into Documents library:
public static void CreateAndUploadFile(List list, string filePath, IDictionary<string, object> itemProperties)
{
using (var document = WordprocessingDocument.Create(filePath, WordprocessingDocumentType.Document))
{
var mainPart = document.AddMainDocumentPart();
mainPart.Document = new Document(new Body());
}
UploadFile(list, filePath, itemProperties);
}
Usage:
var list = web.Lists.GetByTitle(listTitle);
var itemProperties = new Dictionary<string,object>();
itemProperties["Title"] = "SharePoint User Guide";
CreateAndUploadFile(list, "./SharePoint User Guide.docx", itemProperties);

Check if a location is indexed in Windows Search

How to check if a location is indexed or not? I found following code to index a location in Windows which works fine but I want to check if it is indexed or not before I make it indexed.
Uri path = new Uri(location);
string indexingPath = path.AbsoluteUri;
CSearchManager csm = new CSearchManager();
CSearchCrawlScopeManager manager = csm.GetCatalog("SystemIndex").GetCrawlScopeManager();
manager.AddUserScopeRule(indexingPath, 1, 1, 0);
manager.SaveAll();
Guys i have found a way to check if the location has been included for indexing by using IncludedInCrawlScope.
CSearchManager csm = new CSearchManager();
CSearchCrawlScopeManager manager = csm.GetCatalog("SystemIndex").GetCrawlScopeManager();
if (manager.IncludedInCrawlScope(indexingPath) == 0)
{
manager.AddUserScopeRule(indexingPath, 1, 1, 0);
manager.SaveAll();
}
But it only checks if it has been added for indexing, not if the indexing is complete.Since i will be querying on the SystemIndex, i need to make sure that the location is indexed.

I ran into a similar need and this is what I came up with. In my case I have certain file extensions that are going to end up being sent to a document management system.
I have two methods one uses the System.IO to get a list of the files in the directory that contain the extension from the list.
public IEnumerable<string> DirectoryScan(string directory)
{
List<string> extensions = new List<string>
{
"docx","xlsx","pptx","docm","xlsm","pptm","dotx","xltx","xlw","potx","ppsx","ppsm","doc","xls","ppt","doct","xlt","xlm","pot","pps"
};
IEnumerable<string> myFiles =
Directory.GetFiles(directory, "*", SearchOption.AllDirectories)
.Where(s => extensions.Any(s.EndsWith))
.ToList();
return myFiles;
}`
The second method uses the windows index search Microsoft.Search.Interop
public IEnumerable<string> QueryWindowsDesktopSearch(string directory)
{
List<string> extensions = new List<string>
{ "docx","xlsx","pptx","docm","xlsm","pptm","dotx","xltx","xlw","potx","ppsx","ppsm","doc","xls","ppt","doct","xlt","xlm","pot","pps"};
string userQuery = "*";
Boolean fShowQuery = true;
List<string> list = new List<string>();
CSearchManager manager = new CSearchManager();
CSearchCatalogManager catalogManager = manager.GetCatalog("SystemIndex");
CSearchQueryHelper queryHelper = catalogManager.GetQueryHelper();
queryHelper.QueryWhereRestrictions = string.Format("AND (\"SCOPE\" = 'file:{0}')", directory);
if (extensions != null)
{
queryHelper.QueryWhereRestrictions += " AND Contains(System.ItemType,'";
bool fFirst = true;
foreach (string ext in extensions)
{
if (!fFirst)
{
queryHelper.QueryWhereRestrictions += " OR ";
}
queryHelper.QueryWhereRestrictions += "\"" + ext + "\"";
fFirst = false;
}
queryHelper.QueryWhereRestrictions += "') ";
}
string sqlQuery = queryHelper.GenerateSQLFromUserQuery(userQuery);
using (OleDbConnection connection = new OleDbConnection(queryHelper.ConnectionString))
{
using (OleDbCommand command = new OleDbCommand(sqlQuery, connection))
{
connection.Open();
OleDbDataReader dataReader = command.ExecuteReader();
while (dataReader.Read())
{
var file = dataReader.GetString(0);
if (file != null)
{
list.Add(file.Replace("file:", ""));
}
}
}
}
return list;
}
I call both of these methods from another methods that takes the two results and compares them and returns a Boolean value indicating if they two list match. If they do not match then the folder has not been indexed fully.
If you call the QueryWindowsDesktopSearch on a folder that has not been indexed it returns zero files. You could use this as an indication that the folder isn't in the index bt its possible that the file has been added to the index but the file indexing is stopped.
You could check the status by calling something like this
CSearchManager manager = new CSearchManager();
CSearchCatalogManager catalogManager = manager.GetCatalog("SystemIndex");
_CatalogPausedReason pReason;
_CatalogStatus pStatus;
catalogManager.GetCatalogStatus(out pStatus, out pReason);
That may return something like pStatus = CATALOG_STATUS_PAUSED and pReason = CATALOG_PAUSED_REASON_USER_ACTIVE
You would know that the index is not running. Another thing you could do is call the following
int incrementalCount, notificationQueue, highPriorityQueue;
catalogManager.NumberOfItemsToIndex(out incrementalCount, out notificationQueue, out highPriorityQueue);
This is going to return the in plIncrementalCount value which would list the number of file that the entire SystemIndex has queued for indexing.

Check this implementation from a document management system:
https://code.google.com/p/olakedms/source/browse/SearchEngine/CSearchDAL.cs?r=171

Get Sub-Files with folders' IDs - Client Object Model

I want to get some folders - sub files. I have all folders SharePoint ID in a list. My code is working but it's performance very bad because there is a lot of context.ExecuteQuery;
I want to make it maybe with a Caml Query.
using (var context = new ClientContext("http://xxx/haberlesme/"))
{
var web = context.Web;
var list = context.Web.Lists.GetById(new Guid(target));
int[] siraliIdArray;
//siraliIdArray = loadSharePointIDList(); think like this
for (var j = 0; j < siraliIdArray.Count; j++)
{
var folderName = listItemCollection[j]["Title"].ToString();//Folder Name
var currentFolder = web.GetFolderByServerRelativeUrl("/haberlesme/Notice/" + folderName);
var currentFiles = currentFolder.Files;
context.Load(currentFiles);
//I don't want to execute for n folder n times. I want to execute n folder 1 time.
context.ExecuteQuery();
var ek = new LDTOTebligEk();
//I don't want to add one - one
foreach (var file1 in currentFiles)
{
ek.DokumanPath = urlPrefix + folderName + "/" + file1.Name;
ek.DokumanAd = file1.Name;
ekler.Add(ek);
}
}
}
For example I have 100 folder but I want to get 10 folders sub folder in one Execution

Since CSOM API supports Request Batching:
The CSOM programming model is built around request batching. When you
work with the CSOM, you can perform a series of data operations on the
ClientContext object. These operations are submitted to the server in
a single request when you call the ClientContext.BeginExecuteQuery
method.
you could refactor your code as demonstrated below:
var folders = new Dictionary<string,Microsoft.SharePoint.Client.Folder>();
var folderNames = new[] {"Orders","Requests"};
foreach (var folderName in folderNames)
{
var folderKey = string.Format("/Shared Documents/{0}", folderName);
folders[folderKey] = context.Web.GetFolderByServerRelativeUrl(folderKey);
context.Load(folders[folderKey],f => f.Files);
}
context.ExecuteQuery(); //execute request only once
//print all files
var allFiles = folders.SelectMany(folder => folder.Value.Files);
foreach (var file in allFiles)
{
Console.WriteLine(file.Name);
}

Use Caml Query:
Microsoft.SharePoint.Client.List list = clientContext.Web.Lists.GetByTitle("Main Folder");
Microsoft.SharePoint.Client.CamlQuery caml = new Microsoft.SharePoint.Client.CamlQuery();
caml.ViewXml = #"<View><Query><Where><Eq><FieldRef Name='FileLeafRef'/><Value Type='Folder'>SubFolderName</Value></Eq></Where></Query></View>";
caml.FolderServerRelativeUrl = " This line should be added if the main folder is not in the site layer";
Microsoft.SharePoint.Client.ListItemCollection items = list.GetItems(caml);
clientContext.Load(items);
//Get your folder using items[0]