Develop Reference

Usage of Document Extraction cognitive skill

I am trying to utilize Azure Cognitive services to perform basic document extraction.
My intent is to input PDFs and DOCXs (and possibly some other files) into the Cognitive Engine for parsing, but unfortunately, the implementation of this is not as simple as it seems.
According to the documentation (https://learn.microsoft.com/en-us/azure/search/cognitive-search-skill-document-extraction#sample-definition), I must define the skill and then I should be able to input files, but there is no examples on how this should be done.
So far I have been able to define the skill but I am still not sure where I should be dropping the files into.
Please see my code below, as it seeks to replicate the same data structure shown in the example code (albeit using the C# Library)
public static DocumentExtractionSkill CreateDocumentExtractionSkill()
{
List<InputFieldMappingEntry> inputMappings = new List<InputFieldMappingEntry>
{
new("file_data") {Source = "/document/file_data"}
};
List<OutputFieldMappingEntry> outputMappings = new List<OutputFieldMappingEntry>
{
new("content") {TargetName = "extracted_content"}
};
DocumentExtractionSkill des = new DocumentExtractionSkill(inputMappings, outputMappings)
{
Description = "Extract text (plain and structured) from image",
ParsingMode = BlobIndexerParsingMode.Text,
DataToExtract = BlobIndexerDataToExtract.ContentAndMetadata,
Context = "/document",
};
return des;
}
And then I build on this skill like so:
_indexerClient = new SearchIndexerClient(new Uri(Environment.GetEnvironmentVariable("SearchEndpoint")), new AzureKeyCredential(Environment.GetEnvironmentVariable("SearchKey"));
List<SearchIndexerSkill> skills = new List<SearchIndexerSkill> { Skills.DocExtractionSkill.CreateDocumentExtractionSkill() };
SearchIndexerSkillset skillset = new SearchIndexerSkillset("DocumentSkillset", skills)
{
Description = "Document Cracker Skillset",
CognitiveServicesAccount = new CognitiveServicesAccountKey(Environment.GetEnvironmentVariable("CognitiveServicesKey"))
};
await _indexerClient.CreateOrUpdateSkillsetAsync(skillset);
And... then what?
There is no clear method that would fit what I believe the next stage, actually parsing documents.
What is the next step from here to begin dumping files into the _indexerClient (of type SearchIndexerClient)?
As the next stage shown in the documentation is:
{
"values": [
{
"recordId": "1",
"data":
{
"file_data": {
"$type": "file",
"data": "aGVsbG8="
}
}
}
]
}
Which is not clear as to where I would be doing this.

According to the document that you have mentioned. They are actually trying to get the output through postman. They are using a GET Method to receive the extracted document content by sending JSON request to the mentioned URL(i.e. Cognitive skill url) and the files/documents are needed to be uploaded to your storage account in order to get extracted.
you can follow this tutorial to get more insights.

How to set MongoDB Change Stream 'OperationType' in the C# driver?

When running the new MongDB Server, version 3.6, and trying to add a Change Stream watch to a collection to get notifications of new inserts and updates of documents, I only receive notifications for updates, not for inserts.
This is the default way I have tried to add the watch:
IMongoDatabase mongoDatabase = mongoClient.GetDatabase("Sandbox");
IMongoCollection<BsonDocument> collection = mongoDatabase.GetCollection<BsonDocument>("TestCollection");
var changeStream = collection.Watch().ToEnumerable().GetEnumerator();
changeStream.MoveNext();
var next = changeStream.Current;
Then I downloaded the C# source code from MongoDB to see how they did this. Looking at their test code for change stream watches, they create a new document(Insert) and then change that document right away(Update) and THEN set up the Change Stream watch to receive an 'update' notification.
No example is given on how to watch for 'insert' notifications.
I have looked at the Java and NodeJS examples, both on MongoDB website and SO, which seems to be straight forward and defines a way to see both Inserts and Updates:
var changeStream = collection.watch({ '$match': { $or: [ { 'operationType': 'insert' }, { 'operationType': 'update' } ] } });
The API for the C# driver is vastly different, I would have assumed they would have kept the same API for C# as Java and NodeJS. I found no or very few examples for C# to do the same thing.
The closest I have come is with the following attempt but still fails and the documentation for the C# version is very limited (or I have not found the right location). Setup is as follows:
String json = "{ '$match': { 'operationType': { '$in': ['insert', 'update'] } } }";
var options = new ChangeStreamOptions { FullDocument = ChangeStreamFullDocumentOption.UpdateLookup };
PipelineDefinition<ChangeStreamDocument<BsonDocument>, ChangeStreamDocument<BsonDocument>> pipeline = new EmptyPipelineDefinition<ChangeStreamDocument<BsonDocument>>().Match(Builders<ChangeStreamDocument<BsonDocument>>.Filter.Text(json,"json"));
Then running the statement below throws an Exception:
{"Command aggregate failed: $match with $text is only allowed as the
first pipeline stage."}
No other Filter options has worked either, and I have not found a way to just enter the JSON as a string to set the 'operationType'.
var changeStream = collection.Watch(pipeline, options).ToEnumerable().GetEnumerator();
changeStream.MoveNext();
var next = changeStream.Current;
My only goal here is to be able to set the 'operationType' using the C# driver. Does anyone know what I am doing wrong or have tried this using the C# driver and had success?
After reading though a large number of webpages, with very little info on the C# version of the MongoDB driver, I am very stuck!
Any help would be much appreciated.

Here is a sample of code I've used to update the collection Watch to retrieve "events" other than just document updates.
IMongoDatabase sandboxDB = mongoClient.GetDatabase("Sandbox");
IMongoCollection<BsonDocument> collection = sandboxDB.GetCollection<BsonDocument>("TestCollection");
//Get the whole document instead of just the changed portion
ChangeStreamOptions options = new ChangeStreamOptions() { FullDocument = ChangeStreamFullDocumentOption.UpdateLookup };
//The operationType can be one of the following: insert, update, replace, delete, invalidate
var pipeline = new EmptyPipelineDefinition<ChangeStreamDocument<BsonDocument>>().Match("{ operationType: { $in: [ 'replace', 'insert', 'update' ] } }");
var changeStream = collection.Watch(pipeline, options).ToEnumerable().GetEnumerator();
changeStream.MoveNext(); //Blocks until a document is replaced, inserted or updated in the TestCollection
ChangeStreamDocument<BsonDocument> next = changeStream.Current;
enumerator.Dispose();
The EmptyPiplineDefinition...Match() argument could also be:
"{ $or: [ {operationType: 'replace' }, { operationType: 'insert' }, { operationType: 'update' } ] }"
If you wanted to use the $or command, or
"{ operationType: /^[^d]/ }"
to throw a little regex in there. This last one is saying, I want all operationTypes unless they start with the letter 'd'.

Parsing InfluxDB result using JSON.net

I'm trying to make a command line utility for initializing a InfluxDB database, but I'm pretty new to influx, and C# in general.
With the following response from the Influx DB Database, I'm trying to pretty print this in the console window.
Ideally I would have errors show up in the standard error buffer, and warnings or info's show up in the standard output.
However, when running the code below in a debug environment, messages appears to be in an incorrect format according to several jsonpath checkers that I have used.
JSON input as result.Body
{
"results": [
{
"statement_id": 0,
"messages": [
{
"level": "warning",
"text": "deprecated use of 'CREATE RETENTION POLICY Primary ON SensorData DURATION 30d REPLICATION 1' in a read only context, please use a POST request instead"
}
]
}
]
}
JSON output as messages prior to transformation:
messages {{
"level": "warning",
"text": "deprecated use of 'CREATE RETENTION POLICY Primary ON SensorData DURATION 30d REPLICATION 1' in a read only context, please use a POST request instead"
}} Newtonsoft.Json.Linq.JToken {Newtonsoft.Json.Linq.JObject}
As you can see, the messages output is in a nested object {{}} rather then an array as expected...
According to https://jsonpath.curiousconcept.com/ and several other jsonpath checkers, I was expecting something similar to:
[
{
"level":"warning",
"text":"deprecated use of 'CREATE RETENTION POLICY Primary ON SensorData DURATION 30d REPLICATION 1' in a read only context, please use a POST request instead"
}
]
C#
private static void PrintResult(IInfluxDataApiResponse result)
{
var output = result.Success ? System.Console.Out : System.Console.Error;
output.WriteLine("["+result.StatusCode + "] : "+result.Body);
var json = JObject.Parse(result.Body);
var messages = json.SelectToken("$.results[*].messages[*]"); //outputs an array of messages if exists. e.g. [{level:warning,text:test}]
if (messages != null)
{
var transformed = messages.Select(m => new { level = (string)m["level"], text = (string)m["text]"] }).ToList();
foreach (var msg in transformed)
{
output.WriteLine($"[{result.StatusCode}] : {msg.level} - {msg.text}");
}
}
}

For my uses at least, using var messages =
json.SelectTokens("$.results[*].messages[*]");
rather then
json.SelectToken("$.results[*].messages[*]");
allowed me to workaround the issue, as I could then treat the result as a C# enumerable, as opposed to special casing 1 result vs many results for SelectToken as it seems to flatten single results into an object, where as other implementations would have it be an array.

Receiving list of hubs with Autodesk Forge API

I am trying to use the Autodesk forge API for C# to get a list of Hubs.
This is what I have done so far:
HubsApi api = new HubsApi();
Hubs hubs = api.GetHubs();
Pretty simple. But when I do so, I get an exception wicht complains, that one cannot convert from DynamicJsonResponse to Hubs. I think, this is because I receive two warnings in my response string and so it is not a Hub Object any longer. The warning looks like this:
"warnings":[
{
"Id":null,
"HttpStatusCode":"403",
"ErrorCode":"BIM360DM_ERROR",
"Title":"Unable to get hubs from BIM360DM EMEA.",
"Detail":"You don't have permission to access this API",
"AboutLink":null,
"Source":[
],
"meta":[
]
}
All of this is wrapped in an Dictionary with four entries and only two of them are data. However, according to Autodesk, this warning can be ignored.
So after that I tried to convert it in a Dictionary and only select the data entry
HubsApi api = new HubsApi();
DynamicJsonResponse resp = api.GetHubs();
DynamicDictionary hubs = (DynamicDictionary)resp.Dictionary["data"];
Then I looped through it:
for(int i = 0; i < hubs.Dictionary.Count && bim360hub == null; i++)
{
string hub = hubs.Dictionary.ElementAt(i).ToString();
[....]
}
But the string hub isn't an json-hub either. It is an array which looks like this:
[
0,
{
"type": "hubs",
"id": "****",
"attributes": {...},
"links": {...},
"relationships": {...},
}
]
And the second element in the array is my hub. I know, how I can select the second element. But there must be a much easier to get the list of hubs.
In an example in the references it seemd to work with these simple two lines of code:
HubsApi api = new HubsApi();
Hubs hubs = api.GetHubs();
Any ideas, how I manage to get my hubs?

First, consider using the Async version of those methods, avoid using non-async calls as it causes your desktop app to freeze (while is connecting) or allocates more resources on ASP.NET.
The following function is part of this sample, which lists all hubs, projects and files under a user account. It's a good place to start. Note it's organizing the Hubs in a TreeNode list, which is compatible with jsTree.
private async Task<IList<TreeNode>> GetHubsAsync()
{
IList<TreeNode> nodes = new List<TreeNode>();
HubsApi hubsApi = new HubsApi();
hubsApi.Configuration.AccessToken = AccessToken;
var hubs = await hubsApi.GetHubsAsync();
string urn = string.Empty;
foreach (KeyValuePair<string, dynamic> hubInfo in new DynamicDictionaryItems(hubs.data))
{
string nodeType = "hubs";
switch ((string)hubInfo.Value.attributes.extension.type)
{
case "hubs:autodesk.core:Hub":
nodeType = "hubs";
break;
case "hubs:autodesk.a360:PersonalHub":
nodeType = "personalhub";
break;
case "hubs:autodesk.bim360:Account":
nodeType = "bim360hubs";
break;
}
TreeNode hubNode = new TreeNode(hubInfo.Value.links.self.href, (nodeType == "bim360hubs" ? "BIM 360 Projects" : hubInfo.Value.attributes.name), nodeType, true);
nodes.Add(hubNode);
}
return nodes;
}

Transfer values between 2 config files

I have a problem, and i can't figure this out myself..
In my program i have an auto updater, when my program updates a new(changed, some new keys) config file is created. what i want my program to do is, is when it's updating to look at both config files(old and new) and transfer old settings that match a key in the new file to the new file.
This is an example of the old file:
{
"Setting1": false,
"Setting2": 123,
"Setting3": "test",
"LocationList": {
"Country": "NL",
"Locations": [
{
"Latitude": 38.556807486461118,
"Longitude": -121.2383794784546
},
{
"Latitude": -33.859019,
"Longitude": 151.213098
},
{
"Latitude": 47.5014969,
"Longitude": -122.0959568
},
{
"Latitude": 51.5025343,
"Longitude": -0.2055027
}
]
}
}
And this can be the new file(can also be different):
{
"Setting1": null,
"Setting2": null,
"Setting3": "",
"Setting4": ""
"LocationList": {
"Country": "",
"Locations": [
{
"Latitude": null,
"Longitude": null
},
{
"Latitude": null,
"Longitude": null
}
]
}
}
Expected result:
{
"Setting1": false,
"Setting2": 123,
"Setting3": "test",
"Setting4": ""
"LocationList": {
"Country": "NL",
"Locations": [
{
"Latitude": 38.556807486461118,
"Longitude": -121.2383794784546
},
{
"Latitude": -33.859019,
"Longitude": 151.213098
},
{
"Latitude": 47.5014969,
"Longitude": -122.0959568
},
{
"Latitude": 51.5025343,
"Longitude": -0.2055027
}
]
}
}
First, i looked at creating a class in c# and just deserialize it, then, i came to the conclusion that this is not possible because i don't know what the config is going to look like.
Second, i thought using a dynamic would do the trick, it didn't, because i didn't knew any keys that were in it. And couldn't figure out how to figure that out.
And lastly, i've looked if it would be possible using regex, for me, this seems impossible..
Can anybody give me some ideas of how they would do it? I don't need code, just a push in the right direction.
P.S. i do not want to combine the two, when there is a key in the old file but not in the new one, it doesn't need to be transferred(Only lists will be completely transferred from the old file, also when the list is empty/filled in the new one).

If you really want to try something in JSON, I can only recommend the excellent JSON.Net library to parse the json for you. Using LINQ to JSON you could easily find matching keys in a recursive fashion between the old config file and the newer one and simply copy the value from one to the other
see documentation and a small example at http://www.newtonsoft.com/json/help/html/LINQtoJSON.htm
briefly you could do so in pseudoCode. Obviously this would not be super performant because you would recursively walk two files a lot of times and could be optimized, but at the same time unless your configuration files are monstrous blobs this should not pose any problem on any kind of modern hardware
//load both config files
//load the first token in original file and walk recursively
//for each token try to match in the new file and write data if required using the same recursive technique to walk the other file

I don't need code, just a push in the right direction.
Store your configuration in a regular App.config file and leverage the System.Configuration. Depending on the configuration complexity you can either use the ready <appSettings> or construct your own config section(s) and elements. It still be easier and error proof than doing custom config.
The matter is too complicated to start inventing the wheel just to use another (text) format. Even if you solve your current problem, there will be more, which are already solved somewhere within the System.Configration...
You can start exploring the configuration options here; anyways a regular search in your favorite search engine will do the trick...

Oke after some trouble i've managed to fix it(Thanks #Louis).
This is how i've done it:
private static bool TransferConfig(string baseDir, ISession session)
{
//if (!session.LogicSettings.TransferConfigAndAuthOnUpdate)
// return false;
var configDir = Path.Combine(baseDir, "Config");
if (!Directory.Exists(configDir))
return false;
var oldConf = GetJObject(Path.Combine(configDir, "config.json.old"));
var oldAuth = GetJObject(Path.Combine(configDir, "auth.json.old"));
GlobalSettings.Load("");
var newConf = GetJObject(Path.Combine(configDir, "config.json"));
var newAuth = GetJObject(Path.Combine(configDir, "auth.json"));
TransferJSON(oldConf, newConf);
TransferJSON(oldAuth, newAuth);
File.WriteAllText(Path.Combine(configDir, "config.json"), newConf.ToString());
File.WriteAllText(Path.Combine(configDir, "auth.json"), newAuth.ToString());
return true;
}
private static JObject GetJObject(string filePath)
{
return JObject.Parse(File.ReadAllText(filePath));
}
private static bool TransferJSON(JObject oldFile, JObject newFile)
{
try
{
foreach (var newProperty in newFile.Properties())
foreach (var oldProperty in oldFile.Properties())
if (newProperty.Name.Equals(oldProperty.Name))
{
newFile[newProperty.Name] = oldProperty.Value;
break;
}
return true;
}
catch
{
return false;
}
}