Develop Reference

Usage of Document Extraction cognitive skill

I am trying to utilize Azure Cognitive services to perform basic document extraction.
My intent is to input PDFs and DOCXs (and possibly some other files) into the Cognitive Engine for parsing, but unfortunately, the implementation of this is not as simple as it seems.
According to the documentation (https://learn.microsoft.com/en-us/azure/search/cognitive-search-skill-document-extraction#sample-definition), I must define the skill and then I should be able to input files, but there is no examples on how this should be done.
So far I have been able to define the skill but I am still not sure where I should be dropping the files into.
Please see my code below, as it seeks to replicate the same data structure shown in the example code (albeit using the C# Library)
public static DocumentExtractionSkill CreateDocumentExtractionSkill()
{
List<InputFieldMappingEntry> inputMappings = new List<InputFieldMappingEntry>
{
new("file_data") {Source = "/document/file_data"}
};
List<OutputFieldMappingEntry> outputMappings = new List<OutputFieldMappingEntry>
{
new("content") {TargetName = "extracted_content"}
};
DocumentExtractionSkill des = new DocumentExtractionSkill(inputMappings, outputMappings)
{
Description = "Extract text (plain and structured) from image",
ParsingMode = BlobIndexerParsingMode.Text,
DataToExtract = BlobIndexerDataToExtract.ContentAndMetadata,
Context = "/document",
};
return des;
}
And then I build on this skill like so:
_indexerClient = new SearchIndexerClient(new Uri(Environment.GetEnvironmentVariable("SearchEndpoint")), new AzureKeyCredential(Environment.GetEnvironmentVariable("SearchKey"));
List<SearchIndexerSkill> skills = new List<SearchIndexerSkill> { Skills.DocExtractionSkill.CreateDocumentExtractionSkill() };
SearchIndexerSkillset skillset = new SearchIndexerSkillset("DocumentSkillset", skills)
{
Description = "Document Cracker Skillset",
CognitiveServicesAccount = new CognitiveServicesAccountKey(Environment.GetEnvironmentVariable("CognitiveServicesKey"))
};
await _indexerClient.CreateOrUpdateSkillsetAsync(skillset);
And... then what?
There is no clear method that would fit what I believe the next stage, actually parsing documents.
What is the next step from here to begin dumping files into the _indexerClient (of type SearchIndexerClient)?
As the next stage shown in the documentation is:
{
"values": [
{
"recordId": "1",
"data":
{
"file_data": {
"$type": "file",
"data": "aGVsbG8="
}
}
}
]
}
Which is not clear as to where I would be doing this.

According to the document that you have mentioned. They are actually trying to get the output through postman. They are using a GET Method to receive the extracted document content by sending JSON request to the mentioned URL(i.e. Cognitive skill url) and the files/documents are needed to be uploaded to your storage account in order to get extracted.
you can follow this tutorial to get more insights.

Deserializing JSON using UnityEngine.JsonUtility returns null list

I am trying to mod a Unity game using BepInEx. I want to read a list of a type of class I've created from a JSON, then read each object from that class. I used UnityEngine.JsonUtility to do this and created a wrapper class containing my list.
[System.Serializable]
public class StarSignDataList
{
public List<StarSignData> starSigns;
}
// Stores all data pertaining to star signs as read from JSON file.
[System.Serializable]
public class StarSignData
{
public string name;
public int id;
public string description;
}
Then I read the json file from the correct file path, parsing that information into a string and called the JsonUtility.FromJason() method to convert the string into my StarSignDataList class.
// Reads all star sign data from JSON and assigns values to list.
public static void InitializeData()
{
log.LogInfo("Initializing Star Sign data.");
string json = File.ReadAllText("BepInEx/plugins/StarSigns/starsigns.json");
log.LogInfo(json);
StarSignDataList list = UnityEngine.JsonUtility.FromJson<StarSignDataList>(json);
log.LogWarning(list.starSigns.Count == 0 ? "Not null" : "Null!");
log.LogInfo(list.starSigns[0].name);
signs = list.starSigns;
foreach (StarSignData data in signs)
{
log.LogInfo(data.name);
}
}
When I print out the contents of my json immediately after reading it, it comes out as expected, looking exactly like my json file. However when I convert the json string into my list class, the list is completely empty and then throws a NullReferenceException when trying to print the first object. I've trying using arrays instead of lists, I've checked that my json is correctly written, I've pored over countless google search results without success.
Here is my json file.
{
"starSigns": [
{
"name": "Sign of the Fool",
"id": 0,
"description": "You are of normal birth."
},
{
"name": "Sign of Loki",
"id": 1,
"description": "The sign of the god of betrayal shone above your birth place."
},
{
"name": "Sign of Odin",
"id": 2,
"description": "A tumultuous child at birth, your muscles bear the strength of Odin himself."
}
]
}
Out of desperation I tried converting a class like this to a json, and it just wrote to the file "{ }". When I try to do the same with just my StarSignData class it works as expected, but I need to put multiple objects in my json and the list class does not seem to work. I also can't use external libraries such as SimpleJson for this.

Update property in json file with C#

I'm looking to change a specific property for each json record in my json file. I'd like to change the "Completed" property to "true" when a method finishes executing.
My json file looks like:
{
"LoanRecords": [
{
"LoanGUID": "{70dbec7e-5e94-460d-831c-0a5dc2d085e2}",
"RecordDT": "2020-11-10T14:44:34.378Z",
"Completed": "false",
"Environment": "TEBE",
"ProcessType": "RateLock"
},
{
"LoanGUID": "{70dbec7e-5e94-460d-831c-0a5dc2d085e2}",
"RecordDT": "2020-11-10T14:53:12.187Z",
"Completed": "false",
"Environment": "TEBE",
"ProcessType": "RateLock"
}
]
}
My C# code is the following:
private void ExecuteEvent(object sender, ElapsedEventArgs e)
{
string fileRecord = File.ReadAllText(jsonfile);
LoanRecordRoot LoanRecord = JsonConvert.DeserializeObject<LoanRecordRoot>(fileRecord);
foreach (var rec in LoanRecord.LoanRecords)
{
if (rec.Completed == "false")
{
bool recordModified = ManipulateEncompass(rec.LoanGUID, rec.ProcessType);
if (recordModified)
{
// What should I do here to update "rec.Completed" to "true"
// for this particular record and write it back to the json file
// for that specific entry?
}
}
}
Console.WriteLine("Successfully manipulated records!");
}
Is there a way to flip the "Completed" property to "true" for the specific record in my "foreach" iteration, and update the json file accordingly for that specific record? I am hoping to avoid reading the entire file, deserializing, changing the property then writing the entire content back to the json file, I'm looking to just flip that specific property for each record in my "foreach" loop. -- I hope that makes sense.
I've looked at similar questions, which seem close to what I'm looking for, but the examples I've seen don't reflect writing back to the json file specifically without overwriting the file contents -- unless this specific action isn't possible, or I'm failing to understand the entire process (highly possible.)
Ex of a solution that's close to what I'm looking for: How to update a property of a JSON object using NewtonSoft -- but doesn't seem to quite fit the bill for what I'm wanting to do.
Thank you in advance for any helpful leads!

you need to save the complete JSON when you update a property of an element of the array
static void Main(string[] args)
{
const string jsonPath = #"C:\Logs\recordRoot.json";
var loanRecordRoot = JsonConvert.DeserializeObject<LoanRecordRoot>(File.ReadAllText(jsonPath));
foreach (var record in loanRecordRoot.LoanRecords)
{
if (record.Completed == "false")
{
if (ManipulateEncompass(rec.LoanGUID, rec.ProcessType))
{
record.Completed = "true";
}
}
}
//Save Json
var json = JsonConvert.SerializeObject(loanRecordRoot, Formatting.Indented);
File.WriteAllText(jsonPath, json);
}

Looking at your JSON, it appears the "Completed" property is being serialized as of type string
Therefore, all you need to do is set it to "Completed": "true" within your condition in your snippet.
if (recordModified)
{
rec.Completed = "true";
}
At the end of your processing, simply serialize your LoanRecord object and write it back to your file.

using Kitchen_Mini_Project.Constants;
using Kitchen_Mini_Project.Moduls;
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace Kitchen_Mini_Project.Services
{
public class Update
{
public static void UpdateAnyProduct()
{
string readdedFile = File.ReadAllText(PathConst.ProductDBPath);
IList<Restaurant> products = JsonConvert.DeserializeObject<IList<Restaurant>>(readdedFile);
foreach (var product in products[0].FoodItems)
{
if (product.foodName == "Chicken Burrito")
{
product.foodName = "Chicken Burrito is Update ha ha";
}
}
var json = JsonConvert.SerializeObject(products, Formatting.Indented);
File.WriteAllText(PathConst.ProductDBPath, json);
}
}
}

1-install package Newtonsoft.Json
https://learn.microsoft.com/en-us/nuget/consume-packages/install-use-packages-visual-studio
2-use
string json = File.ReadAllText("server_client _input.json");
dynamic jsonObj = Newtonsoft.Json.JsonConvert.DeserializeObject(json);
jsonObj["Bots"][0]["Password"] = "new password";
string output = Newtonsoft.Json.JsonConvert.SerializeObject(jsonObj, Newtonsoft.Json.Formatting.Indented);
File.WriteAllText("settings.json", output);
(For Install package use of this page :https://www.newtonsoft.com/json)

How to set MongoDB Change Stream 'OperationType' in the C# driver?

When running the new MongDB Server, version 3.6, and trying to add a Change Stream watch to a collection to get notifications of new inserts and updates of documents, I only receive notifications for updates, not for inserts.
This is the default way I have tried to add the watch:
IMongoDatabase mongoDatabase = mongoClient.GetDatabase("Sandbox");
IMongoCollection<BsonDocument> collection = mongoDatabase.GetCollection<BsonDocument>("TestCollection");
var changeStream = collection.Watch().ToEnumerable().GetEnumerator();
changeStream.MoveNext();
var next = changeStream.Current;
Then I downloaded the C# source code from MongoDB to see how they did this. Looking at their test code for change stream watches, they create a new document(Insert) and then change that document right away(Update) and THEN set up the Change Stream watch to receive an 'update' notification.
No example is given on how to watch for 'insert' notifications.
I have looked at the Java and NodeJS examples, both on MongoDB website and SO, which seems to be straight forward and defines a way to see both Inserts and Updates:
var changeStream = collection.watch({ '$match': { $or: [ { 'operationType': 'insert' }, { 'operationType': 'update' } ] } });
The API for the C# driver is vastly different, I would have assumed they would have kept the same API for C# as Java and NodeJS. I found no or very few examples for C# to do the same thing.
The closest I have come is with the following attempt but still fails and the documentation for the C# version is very limited (or I have not found the right location). Setup is as follows:
String json = "{ '$match': { 'operationType': { '$in': ['insert', 'update'] } } }";
var options = new ChangeStreamOptions { FullDocument = ChangeStreamFullDocumentOption.UpdateLookup };
PipelineDefinition<ChangeStreamDocument<BsonDocument>, ChangeStreamDocument<BsonDocument>> pipeline = new EmptyPipelineDefinition<ChangeStreamDocument<BsonDocument>>().Match(Builders<ChangeStreamDocument<BsonDocument>>.Filter.Text(json,"json"));
Then running the statement below throws an Exception:
{"Command aggregate failed: $match with $text is only allowed as the
first pipeline stage."}
No other Filter options has worked either, and I have not found a way to just enter the JSON as a string to set the 'operationType'.
var changeStream = collection.Watch(pipeline, options).ToEnumerable().GetEnumerator();
changeStream.MoveNext();
var next = changeStream.Current;
My only goal here is to be able to set the 'operationType' using the C# driver. Does anyone know what I am doing wrong or have tried this using the C# driver and had success?
After reading though a large number of webpages, with very little info on the C# version of the MongoDB driver, I am very stuck!
Any help would be much appreciated.

Here is a sample of code I've used to update the collection Watch to retrieve "events" other than just document updates.
IMongoDatabase sandboxDB = mongoClient.GetDatabase("Sandbox");
IMongoCollection<BsonDocument> collection = sandboxDB.GetCollection<BsonDocument>("TestCollection");
//Get the whole document instead of just the changed portion
ChangeStreamOptions options = new ChangeStreamOptions() { FullDocument = ChangeStreamFullDocumentOption.UpdateLookup };
//The operationType can be one of the following: insert, update, replace, delete, invalidate
var pipeline = new EmptyPipelineDefinition<ChangeStreamDocument<BsonDocument>>().Match("{ operationType: { $in: [ 'replace', 'insert', 'update' ] } }");
var changeStream = collection.Watch(pipeline, options).ToEnumerable().GetEnumerator();
changeStream.MoveNext(); //Blocks until a document is replaced, inserted or updated in the TestCollection
ChangeStreamDocument<BsonDocument> next = changeStream.Current;
enumerator.Dispose();
The EmptyPiplineDefinition...Match() argument could also be:
"{ $or: [ {operationType: 'replace' }, { operationType: 'insert' }, { operationType: 'update' } ] }"
If you wanted to use the $or command, or
"{ operationType: /^[^d]/ }"
to throw a little regex in there. This last one is saying, I want all operationTypes unless they start with the letter 'd'.

Save WinForm or Controls to File

I have been working on an application which allows the user to make a label template for printing purposes by adding label controls to a panel(which I use as a container). I have reached the point where I need to be able to save the template to a file which I can load into memory later for printing. Since the form is not serializable does anyone have suggestions on how I can save the form or container(with added label controls) to a file that can be reused later?
Thanks.

I wouldn't directly serialize a form to a file. Sounds like you need to create a class that will hold the state of the user's work. You should then serialize that class to and from a file. There are built in methods for that using either Binary or XML Serialization.

Create a struct that contains enough information (and no more) about each Label that you can reconstitute the Label from it.
Write a method that takes a List<MyStruct> and populates a Panel from your structs.
Write methods to serialize and deserialize this list.
Encapsulate the whole thing in a class.

Try this. It uses the ISerializationSurrogate interface to get around the problem of the form object not being serializable:
How to serialize an object which is NOT marked as 'Serializable' using a surrogate.
http://www.codeproject.com/KB/dotnet/Surrogate_Serialization.aspx

Personally, I would serialize it as JSON.
When bringing it back you can use a generic method
that loops through and sets the properties through reflection.
Also take notice that the library I've linked to will automatically serialize objects that you pass to it.
JSON
JSON.NET
[{ "Label": [{"Top": 102}, {"Left": 105}, {"Text": "blah, blah"}] }]
From JSON.NET
Product product = new Product();
product.Name = "Apple";
product.Expiry = new DateTime(2008, 12, 28);
product.Price = 3.99M;
product.Sizes = new string[] { "Small", "Medium", "Large" };
string json = JsonConvert.SerializeObject(product);
//{
// "Name": "Apple",
// "Expiry": new Date(1230422400000),
// "Price": 3.99,
// "Sizes": [
// "Small",
// "Medium",
// "Large"
// ]
//}
Product deserializedProduct = JsonConvert.DeserializeObject<Product>(json);

You can get the position, size and other properties about the form's controls at runtime and save that state in an XML or JSON file.

This isn't trivial, but personally I would set up a function that can be called recursively that would add nodes to an XML file.
I don't have actual code, but pseudo-code looks like this: (you will need to do some clean-up, because I'm doing this off the top of my head without the aid of Intellisense.)
XmlDocument doc;
function SaveForm()
{
doc = new XmlDocument("FormInfo");
foreach(Control ctrl in this.Controls)
{
AddControlToXml(ctrl, doc.Documentelement);
}
}
function AddControlToXml(Control ctrl, XmlNode currentNode)
{
XmlNode n = new XmlNode;
Node.InnerText = ctrl.Name;
foreach(Control ctrl2 in ctrl.Controls)
{
AddControlToXml(ctrl2);
}
}