Develop Reference

How to serialize tensor input required by dnnclassifier (serving_input_reciever)

I want to be able to use the dnnclassifier (estimator) on top of IIS using tensorflowsharp. The model has previously been trained in python. I got so far that I can now generate PB files, know the correct input/outputs, however I am stuck in tensorflowsharp using string inputs.
I can create a valid .pb file of the iris dataset. It uses the following feate_spec:
{'SepalLength': FixedLenFeature(shape=(1,), dtype=tf.float32, default_value=None), 'SepalWidth': FixedLenFeature(shape=(1,), dtype=tf.float32, default_value=None), 'PetalLength': FixedLenFeature(shape=(1,), dtype=tf.float32, default_value=None), 'PetalWidth': FixedLenFeature(shape=(1,), dtype=tf.float32, default_value=None)}
I have created a simple c# console to try and spin it up. The input should be an "input_example_tensor" and the output is located in "dnn/head/predictions/probabilities". This I discoved after alex_zu provided help using the saved_model_cli command here.
As far as I am aware all tensorflow estimator API's work like this.
Here comes the problem: the input_example_tensor should be of a string format which will be parsed internally by the ParseExample function. Now i am stuck. I have found TFTensor.CreateString, but this doesn't solve the problem.
using System;
using TensorFlow;
namespace repository
{
class Program
{
static void Main(string[] args)
{
using (TFGraph tfGraph = new TFGraph()){
using (var tmpSess = new TFSession(tfGraph)){
using (var tfSessionOptions = new TFSessionOptions()){
using (var metaGraphUnused = new TFBuffer()){
//generating a new session based on the pb folder location with the tag serve
TFSession tfSession = tmpSess.FromSavedModel(
tfSessionOptions,
null,
#"path/to/model/pb",
new[] { "serve" },
tfGraph,
metaGraphUnused
);
//generating a new runner, which will fetch the tensorflow results later
var runner = tfSession.GetRunner();
//this is in the actual tensorflow documentation, how to implement this???
string fromTensorflowPythonExample = "{'SepalLength': [5.1, 5.9, 6.9],'SepalWidth': [3.3, 3.0, 3.1],'PetalLength': [1.7, 4.2, 5.4],'PetalWidth': [0.5, 1.5, 2.1],}";
//this is the problem, it's not working...
TFTensor rawInput = new TFTensor(new float[4]{5.1f,3.3f,1.7f,0.5f});
byte[] serializedTensor = System.Text.Encoding.ASCII.GetBytes(rawInput.ToString());
TFTensor inputTensor = TensorFlow.TFTensor.CreateString (serializedTensor);
runner.AddInput(tfGraph["input_example_tensor"][0], inputTensor);
runner.Fetch("dnn/head/predictions/probabilities", 0);
//start the run and get the results of the iris example
var output = runner.Run();
TFTensor result = output[0];
//printing response to the client
Console.WriteLine(result.ToString());
Console.ReadLine();
}
}
}
}
}
}
}
This example will give the following error:
An unhandled exception of type 'TensorFlow.TFException' occurred in TensorFlowSharp.dll: 'Expected serialized to be a vector, got shape: []
[[Node: ParseExample/ParseExample = ParseExample[Ndense=4, Nsparse=0, Tdense=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], dense_shapes=[[1], [1], [1], [1]], sparse_types=[], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_input_example_tensor_0_0, ParseExample/ParseExample/names, ParseExample/ParseExample/dense_keys_0, ParseExample/ParseExample/dense_keys_1, ParseExample/ParseExample/dense_keys_2, ParseExample/ParseExample/dense_keys_3, ParseExample/Const, ParseExample/Const, ParseExample/Const, ParseExample/Const)]]'
How can I serialize tensors in such a way that I can use the pb file correctly?
I also posted the issue on github, here you can find the iris example python file, pb file and the console applications. In my opinion solving this creates a
neat solution for all tensorflow users having ancient production environments (like me).

The Expected serialized to be a vector, got shape: [] error can be fixed by using an overload of the TFTensor.CreateString function: Instead of directly taking a string, the model apparently expects a vector containing a single string:
TFTensor inputTensor = TFTensor.CreateString(new byte[][] { bytes }, new TFShape(1));
The input_example_tensor in your case now expects a serialized Example protobuf message (see also the docs and the example.proto file).
Using the protobuf compiler, I've generated a C# file containing the Example class. You can download it from here: https://pastebin.com/iLT8MUdR. Specifically, I used this online tool with CSharpProtoc and replaced the import "tensorflow/core/example/feature.proto"; line by the messages defined in that file.
Once you've added the file to your project, you'll need a package reference to Google.Protobuf. Then, you can pass serialized examples to the model like this:
Func<float, Tensorflow.Feature> makeFeature = (float x) => {
var floatList = new Tensorflow.FloatList();
floatList.Value.Add(x);
return new Tensorflow.Feature { FloatList = floatList };
};
var example = new Tensorflow.Example { Features = new Tensorflow.Features() };
example.Features.Feature.Add("SepalLength", makeFeature(5.1f));
example.Features.Feature.Add("SepalWidth", makeFeature(3.3f));
example.Features.Feature.Add("PetalLength", makeFeature(1.7f));
example.Features.Feature.Add("PetalWidth", makeFeature(0.5f));
TFTensor inputTensor = TFTensor.CreateString(
new [] { example.ToByteArray() }, new TFShape(1));
runner.AddInput(tfGraph["input_example_tensor"][0], inputTensor);
runner.Fetch("dnn/head/predictions/probabilities", 0);
//start the run and get the results of the iris example
var output = runner.Run();
TFTensor result = output[0];

How to get all installations when using Azure Notification Hubs installation model?

Using NotificationHubClient I can get all registered devices using GetAllRegistrationsAsync(). But if I do not use the registration model but the installation model instead, how can I get all installations? There are methods to retrieve a specific installation but none to get everything.

You're correct, as of July 2016 there's no way to get all installations for a hub. In the future, the product team is planning to add this feature to the installations model, but it will work in a different way. Instead of making it a runtime operation, you'll provide your storage connection string and you'll get a blob with everything associated with the hub.

Sorry for visiting an old thread... but in theory you could use the GetAllRegistrationsAsyc to get all the installations. I guess this will return everything without an installation id as well, but you could just ignore those if you choose.
Could look something like this
var allRegistrations = await _hub.GetAllRegistrationsAsync(0);
var continuationToken = allRegistrations.ContinuationToken;
var registrationDescriptionsList = new List<RegistrationDescription>(allRegistrations);
while (!string.IsNullOrWhiteSpace(continuationToken))
{
var otherRegistrations = await _hub.GetAllRegistrationsAsync(continuationToken, 0);
registrationDescriptionsList.AddRange(otherRegistrations);
continuationToken = otherRegistrations.ContinuationToken;
}
// Put into DeviceInstallation object
var deviceInstallationList = new List<DeviceInstallation>();
foreach (var registration in registrationDescriptionsList)
{
var deviceInstallation = new DeviceInstallation();
var tags = registration.Tags;
foreach(var tag in tags)
{
if (tag.Contains("InstallationId:"))
{
deviceInstallation.InstallationId = new Guid(tag.Substring(tag.IndexOf(":")+1));
}
}
deviceInstallation.PushHandle = registration.PnsHandle;
deviceInstallation.Tags = new List<string>(registration.Tags);
deviceInstallationList.Add(deviceInstallation);
}
I am not suggesting this to be the cleanest chunk of code written, but it does the trick for us. We only use this for debugging type purposes anyways

NetSuite custom record search through suiteTalk using C#

We are having an issue with searching a custom record through SuiteTalk. Below is a sample of what we are calling. The issue we are having is in trying to set up the search using the internalId of the record. The issue here lies in in our initial development account the internal id of this custom record is 482 but when we deployed it through the our bundle the record was assigned with the internal Id of 314. It would stand to reason that this internal id is not static in a site per site install so we wondered what property to set up to reference the custom record. When we made the record we assigned its “scriptId’ to be 'customrecord_myCustomRecord' but through suitetalk we do not have a “scriptId”. What is the best way for us to allow for this code to work in all environments and not a specific one? And if so, could you give an example of how it might be used.
Code (C#) that we are attempting to make the call from. We are using the 2013.2 endpoints at this time.
private SearchResult NetSuite_getPackageContentsCustomRecord(string sParentRef)
{
List<object> PackageSearchResults = new List<object>();
CustomRecord custRec = new CustomRecord();
CustomRecordSearch customRecordSearch = new CustomRecordSearch();
SearchMultiSelectCustomField searchFilter1 = new SearchMultiSelectCustomField();
searchFilter1.internalId = "customrecord_myCustomRecord_sublist";
searchFilter1.#operator = SearchMultiSelectFieldOperator.anyOf;
searchFilter1.operatorSpecified = true;
ListOrRecordRef lRecordRef = new ListOrRecordRef();
lRecordRef.internalId = sParentRef;
searchFilter1.searchValue = new ListOrRecordRef[] { lRecordRef };
CustomRecordSearchBasic customRecordBasic = new CustomRecordSearchBasic();
customRecordBasic.recType = new RecordRef();
customRecordBasic.recType.internalId = "314"; // "482"; //THIS LINE IS GIVING US THE TROUBLE
//customRecordBasic.recType.name = "customrecord_myCustomRecord";
customRecordBasic.customFieldList = new SearchCustomField[] { searchFilter1 };
customRecordSearch.basic = customRecordBasic;
// Search for the customer entity
SearchResult results = _service.search(customRecordSearch);
return results;
}

I searched all over for a solution to avoid hardcoding internalId's. Even NetSuite support failed to give me a solution. Finally I stumbled upon a solution in NetSuite's knowledgebase, getCustomizationId.
This returns the internalId, scriptId and name for all customRecord's (or customRecordType's in NetSuite terms! Which is what made it hard to find.)
public string GetCustomizationId(string scriptId)
{
// Perform getCustomizationId on custom record type
CustomizationType ct = new CustomizationType();
ct.getCustomizationTypeSpecified = true;
ct.getCustomizationType = GetCustomizationType.customRecordType;
// Retrieve active custom record type IDs. The includeInactives param is set to false.
GetCustomizationIdResult getCustIdResult = _service.getCustomizationId(ct, false);
foreach (var customizationRef in getCustIdResult.customizationRefList)
{
if (customizationRef.scriptId == scriptId) return customizationRef.internalId;
}
return null;
}

you can make the internalid as an external property so that you can change it according to environment.
The internalId will be changed only when you install first time into an environment. when you deploy it into that environment, the internalid will not change with the future deployments unless you choose Add/Rename option during deployment.

Creating FedEx Shipping Documnents using FedEx ship service WSDL

I am in the process of integrating with the FedEx international Ship Service. But I am really stuck on one part. I am trying to create a certificate of origin using their test environment. I have followed the xml schema and have come up with the code below
private static void SetCustomInvoice(ProcessShipmentRequest request)
{
request.RequestedShipment.ShippingDocumentSpecification = new ShippingDocumentSpecification();
request.RequestedShipment.ShippingDocumentSpecification.ShippingDocumentTypes = new RequestedShippingDocumentType[1] { new RequestedShippingDocumentType() };
request.RequestedShipment.ShippingDocumentSpecification.ShippingDocumentTypes[0] = RequestedShippingDocumentType.CERTIFICATE_OF_ORIGIN;
request.RequestedShipment.ShippingDocumentSpecification.CertificateOfOrigin = new CertificateOfOriginDetail();
request.RequestedShipment.ShippingDocumentSpecification.CertificateOfOrigin.DocumentFormat = new ShippingDocumentFormat { StockType = ShippingDocumentStockType.STOCK_4X6, ImageType = ShippingDocumentImageType.PDF, ImageTypeSpecified = true, StockTypeSpecified = true };
request.RequestedShipment.SpecialServicesRequested = new ShipmentSpecialServicesRequested();
request.RequestedShipment.SpecialServicesRequested.SpecialServiceTypes = new ShipmentSpecialServiceType[1] { new ShipmentSpecialServiceType() };
request.RequestedShipment.SpecialServicesRequested.SpecialServiceTypes[0] = ShipmentSpecialServiceType.ELECTRONIC_TRADE_DOCUMENTS;
request.RequestedShipment.SpecialServicesRequested.EtdDetail = new EtdDetail();
request.RequestedShipment.SpecialServicesRequested.EtdDetail.RequestedDocumentCopies = new RequestedShippingDocumentType[1] { RequestedShippingDocumentType.CERTIFICATE_OF_ORIGIN };
request.RequestedShipment.SpecialServicesRequested.EtdDetail.DocumentReferences = new UploadDocumentReferenceDetail[1] { new UploadDocumentReferenceDetail() };
request.RequestedShipment.SpecialServicesRequested.EtdDetail.RequestedDocumentCopies[0] = RequestedShippingDocumentType.CERTIFICATE_OF_ORIGIN;
}
But I keep getting an error message back from the web service stating “Invalid Stock Type”. Even though the shipmentDocumentStockType is an enum and I am using one of the values from it. I am still getting this error. Any ideas where I might be going wrong?
Any information will be a great help. I have tried getting in touch with FedEx technical support and they were not really a great help.

This might be a little late for you but just wanted to provide an answer in case someone else might be looking for it.
I was having a similar problem but instead of the Certificate of Origin it was with the Commercial Invoice. Turns out these forms need to be printed on a full 8.5 x 11 page in PDF format, so changing the StockType from STOCK_4x6 to PAPER_LETTER fixed it for me:
From:
request.RequestedShipment.ShippingDocumentSpecification.CertificateOfOrigin.DocumentFormat = new ShippingDocumentFormat { StockType = ShippingDocumentStockType.STOCK_4X6, ImageType = ShippingDocumentImageType.PDF, ImageTypeSpecified = true, StockTypeSpecified = true };
To:
request.RequestedShipment.ShippingDocumentSpecification.CertificateOfOrigin.DocumentFormat = new ShippingDocumentFormat { StockType = ShippingDocumentStockType.PAPER_LETTER, ImageType = ShippingDocumentImageType.PDF, ImageTypeSpecified = true, StockTypeSpecified = true };
Hope this helps

Processing PHP scripts from within C#

I'm not sure if this has even been done before, but what I am trying to accomplish can't be explained in any great detail, but basically, what I am trying to do is Process PHP scripts from within my C# Windows Forms Application.
I have already created a HTTP server, which works just fine. But now I need to be able to process PHP scripts aswell. We're working on a new privatised language over here, purely for learning and fun.
(Just a little background, not completely related):
We are now able to create a webpage like so:
Using System.Core,
System.Http,
System.Graphics and System.IO;
protected void -> Webpage(object homepage)
{
// Set Webpage properties.
homepage.Name = "Home";
homepage.Size = new Size(960[px], 100[%]);
homepage.Alignment = new AlignmentType.Vertical;
homepage.Alignment = new AlignmentType.Horizontal;
// This is a comment.
// Create objects to be rendered on the page.
Text text = new Text();
FormElements formElements = new FormElements();
private void -> Webpage.Load(object homepage)
{
text.Text = "Please enter your name below:";
text.Style = new Style(
Alignment = new AlignmentType.Horizontal,
Alignment = new AlignmentType.ManualAlignment(15[Y, px]),
Font = new Font(
Font.Family("Arial"),
Font.Size = new Size(9[pt], LineHeight(4[px])),
Font.Color = new Color.FromArgb(15, 15, 15))
);
formElements.CreateElements(TextField["textField"], SubmitButton["submitButton"], Form["form"]);
textField.Name = "name";
submitButton.Name = "submit";
form.Encapsulate(name, submit);
form.Alignment = new AlignmentType.RelativeTo(text.Bottom);
Elements[] elements = new Elements[]
{
text, form;
};
homepage.Display.Element.ElementCollection(elements);
}
private void -> Webpage.FormSubmission(object form)
{
form.Element(name).OmitSpecialCharacters();
if(form.Value is not Empty)
{
text.Text = "Hello, " + form.Element(name).Value;
}
}
}
The above sample demonstrates the ability to create a whole webpage, style it, and process form input in a nice, clean way. However, we've come to a complete dead end (trying to support PHP) and we do not wish to delve too far into server-side languages (lack of experience in that area is the main reason), so we would like to be able to "support" PHP scripts from within our WinForms app.
Anyone know of any way to process PHP scripts from within a C# winforms app?

Here is a simple http server that supports PHP:
MiniHttpd: an HTTP web server library
And here is Phalanger - The PHP Language Compiler for the .NET Framework

Use an HttpWebRequest, and request the url to the PHP page like you would to run the PHP page in your browser.
See here for more info: http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx