Pass JSON to C# python process

Pass JSON to C# python process - c#

I have a little problem. I want to start a python process from C# and I need to pass some data to it. The data is in json format, but if I serialize the data from c# it look like this
"[{\"SearchTerm_id\":1,\"Term\":\"lorem ipsum\"},{\"SearchTerm_id\":2,\"Term\":\"lorem ipsum\"}]}"
and is not valid for python because of the scapation for double quote.
How I can pass the data from c# to a python script ?
This is my code:
List<SearchTerms> searchTerms = await _context.SearchTerms.ToListAsync();
var json = JsonConvert.SerializeObject(searchTerms);
ProcessStartInfo processInfo = new ProcessStartInfo();
string scriptPath = Path.Combine(System.IO.Directory.GetCurrentDirectory(), "Python\\scrapeGoogle.py");
processInfo.FileName = "python3";
processInfo.Arguments = string.Format("{0} {1}", scriptPath, json);
processInfo.UseShellExecute = false;
processInfo.CreateNoWindow = true;
processInfo.RedirectStandardOutput = true;
processInfo.RedirectStandardError = true;
process.StartInfo = processInfo;
process.Start();

If you are viewing the result of var json = JsonConvert.SerializeObject(searchTerms); in the debugger. The debugger only shows the \ as a visual aid, representing the string just as it would have to be written in c#.
Try running Console.Write(json); and view the output.
The output won't contain the escape character. This is the true value.

Just solved this for myself, so thought I would comment.
The Problem
#Filip Laurentiu (OP) correctly identified his problem when replying to #A Redfearn's answer.
You are right. So the problem seams to be elsewhere because if I have
only a print(sys.argv1) in my python, I will recive my data as
follow : "[{SearchTerm_id:1,Term:Lorem ipsun},{SearchTerm_id:2,Lorem
Ipsum}]"
C# is actually formatting the JSON without escape characters, contrary to what the original question said. So it is doing what it is supposed to do. If you print the text, you see {"SearchTerm_id":1,"Term":"lorem ipsum"}.
However, Python is receiving the JSON without the double quotes. Printing from Python shows {SearchTerm_id:1,Term:lorem ipsum}. And when you call json.loads with that bad JSON, it throws.
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
It seems that when you call Process.Start() in C#, the shell strips the double quotes from the JSON in your ProcessStartInfo.Arguments list. So Python receives the argument without double quotes.
The Solution
In C#, update the JSON string after serializing, so that you escape the double quotes. Here is some code to help with that.
using Newtonsoft.Json;
using System.Text;
public static class JsonHelper
{
public static string ToJsonString(
object obj,
bool escapeDoubleQuotes = false)
{
string serialized = JsonConvert.SerializeObject(obj);
if (escapeDoubleQuotes)
{
// e.g., '{"key":"value"}' -> '{\"key1\":\"value\"}'
// Do this when need to pass json as cmd line arg via System.Diagnostics.Process. Else shell will strip the double quotes, so your new process might not know what to do with it
serialized = serialized.Replace(#"""", #"\""");
}
return serialized;
}
}
So to get his original code working, OP would just change
var json = JsonConvert.SerializeObject(searchTerms);
to
var json = JsonHelper.ToJsonString(searchTerms, escapeDoubleQuotes: true);

You can use messagePack library. I managed to pass strings and other types between python and c# with high performance.

Related

How to serialize tensor input required by dnnclassifier (serving_input_reciever)

I want to be able to use the dnnclassifier (estimator) on top of IIS using tensorflowsharp. The model has previously been trained in python. I got so far that I can now generate PB files, know the correct input/outputs, however I am stuck in tensorflowsharp using string inputs.
I can create a valid .pb file of the iris dataset. It uses the following feate_spec:
{'SepalLength': FixedLenFeature(shape=(1,), dtype=tf.float32, default_value=None), 'SepalWidth': FixedLenFeature(shape=(1,), dtype=tf.float32, default_value=None), 'PetalLength': FixedLenFeature(shape=(1,), dtype=tf.float32, default_value=None), 'PetalWidth': FixedLenFeature(shape=(1,), dtype=tf.float32, default_value=None)}
I have created a simple c# console to try and spin it up. The input should be an "input_example_tensor" and the output is located in "dnn/head/predictions/probabilities". This I discoved after alex_zu provided help using the saved_model_cli command here.
As far as I am aware all tensorflow estimator API's work like this.
Here comes the problem: the input_example_tensor should be of a string format which will be parsed internally by the ParseExample function. Now i am stuck. I have found TFTensor.CreateString, but this doesn't solve the problem.
using System;
using TensorFlow;
namespace repository
{
class Program
{
static void Main(string[] args)
{
using (TFGraph tfGraph = new TFGraph()){
using (var tmpSess = new TFSession(tfGraph)){
using (var tfSessionOptions = new TFSessionOptions()){
using (var metaGraphUnused = new TFBuffer()){
//generating a new session based on the pb folder location with the tag serve
TFSession tfSession = tmpSess.FromSavedModel(
tfSessionOptions,
null,
#"path/to/model/pb",
new[] { "serve" },
tfGraph,
metaGraphUnused
);
//generating a new runner, which will fetch the tensorflow results later
var runner = tfSession.GetRunner();
//this is in the actual tensorflow documentation, how to implement this???
string fromTensorflowPythonExample = "{'SepalLength': [5.1, 5.9, 6.9],'SepalWidth': [3.3, 3.0, 3.1],'PetalLength': [1.7, 4.2, 5.4],'PetalWidth': [0.5, 1.5, 2.1],}";
//this is the problem, it's not working...
TFTensor rawInput = new TFTensor(new float[4]{5.1f,3.3f,1.7f,0.5f});
byte[] serializedTensor = System.Text.Encoding.ASCII.GetBytes(rawInput.ToString());
TFTensor inputTensor = TensorFlow.TFTensor.CreateString (serializedTensor);
runner.AddInput(tfGraph["input_example_tensor"][0], inputTensor);
runner.Fetch("dnn/head/predictions/probabilities", 0);
//start the run and get the results of the iris example
var output = runner.Run();
TFTensor result = output[0];
//printing response to the client
Console.WriteLine(result.ToString());
Console.ReadLine();
}
}
}
}
}
}
}
This example will give the following error:
An unhandled exception of type 'TensorFlow.TFException' occurred in TensorFlowSharp.dll: 'Expected serialized to be a vector, got shape: []
[[Node: ParseExample/ParseExample = ParseExample[Ndense=4, Nsparse=0, Tdense=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], dense_shapes=[[1], [1], [1], [1]], sparse_types=[], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_input_example_tensor_0_0, ParseExample/ParseExample/names, ParseExample/ParseExample/dense_keys_0, ParseExample/ParseExample/dense_keys_1, ParseExample/ParseExample/dense_keys_2, ParseExample/ParseExample/dense_keys_3, ParseExample/Const, ParseExample/Const, ParseExample/Const, ParseExample/Const)]]'
How can I serialize tensors in such a way that I can use the pb file correctly?
I also posted the issue on github, here you can find the iris example python file, pb file and the console applications. In my opinion solving this creates a
neat solution for all tensorflow users having ancient production environments (like me).

The Expected serialized to be a vector, got shape: [] error can be fixed by using an overload of the TFTensor.CreateString function: Instead of directly taking a string, the model apparently expects a vector containing a single string:
TFTensor inputTensor = TFTensor.CreateString(new byte[][] { bytes }, new TFShape(1));
The input_example_tensor in your case now expects a serialized Example protobuf message (see also the docs and the example.proto file).
Using the protobuf compiler, I've generated a C# file containing the Example class. You can download it from here: https://pastebin.com/iLT8MUdR. Specifically, I used this online tool with CSharpProtoc and replaced the import "tensorflow/core/example/feature.proto"; line by the messages defined in that file.
Once you've added the file to your project, you'll need a package reference to Google.Protobuf. Then, you can pass serialized examples to the model like this:
Func<float, Tensorflow.Feature> makeFeature = (float x) => {
var floatList = new Tensorflow.FloatList();
floatList.Value.Add(x);
return new Tensorflow.Feature { FloatList = floatList };
};
var example = new Tensorflow.Example { Features = new Tensorflow.Features() };
example.Features.Feature.Add("SepalLength", makeFeature(5.1f));
example.Features.Feature.Add("SepalWidth", makeFeature(3.3f));
example.Features.Feature.Add("PetalLength", makeFeature(1.7f));
example.Features.Feature.Add("PetalWidth", makeFeature(0.5f));
TFTensor inputTensor = TFTensor.CreateString(
new [] { example.ToByteArray() }, new TFShape(1));
runner.AddInput(tfGraph["input_example_tensor"][0], inputTensor);
runner.Fetch("dnn/head/predictions/probabilities", 0);
//start the run and get the results of the iris example
var output = runner.Run();
TFTensor result = output[0];

HTTP POST read response as regular string

I am currently doing a API HTTP POST project.
I should be getting
"[{\"ID\":\"311d1977-4772-435a-92aa-028791c53154\",\"ParentID\":\"187a064e-ffea-45a2-9264-9acecff911e1\",\"Type\":\"txt\",\"OwnerID\":\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\",\"Name\":\"Davinshi.txt\",\"CreatedBy\":\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\",\"CreatedDate\":\"2015-06-05T04:11:33.187\",\"Status\":0,\"Mark\":null},{\"ID\":\"25c80f4e-679c-4093-ade9-0b99da480153\",\"ParentID\":\"187a064e-ffea-45a2-9264-9acecff911e1\",\"Type\":\"jpg\",\"OwnerID\":\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\",\"Name\":\"leonid afremov.jpg\",\"CreatedBy\":\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\",\"CreatedDate\":\"2015-06-05T04:11:46.61\",\"Status\":0,\"Mark\":null}
However with the follow code I am getting a weird output
using (var client = new WebClient())
{
var val = new NameValueCollection();
foreach (var item in values)
{
val.Add(item.Key, item.Value);
}
var response = client.UploadValues(url, val);
return Encoding.Default.GetString(response);
}
This is the output
\"[{\\\"ID\\\":\\\"311d1977-4772-435a-92aa-028791c53154\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"txt\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"Davinshi.txt\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:33.187\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"25c80f4e-679c-4093-ade9-0b99da480153\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"jpg\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"leonid afremov.jpg\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:46.61\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"5648cc67-1408-4935-a656-0f9b5116db8d\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"jpg\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"misty mood leonid.jpg\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:49.437\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"7402e7b8-1ec4-4bf8-9042-142a69cecbcd\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"jpg\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"winter sparkle.jpg\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:12:03.26\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"7159e926-1601-41d6-8dce-20e5daab90a6\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"txt\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"gentle rain beata sasik.txt\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:40.6\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"e61f7bc4-bebf-403b-a478-22b36856a6df\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"jpg\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"MoonLight.jpg\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:50.797\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"e46e6250-655b-4105-a888-2574750f2944\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"jpg\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"blue fire.jpg\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:27.037\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"bfca703b-1720-4a67-b0a4-2f6146c820aa\\\",\\\"ParentID\\\":null,\\\"Type\\\":\\\"folder\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"School\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:17.96\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"5c1bf9ca-b368-46c8-9804-3254eda00806\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"txt\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"Woman.txt\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:12:06.607\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"8ad07f3c-7a57-4add-92c0-329aca372c8b\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"jpg\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"intimacy worship.jpg\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:40.727\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"ac5384c4-66ca-45cd-99f0-437c0789a26a\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"txt\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"Under the Rain.txt\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:12:01.007\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"aa731f04-cc4e-4349-9268-4476ff04b473\\\",\\\"ParentID\\\":\\\"bfca703b-1720-4a67-b0a4-2f6146c820aa\\\",\\\"Type\\\":\\\"folder\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"Marsiling Secondary School\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:20.983\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"af650810-0538-432e-8bf3-47429eb17d27\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"jpg\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"finger paintings.jpg\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:35.65\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"ddac31de-2b67-4f56-aefd-508a8f49926f\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"txt\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"My Love.txt\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:56.55\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"f2454474-d834-4bd1-992e-548cdc2734c1\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"txt\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"boat.txt\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:28.813\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"e1f5be01-29c7-4ca9-ae6f-5ddf8638cc67\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"jpg\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"Davinshi.jpg\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:34.687\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"ff3a7365-7b4e-4a46-b545-6a2f889ff898\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"jpg\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"landscape paintings.jpg\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:44.48\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"327bf043-1e88-4265-ba5e-6de8394a0d84\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"jpg\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"lord rama.jpg\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:47.753\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"1d8a01de-da06-466d-8b7d-6e0bb8478d5e\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"txt\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"intimacy worship.txt\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:41.617\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"f23103ec-0d8c-407b-9f20-719f0577c274\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"jpg\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"Coast of Sicily.jpg\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:32.187\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"374779d5-16e9-417f-808f-87fcf85035b0\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"txt\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"misty mood leonid.txt\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:54.203\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"f0bb2dc7-eac3-407d-a85a-886aaf161fc2\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"jpg\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"boat.jpg\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:28.907\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"607ee7b8-7287-4f3f-9867-8a8069562d3c\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"jpg\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"Ballet.jpg\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:24.167\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"246b349e-ab32-407a-90ff-8df18bbabc68\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"txt\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"Ballet.txt\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:23.917\\\",\\\"Status\\\":0,\\\"Mark\\\":null},{\\\"ID\\\":\\\"457d722f-a675-4523-bd83-8f309dcd254e\\\",\\\"ParentID\\\":\\\"187a064e-ffea-45a2-9264-9acecff911e1\\\",\\\"Type\\\":\\\"txt\\\",\\\"OwnerID\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"Name\\\":\\\"finger paintings.txt\\\",\\\"CreatedBy\\\":\\\"5bd087f0-070d-48bd-98b4-10ffa4b3b792\\\",\\\"CreatedDate\\\":\\\"2015-06-05T04:11:39.017\\\",\\\"Status\\\":0,\\\
Which I think the HTTP POST Response is reading the string as a verbatim string.
Any idea how can i fix it ?
Thanks !

When I run your code, the output of Encoding.Default.GetString(response) is exactly the string you posted at the top of your question.
The excessively backslash-escaped string appears in the Visual Studio debugging window simply because VS escapes special characters when displaying a string, but if you try to save this string into a file or use it in any other way in your code, you will see that only the double quote marks are escaped, like the string at the top of your question.

I found out that the API will automatically convert the object to json format.
The problem occur when I try to serialize the object before sending trough the API.Hence, like what Denis Yarkovoy said. It's double serialized.

Unexpected character encountered while parsing value

Currently, I have some issues. I'm using C# with Json.NET. The issue is that I always get:
{"Unexpected character encountered while parsing value: e. Path '', line 0, position 0."}
So the way I'm using Json.NET is the following. I have a Class which should be saved. The class looks like this:
public class stats
{
public string time { get; set; }
public string value { get; set; }
}
public class ViewerStatsFormat
{
public List<stats> viewerstats { get; set; }
public String version { get; set; }
public ViewerStatsFormat(bool chk)
{
this.viewerstats = new List<stats>();
}
}
One object of this class will be filled and saved with:
File.WriteAllText(tmpfile, JsonConvert.SerializeObject(current), Encoding.UTF8);
The saving part works fine and the file exists and is filled. After that the file will be read back into the class with:
try
{
ViewerStatsFormat current = JsonConvert.DeserializeObject<ViewerStatsFormat>(tmpfile);
//otherstuff
}
catch(Exception ex)
{
//error loging stuff
}
Now on the current= line comes the exception:
{"Unexpected character encountered while parsing value: e. Path '', line 0, position 0."}
I don't know why this comes. The JSON file is the following -> Click me I am the JSON link
Does anyone have any ideas?

Possibly you are not passing JSON to DeserializeObject.
It looks like from File.WriteAllText(tmpfile,... that type of tmpfile is string that contain path to a file. JsonConvert.DeserializeObject takes JSON value, not file path - so it fails trying to convert something like #"c:\temp\fooo" - which is clearly not JSON.

I solved the problem with these online tools:
To check if the Json structure is OKAY: http://jsonlint.com/
To generate my Object class from my Json structure: https://www.jsonutils.com/
The simple code:
RootObject rootObj= JsonConvert.DeserializeObject<RootObject>(File.ReadAllText(pathFile));

In my case, the file containing JSON string had BOM. Once I removed BOM the problem was solved.

I experienced the same error in my Xamarin.Android solution.
I verified that my JSON was correct, and noticed that the error only appeared when I ran the app as a Release build.
It turned out that the Linker was removing a library from Newtonsoft.JSON, causing the JSON to be parsed incorrectly.
I fixed the error by adding Newtonsoft.Json to the Ignore assemblies setting in the Android Build Configuration (screen shot below)
JSON Parsing Code
static readonly JsonSerializer _serializer = new JsonSerializer();
static readonly HttpClient _client = new HttpClient();
static async Task<T> GetDataObjectFromAPI<T>(string apiUrl)
{
using (var stream = await _client.GetStreamAsync(apiUrl).ConfigureAwait(false))
using (var reader = new StreamReader(stream))
using (var json = new JsonTextReader(reader))
{
if (json == null)
return default(T);
return _serializer.Deserialize<T>(json);
}
}
Visual Studio Mac Screenshot
Visual Studio Screenshot

I have also encountered this error for a Web API (.Net Core 3.0) action that was binding to a string instead to an object or a JObject. The JSON was correct, but the binder tried to get a string from the JSON structure and failed.
So, instead of:
[HttpPost("[action]")]
public object Search([FromBody] string data)
I had to use the more specific:
[HttpPost("[action]")]
public object Search([FromBody] JObject data)

This issue is related to Byte Order Mark in the JSON file. JSON file is not encoded as UTF8 encoding data when saved. Using File.ReadAllText(pathFile) fix this issue.
When we are operating on Byte data and converting that to string and then passing to JsonConvert.DeserializeObject, we can use UTF32 encoding to get the string.
byte[] docBytes = File.ReadAllBytes(filePath);
string jsonString = Encoding.UTF32.GetString(docBytes);

I had the same problem with webapi in ASP.NET core, in my case it was because my application needs authentication, then it assigns the annotation [AllowAnonymous] and it worked.
[AllowAnonymous]
public async Task <IList <IServic >> GetServices () {
        
}

I ran into this issue and it ended up being because of BOM characters in my input string.
Here's what I ended up doing:
String.Trim(new char[] { '\uFEFF', '\u200B' });
This resolved the issue for me.

In my case, I was getting an error on JsonConvert.PopulateObject().
My request was returning JSON that was wrapped in an extra pair of '[ ]' brackets, making my result an array of one object rather than just an object. Here's what I did to get inside these brackets (only for that type of model):
T jsonResponse = new T();
var settings = new JsonSerializerSettings
{
DateParseHandling = DateParseHandling.DateTimeOffset,
NullValueHandling = NullValueHandling.Ignore,
};
var jRslt = response.Content.ReadAsStringAsync().Result;
if (jsonResponse.GetType() == typeof(myProject.Models.MyModel))
{
var dobj = JsonConvert.DeserializeObject<MyModel[]>(jRslt);
var y = dobj.First();
var szObj = JsonConvert.SerializeObject(y);
JsonConvert.PopulateObject(szObj, jsonResponse, settings);
}
else
{
JsonConvert.PopulateObject(jRslt, jsonResponse);
}

If you are using downloading data using url...may need to use
var result = client.DownloadData(url);

In my scenario I had a slightly different message, where the line and position were not zero.
E. Path 'job[0].name', line 1, position 12.
This was the top Google answer for the message I quoted.
This came about because I had called a program from the Windows command line, passing JSON as a parameter.
When I reviewed the args in my program, all the double quotes got stripped.
You have to reconstitute them.
I posted a solution here. Though it could probably be enhanced with a Regex.

I had a similar error and thought I'd answer in case anyone was having something similar. I was looping over a directory of json files and deserializing them but was getting this same error.
The problem was that it was trying to grab hidden files as well. Make sure the file you're passing in is a .json file. I'm guessing it'll handle text as well. Hope this helps.

I had simular problem. In my case the problem was in DateTime format. It was just numbers and it is also know as EpochFormat or UnixTimestamp.
A part from my JSON:
"direction": "outbound",
"date_archive": 1554691800224,
"date_doc": 1524700800000,
"date_sent": 1524704189000,
"date_received": 1524704189000,
"date_store_till": 1712544600224,
So I've used an attribute like this:
[JsonProperty("date_received")]
[JsonConverter(typeof(MicrosecondEpochConverter))]
public DateTime? DateReceived { get; set; }
You can find MicrosecondEpochConverter code here: https://stackoverflow.com/a/19972214/4324624

I faced similar error message in Xamarin forms when sending request to webApi to get a Token,
Make sure all keys (key : value) (ex.'username', 'password', 'grant_type') in the Json file are exactly what the webApi expecting, otherwise it fires this exception.
Unhandled Exception: Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0

Please check the model you shared between client and server is same. sometimes you get this error when you not updated the Api version and it returns a updated model, but you still have an old one. Sometimes you get what you serialize/deserialize is not a valid JSON.

In my case, it was the lack of a default parameterless constructor !!!

In my case, I was calling the async service method without using await, so before Task is completed I was trying to return the result!

Suppose this is your json
{
"date":"11/05/2016",
"venue": "{\"ID\":12,\"CITY\":Delhi}"
}
if you again want deserialize venue, modify json as below
{
"date":"11/05/2016",
"venue": "{\"ID\":\"12\",\"CITY\":\"Delhi\"}"
}
then try to deserialize to respective class by taking the value of venue

This error occurs when we parse json content to model object. Json content type is string.
For example:
https://dotnetfiddle.net/uFClKj
Some times, an api that we call may return an error. If we do not check the response status, but proceed to parse the response to model, this issue will occur.

When I encountered a similar problem, I fixed it by substituting &mode=xml for &mode=json in the request.

Any faster solution for Powershell.Invoke?

I'm writing a CmdLet in C# and use the Powershell-Class and invoke different commands. Here my Code:
PowerShell poswershell;
public Construcor()
{
powershell = PowerShell.Create()
}
public Collection<PSObject> InvokeRestMethod(string uri)
{
return InvokePSCommand("Invoke-RestMethod", new[] { uri });
}
public Collection<PSObject> InvokePSCommand(string command, string[] args)
{
var execute = command;
execute = GetExecuteCommand(args, execute);
powershell.Commands.AddScript(execute);
return powershell.Invoke();
}
private static string GetExecuteCommand(string[] args, string execute)
{
if (args == null) return execute;
for (var i = 0; i < args.Count(); i++)
{
execute += " " + args[i];
}
return execute;
}
It works like I want, but really slowly. I want the same function, which gives me a Collection<PSObject> back. But when I call the InvokeRestMehtod several times, it takes a long time to get trough this.
Why I use this just for a simple WebRequest you ask?
The answer is, that I have to read from an uri (which returns json) some information. Fact is that the json structure is always different. Therefor the Invoke-RestMehtod gives me exactly what I need, a dynamic object(PSObject). I have to have this kind of object, because after that process I need to give this in the powershell user back, so he can pipeline the object and use it furhter.
My question is now, how could I get the same result, which I can pass into powershell, from an uri which return json?
EDIT
I found this dll=> Microsoft.PowerShell.Commands.Utility.dll, it includes in C# code the InvokeRestMethod-CmdLet what is called by the powershell. If I could read this code well, I could (would) use it in my code. Then I wouldn't create a PowerShell instance and invoke from my powershell-CmdLet an other CmdLet, which I do not like much and takes a way too long. Someone knows this dll and could help me to customize this code for my project?
I found dottrace and analyzed the process, here a screenshot I can't get any useful information out of that, perhaps someone of you? But I'm pretty sure Powershell.Invoke() takes most of the time while executing.

Why don't you reuse the same PowerShell object rather than create a new one each time? Each instantiation causes the PowerShell to have to initialize again. Just be sure to call powershell.Commands.Clear() between Invoke() calls.

I solved the problem now, here is the basic code which reads an uri (returns JSON) and returns an object:
private object InvokeRest(string uri)
{
return InvokeRest(uri, null);
}
private object InvokeRest(string uri, NetworkCredential networkCredential)
{
var webRequest = WebRequest.Create(uri);
if (networkCredential!=null)
webRequest.Credentials = networkCredential;
_webResponse = webRequest.GetResponse();
var streamReader = new StreamReader(_webResponse.GetResponseStream());
var str = streamReader.ReadToEnd();
ErrorRecord exRef;
var doc = JsonObject.ConvertFromJson(str, out exRef);
return doc;
}
Because I found out that every object you output in the powershell, converts in a PSObject and formats it.

Call function who's name is stored in a variable

I have created a small classic asp file that I indend to call form asp.net using the below. I works, but it feels wrong somehow:
Booking.BelongsToSite = file_get_contents("http://localhost:82/test2.asp?functionName=RetBTS&param=" + User.ID);
protected string file_get_contents(string fileName)
{
string sContents = string.Empty;
if (fileName.ToLower().IndexOf("http:") > -1)
{ // URL
System.Net.WebClient wc = new System.Net.WebClient();
byte[] response = wc.DownloadData(fileName);
sContents = System.Text.Encoding.ASCII.GetString(response);
}
else
{
// Regular Filename
System.IO.StreamReader sr = new System.IO.StreamReader(fileName);
sContents = sr.ReadToEnd();
sr.Close();
}
return sContents;
}
Can anyone see any problems with doing this?
Also is it possible to do something like the below in Classic ASP / VB script. I can't get it to call a dynamic function name:
dim functionName, param, result
functionName = request("functionName")
param = request("param")
result = functionName(param)
Also any idea how to parse the parameters. Say if I pass the parameters in the head like "1,2,3,4", how can I pass the into the parenthesis?

I think you're looking for the Eval function in VBScript. You can use it to call your method in the following way (demo script is vbs in wsh):
Dim func, param
func = "Hello"
param = "everybody"
MsgBox(Eval(func & "(""" & param & """)"))
Function Hello(name)
Hello = "Hello " & name
End Function
That returns "Hello everybody", as expected.
As for your asp.net code: I'm not going to judge whether or not your solution is sensible, I don't know the situation. It can definitely work though. Just two remarks:
Both WebClient and StreamReader implement IDisposable. Wrap them in a using block.
If you're going to download urls like that, make sure to validate your string, so that you don't go and download bad urls. Same for the function name in your classic asp that you pass in your querystring.
Menno

Try to use GetRef instead of Eval.
dim functionName, param, result
functionName = request("functionName")
param = request("param")
result = GetRef(functionName)(param)

Develop Reference

C# (C-Sharp) is a programming language developed by Microsoft that runs on the .NET Framework.

Pass JSON to C# python process - c#

You can use messagePack library. I managed to pass strings and other types between python and c# with high performance.

Related

How to serialize tensor input required by dnnclassifier (serving_input_reciever)

HTTP POST read response as regular string

Unexpected character encountered while parsing value

Any faster solution for Powershell.Invoke?

Call function who's name is stored in a variable

Categories

Resources