Develop Reference

Pass JSON to C# python process

I have a little problem. I want to start a python process from C# and I need to pass some data to it. The data is in json format, but if I serialize the data from c# it look like this
"[{\"SearchTerm_id\":1,\"Term\":\"lorem ipsum\"},{\"SearchTerm_id\":2,\"Term\":\"lorem ipsum\"}]}"
and is not valid for python because of the scapation for double quote.
How I can pass the data from c# to a python script ?
This is my code:
List<SearchTerms> searchTerms = await _context.SearchTerms.ToListAsync();
var json = JsonConvert.SerializeObject(searchTerms);
ProcessStartInfo processInfo = new ProcessStartInfo();
string scriptPath = Path.Combine(System.IO.Directory.GetCurrentDirectory(), "Python\\scrapeGoogle.py");
processInfo.FileName = "python3";
processInfo.Arguments = string.Format("{0} {1}", scriptPath, json);
processInfo.UseShellExecute = false;
processInfo.CreateNoWindow = true;
processInfo.RedirectStandardOutput = true;
processInfo.RedirectStandardError = true;
process.StartInfo = processInfo;
process.Start();

If you are viewing the result of var json = JsonConvert.SerializeObject(searchTerms); in the debugger. The debugger only shows the \ as a visual aid, representing the string just as it would have to be written in c#.
Try running Console.Write(json); and view the output.
The output won't contain the escape character. This is the true value.

Just solved this for myself, so thought I would comment.
The Problem
#Filip Laurentiu (OP) correctly identified his problem when replying to #A Redfearn's answer.
You are right. So the problem seams to be elsewhere because if I have
only a print(sys.argv1) in my python, I will recive my data as
follow : "[{SearchTerm_id:1,Term:Lorem ipsun},{SearchTerm_id:2,Lorem
Ipsum}]"
C# is actually formatting the JSON without escape characters, contrary to what the original question said. So it is doing what it is supposed to do. If you print the text, you see {"SearchTerm_id":1,"Term":"lorem ipsum"}.
However, Python is receiving the JSON without the double quotes. Printing from Python shows {SearchTerm_id:1,Term:lorem ipsum}. And when you call json.loads with that bad JSON, it throws.
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
It seems that when you call Process.Start() in C#, the shell strips the double quotes from the JSON in your ProcessStartInfo.Arguments list. So Python receives the argument without double quotes.
The Solution
In C#, update the JSON string after serializing, so that you escape the double quotes. Here is some code to help with that.
using Newtonsoft.Json;
using System.Text;
public static class JsonHelper
{
public static string ToJsonString(
object obj,
bool escapeDoubleQuotes = false)
{
string serialized = JsonConvert.SerializeObject(obj);
if (escapeDoubleQuotes)
{
// e.g., '{"key":"value"}' -> '{\"key1\":\"value\"}'
// Do this when need to pass json as cmd line arg via System.Diagnostics.Process. Else shell will strip the double quotes, so your new process might not know what to do with it
serialized = serialized.Replace(#"""", #"\""");
}
return serialized;
}
}
So to get his original code working, OP would just change
var json = JsonConvert.SerializeObject(searchTerms);
to
var json = JsonHelper.ToJsonString(searchTerms, escapeDoubleQuotes: true);

You can use messagePack library. I managed to pass strings and other types between python and c# with high performance.

Postdata formatting

I have a quick question about the formatting of postData in c#. Below is what I believe to be the post data extracted from google chrome. The data is related to creating an email. If anyone has any advice or spots any errors that would be really helpful. Thanks in advance
The Post Data
/reg_json?GivenName=findmeeasy&FamilyName=notebook&AccountID=#yahoo.com&PartnerName=yahoo_default&ApiName=ValidateFields&RequestVersion=1&intl=us&u=r333.member.bf1.yahoo.com56a54898077dc5.76327249&t=omaWrw2j1XMYYqsuNAvhoksAu08PTvyAF.VLF5WV3CEDJjE1oXISM8YYtNCMH4jqCmeEiFNVFSEj9MUCaEI4XkBtmeyboO6R4H0SI2tgb1vIz0gsI_3fMFByGY9uDyBssJxV0IHFuNAH4vDMGF0aJrn9AOLwjvxvEjBnGAdd0a_O1K4EyNP6xBkP59d9htdyrvTfsWsjfxNOSEJ1Cz4I0ynjBX7jBtorMVaPeGNxPxOhkNhIeaRR1M7pwIqaN9JxS8Sz4dhVccpdLlwvct.a6PGX1fEWZnvPowfODogHPIdJ1wq3dcwa4r4uK7K.QMHdzJobys_bJwGdKBXjQx7VL6WA_0Xoe9wk3xXZ13hIaR0zaCnDvWINPyX4.40.cw59.W5So13znpBXX.B4ZiD89OByV7c_cc1gzmjPVSOOcBQVAFljk72gPqq5GNA_iWAMAgYSH0lMacvUQSOpc6XnVjshMJRqRwQncoYn7j83WmOximNzdAoOOH.C8yeF6v3uxJVhOS111OCd2N37WucRBTdMWDOSTzDCx53a7E9fCNn9p_4PUGKr07WPRj5.9svMxv.PrKg_bJdLnJI-~B&1453672621368 HTTP/1.1
My Attempt at formatting
StringBuilder postData = new StringBuilder();
postData.Append("/reg_json?GivenName=findmeeasy");
postData.Append("&FamilyName=notebook");
postData.Append("&AccountID=#yahoo.com");
postData.Append("&PartnerName=yahoo_default");
postData.Append("&ApiName=ValidateFields");
postData.Append("&RequestVersion=1");
postData.Append("&intl=us");
postData.Append("&u=r333.member.bf1.yahoo.com56a54898077dc5.76327249");
postData.Append("&t=omaWrw2j1XMYYqsuNAvhoksA.....");

You have to make sure that both the variable name and the data value are url escaped - which they look to be. But that may be a co-incidence.
I'd do something like this
using System;
private static string MakeData(string name, string value)
{
return Uri.Escape(name) + "=" + Uri.Escape(value);
}
...
postData.Append("/reg_json?" + MakeData("GivenName", "findmeeasy"));
postData.Append("&" + MakeData("FamilyName", "notebook"));
/// etc.

Input string error when posting for next set of data

The other day I had my scraper working very well, when I realized that I wasn't getting enough records, I decided to search by postal code and had to change some of my code. The first POST and RESPONSE goes off without a hitch, but when I post to get the next page of results it fails on the Response.
the starting code is...
public void StartScrape()
{
List<string> a = lstPostalCodes();
for (int i = 0; i <= lstPostalCodes().Count; i++)
{
b = a[i];
FirstRequestResponse(b);
if (GoBackToStartSearch == "CONTINUEON")
StartNextRequest(GetViewState(ResponseData));
else
{
WriteDataToFile(ResponseData);
FinalClean();
}
}
}
The method that I am calling is StartNextRequest. In the StartNextRequest method is
private void GetResults(HttpWebRequest wr)
{
using (StreamReader responseReader = new StreamReader(wr.GetResponse().GetResponseStream()))
{
// Add response/results to string
ResponseData = responseReader.ReadToEnd();
}
string strFind = "<li id='nextdisabled'>";
if (ResponseData.Contains(strFind)) GoBackToStartSearch = "BACKTOSTART";
else
GoBackToStartSearch = "CONTINUEON";
}
The error that its throwing is saying that the
Input string was not in correct format
and pointing at the using (StreamReader...)
I have gotten other errors that I seem to fix, but then it creates another error which has to do with waiting for bytes to be written. I'm using Fiddler2 to give me more info on the errors, the headers are pretty much the same, I'm not seeing much of a difference, other than 1 being a request and the other a response.
I have no idea on why its saying that, I have stepped through the working code that I wrote to scrape all the data results, and compared the two. Everything looks the same. The values are all the same.
Any ideas on where I should look to fix this?

Unexpected character encountered while parsing value

Currently, I have some issues. I'm using C# with Json.NET. The issue is that I always get:
{"Unexpected character encountered while parsing value: e. Path '', line 0, position 0."}
So the way I'm using Json.NET is the following. I have a Class which should be saved. The class looks like this:
public class stats
{
public string time { get; set; }
public string value { get; set; }
}
public class ViewerStatsFormat
{
public List<stats> viewerstats { get; set; }
public String version { get; set; }
public ViewerStatsFormat(bool chk)
{
this.viewerstats = new List<stats>();
}
}
One object of this class will be filled and saved with:
File.WriteAllText(tmpfile, JsonConvert.SerializeObject(current), Encoding.UTF8);
The saving part works fine and the file exists and is filled. After that the file will be read back into the class with:
try
{
ViewerStatsFormat current = JsonConvert.DeserializeObject<ViewerStatsFormat>(tmpfile);
//otherstuff
}
catch(Exception ex)
{
//error loging stuff
}
Now on the current= line comes the exception:
{"Unexpected character encountered while parsing value: e. Path '', line 0, position 0."}
I don't know why this comes. The JSON file is the following -> Click me I am the JSON link
Does anyone have any ideas?

Possibly you are not passing JSON to DeserializeObject.
It looks like from File.WriteAllText(tmpfile,... that type of tmpfile is string that contain path to a file. JsonConvert.DeserializeObject takes JSON value, not file path - so it fails trying to convert something like #"c:\temp\fooo" - which is clearly not JSON.

I solved the problem with these online tools:
To check if the Json structure is OKAY: http://jsonlint.com/
To generate my Object class from my Json structure: https://www.jsonutils.com/
The simple code:
RootObject rootObj= JsonConvert.DeserializeObject<RootObject>(File.ReadAllText(pathFile));

In my case, the file containing JSON string had BOM. Once I removed BOM the problem was solved.

I experienced the same error in my Xamarin.Android solution.
I verified that my JSON was correct, and noticed that the error only appeared when I ran the app as a Release build.
It turned out that the Linker was removing a library from Newtonsoft.JSON, causing the JSON to be parsed incorrectly.
I fixed the error by adding Newtonsoft.Json to the Ignore assemblies setting in the Android Build Configuration (screen shot below)
JSON Parsing Code
static readonly JsonSerializer _serializer = new JsonSerializer();
static readonly HttpClient _client = new HttpClient();
static async Task<T> GetDataObjectFromAPI<T>(string apiUrl)
{
using (var stream = await _client.GetStreamAsync(apiUrl).ConfigureAwait(false))
using (var reader = new StreamReader(stream))
using (var json = new JsonTextReader(reader))
{
if (json == null)
return default(T);
return _serializer.Deserialize<T>(json);
}
}
Visual Studio Mac Screenshot
Visual Studio Screenshot

I have also encountered this error for a Web API (.Net Core 3.0) action that was binding to a string instead to an object or a JObject. The JSON was correct, but the binder tried to get a string from the JSON structure and failed.
So, instead of:
[HttpPost("[action]")]
public object Search([FromBody] string data)
I had to use the more specific:
[HttpPost("[action]")]
public object Search([FromBody] JObject data)

This issue is related to Byte Order Mark in the JSON file. JSON file is not encoded as UTF8 encoding data when saved. Using File.ReadAllText(pathFile) fix this issue.
When we are operating on Byte data and converting that to string and then passing to JsonConvert.DeserializeObject, we can use UTF32 encoding to get the string.
byte[] docBytes = File.ReadAllBytes(filePath);
string jsonString = Encoding.UTF32.GetString(docBytes);

I had the same problem with webapi in ASP.NET core, in my case it was because my application needs authentication, then it assigns the annotation [AllowAnonymous] and it worked.
[AllowAnonymous]
public async Task <IList <IServic >> GetServices () {
        
}

I ran into this issue and it ended up being because of BOM characters in my input string.
Here's what I ended up doing:
String.Trim(new char[] { '\uFEFF', '\u200B' });
This resolved the issue for me.

In my case, I was getting an error on JsonConvert.PopulateObject().
My request was returning JSON that was wrapped in an extra pair of '[ ]' brackets, making my result an array of one object rather than just an object. Here's what I did to get inside these brackets (only for that type of model):
T jsonResponse = new T();
var settings = new JsonSerializerSettings
{
DateParseHandling = DateParseHandling.DateTimeOffset,
NullValueHandling = NullValueHandling.Ignore,
};
var jRslt = response.Content.ReadAsStringAsync().Result;
if (jsonResponse.GetType() == typeof(myProject.Models.MyModel))
{
var dobj = JsonConvert.DeserializeObject<MyModel[]>(jRslt);
var y = dobj.First();
var szObj = JsonConvert.SerializeObject(y);
JsonConvert.PopulateObject(szObj, jsonResponse, settings);
}
else
{
JsonConvert.PopulateObject(jRslt, jsonResponse);
}

If you are using downloading data using url...may need to use
var result = client.DownloadData(url);

In my scenario I had a slightly different message, where the line and position were not zero.
E. Path 'job[0].name', line 1, position 12.
This was the top Google answer for the message I quoted.
This came about because I had called a program from the Windows command line, passing JSON as a parameter.
When I reviewed the args in my program, all the double quotes got stripped.
You have to reconstitute them.
I posted a solution here. Though it could probably be enhanced with a Regex.

I had a similar error and thought I'd answer in case anyone was having something similar. I was looping over a directory of json files and deserializing them but was getting this same error.
The problem was that it was trying to grab hidden files as well. Make sure the file you're passing in is a .json file. I'm guessing it'll handle text as well. Hope this helps.

I had simular problem. In my case the problem was in DateTime format. It was just numbers and it is also know as EpochFormat or UnixTimestamp.
A part from my JSON:
"direction": "outbound",
"date_archive": 1554691800224,
"date_doc": 1524700800000,
"date_sent": 1524704189000,
"date_received": 1524704189000,
"date_store_till": 1712544600224,
So I've used an attribute like this:
[JsonProperty("date_received")]
[JsonConverter(typeof(MicrosecondEpochConverter))]
public DateTime? DateReceived { get; set; }
You can find MicrosecondEpochConverter code here: https://stackoverflow.com/a/19972214/4324624

I faced similar error message in Xamarin forms when sending request to webApi to get a Token,
Make sure all keys (key : value) (ex.'username', 'password', 'grant_type') in the Json file are exactly what the webApi expecting, otherwise it fires this exception.
Unhandled Exception: Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0

Please check the model you shared between client and server is same. sometimes you get this error when you not updated the Api version and it returns a updated model, but you still have an old one. Sometimes you get what you serialize/deserialize is not a valid JSON.

In my case, it was the lack of a default parameterless constructor !!!

In my case, I was calling the async service method without using await, so before Task is completed I was trying to return the result!

Suppose this is your json
{
"date":"11/05/2016",
"venue": "{\"ID\":12,\"CITY\":Delhi}"
}
if you again want deserialize venue, modify json as below
{
"date":"11/05/2016",
"venue": "{\"ID\":\"12\",\"CITY\":\"Delhi\"}"
}
then try to deserialize to respective class by taking the value of venue

This error occurs when we parse json content to model object. Json content type is string.
For example:
https://dotnetfiddle.net/uFClKj
Some times, an api that we call may return an error. If we do not check the response status, but proceed to parse the response to model, this issue will occur.

When I encountered a similar problem, I fixed it by substituting &mode=xml for &mode=json in the request.

Extracting a key-value pair from a web service XML response

I'm working with C# and the .NET 2.0 framework in Visual Studio 2010.
I'm trying to extract a URL which is returned by a web service.
This URL is returned in an array of features containing keys and values. (I think this is similar to what I learned in school is called a hash table).
My intellisense doesn't pick up anything useful and I can't figure out what I'm doing wrong.
This is the code. What goes in serverInfo.FeatureSet[]?
public string wfl_reqURL(string username, string password)
{
MyWorkflow.ServerInfo serverInfo = new MyWorkflow.ServerInfo();
myURL = serverInfo.FeatureSet[];
}
This is how it's described in the WSDL. FeatureSet is being returned as an array with a string key and a string value:
<ServerInfo>
<FeatureSet>
<Feature>
<Key>FileUploadUrl</Key>
<Value>http://localhost/transferindex.php</Value>
</Feature>
</FeatureSet>
</ServerInfo>
Have I provided enough detail about my problem? Most of the information I've found seems to be about how to create such arrays in web services, not select one from a web service as I'd like to do.

Try something like this:
object neededItem = null;
foreach (string item in serverInfo.FeatureSet.Keys)
{
if (item == "FileUploadUrl")
{
neededItem = serverInfo.FeatureSet[item];
break;
}
}
if (neededItem != null)
{
//Do something
}

If you're using c# 3.5 then something in linq like
myURL = serverInfo.FeatureSet.First(o=>o.Key == "FileUploadUrl").Value

The problem was in the data type. Changing the code to this solved the problem, albeit in a messy way. I thought it had something to do with types and how it was defined...either as dictionary or arrays, but it was a bit different than I'd thought...
foreach( MyWorkFlow.Feature feature in serverInfo.FeatureSet) {
if (feature.Key.ToString() == "FileUploadUrl") {
string myURL = feature.Value;
Console.WriteLine(myURL);
}