RESOLVED: I stopped treating the string like an array and instead added a line-feed delimiter to the connection manager, so that each row was treated like a new line. This has resolved the issue and eliminated the need to handle this via a C# script.
I've built an SSIS package that uses a C# script task to deserialize JSON strings and insert them into a table.
I've got this working with a prototype JSON string:
{"name":"Test 1","code":"398057008","table":"SNOMEDCT","addedby":"morgan.baxter","dateadded":1544523489235,"qualifier":[{"name":"Qualifier","value":"Confirmed Diagnosis","code":"410605003","prefix":"[C] "}],"prefix":"[C] "}
But when I try to add a second item to the JSON string I receive an error:
[{"name":"Test 2","code":"398057008","table":"SNOMEDCT","addedby":"morgan.baxter","dateadded":1544523489235,"qualifier":[{"name":"Qualifier","value":"Confirmed Diagnosis","code":"410605003","prefix":"[C] "}],"prefix":"[C] "},{"name":"Test 2","code":"44255352","table":"SNOMEDCT","addedby":"morgan.baxter","dateadded":1544523489235,"qualifier":[{"name":"Qualifier","value":"Confirmed Diagnosis","code":"53252355","prefix":"[C] "}],"prefix":"[C] "}]
Error:
Type 'SC_8aae662509ae4bab8491323924579173.Diagnosis' is not supported
for deserialization of an array.
From what I can understand, this is simply saying that my 'Parser' essentially cannot support an array of JSON fields in a string.
Here is my main code block:
#region Namespaces
using System;
using System.Data;
using System.Collections.Generic;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using System.Web.Script.Serialization;
using Microsoft.SqlServer.Dts.Pipeline;
namespace SC_8aae662509ae4bab8491323924579173
#endregion
{
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
///<param name="Row">The row that is currently passing through the component</param>;
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
JavaScriptSerializer js = new JavaScriptSerializer();
// Give the input column a variable to make it easier to reference.
BlobColumn combinedColumn = Row.Column0;
// Convert from blob to string
string reviewConverted = System.Text.Encoding.ASCII.GetString(combinedColumn.GetBlobData(0, Convert.ToInt32(combinedColumn.Length)));
Diagnosis diagnosis = new Diagnosis();
// Deserialize the string
diagnosis = js.Deserialize<Diagnosis>(reviewConverted);
// Assign values to output columns
Row.name = diagnosis.name;
Row.code = diagnosis.code;
Row.table = diagnosis.table;
Row.addedby = diagnosis.addedby;
Row.dateadded = diagnosis.dateadded;
Row.qualifierName = diagnosis.Qualifier[0].name;
Row.qualifierValue = diagnosis.Qualifier[0].value;
Row.qualifierCode = diagnosis.Qualifier[0].code;
Row.qualifierPrefix = diagnosis.Qualifier[0].prefix;
if (diagnosis.Qualifier.Length == 2)
{
Row.lrName = diagnosis.Qualifier[1].name;
Row.lrValue = diagnosis.Qualifier[1].value;
Row.lrCode = diagnosis.Qualifier[1].code;
Row.lrSuffix = diagnosis.Qualifier[1].prefix;
}
Row.jsonString = reviewConverted;
Row.prefix = diagnosis.prefix;
Row.suffix = diagnosis.suffix;
}
}
}
Qualifier class:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace SC_8aae662509ae4bab8491323924579173
{
class qualifier
{
public string name { get; set; }
public string value { get; set; }
public string code { get; set; }
public string prefix { get; set; }
}
}
Diagnosis class:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace SC_8aae662509ae4bab8491323924579173
{
class Diagnosis
{
public string name { get; set; }
public string code { get; set; }
public string table { get; set; }
public string addedby { get; set; }
public string dateadded { get; set; }
public qualifier[] Qualifier { get; set; }
public string prefix { get; set; }
public string suffix { get; set; }
public string jsonString { get; set; }
}
}
I've tried to cater for more than one diagnosis with an array and my if function, but to no avail. Any ideas? Many thanks.
You say, that you pass array instead of an object in json. So why do you keep trying to deserialize into Diagnosis object?
You should really use Collection with your type Diagnosis here:
// Deserialize the string
var diagnosisCollection = js.Deserialize<ICollection<Diagnosis>>(reviewConverted);
instead of old
// Deserialize the string
diagnosis = js.Deserialize<Diagnosis>(reviewConverted);
And then assign values for you output columns, using foreach
foreach (var diagnosis in diagnosisCollection ) {
// do stuff with your buffer here
}
SSIS script can be a wierd environment. And you're right to avoid using a 3rd party JSON parser here. But somewhere you've got the type definitions messed up.
I often find it useful to write, test and troubleshoot code in a console app before integrating it into an SSIS script component/task. EG
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace ConsoleApp14
{
class Program
{
class qualifier
{
public string name { get; set; }
public string value { get; set; }
public string code { get; set; }
public string prefix { get; set; }
}
class Diagnosis
{
public string name { get; set; }
public string code { get; set; }
public string table { get; set; }
public string addedby { get; set; }
public string dateadded { get; set; }
public qualifier[] Qualifier { get; set; }
public string prefix { get; set; }
public string suffix { get; set; }
public string jsonString { get; set; }
}
static void Main(string[] args)
{
var json = #"
[
{
""name"": ""Test 2"",
""code"": ""398057008"",
""table"": ""SNOMEDCT"",
""addedby"": ""morgan.baxter"",
""dateadded"": 1544523489235,
""qualifier"": [
{
""name"": ""Qualifier"",
""value"": ""Confirmed Diagnosis"",
""code"": ""410605003"",
""prefix"": ""[C] ""
}
],
""prefix"": ""[C] ""
},
{
""name"": ""Test 2"",
""code"": ""44255352"",
""table"": ""SNOMEDCT"",
""addedby"": ""morgan.baxter"",
""dateadded"": 1544523489235,
""qualifier"": [
{
""name"": ""Qualifier"",
""value"": ""Confirmed Diagnosis"",
""code"": ""53252355"",
""prefix"": ""[C] ""
}
],
""prefix"": ""[C] ""
}
]
";
System.Web.Script.Serialization.JavaScriptSerializer js = new System.Web.Script.Serialization.JavaScriptSerializer();
// Deserialize the string
var diagnoses = js.Deserialize<Diagnosis[]>(json);
Console.WriteLine("Complete");
Console.ReadKey();
}
}
}
Related
I'm trying to deserialize this json with System.Text.Json, but I can't, and I don't know why it's not working, here's the result:
PS: And I don't know if that influences, but the array where the elements are inside, has no name, just enter the link and see
PS/2: This happens on any object in the list
Here is my code:
using System.Text.Json.Serialization;
namespace AceOfSpadesServersList
{
internal sealed class ServerList
{
[JsonPropertyName("name")]
public string Name { get; internal set; }
[JsonPropertyName("identifier")]
public string IP { get; internal set; }
[JsonPropertyName("map")]
public string Map { get; internal set; }
[JsonPropertyName("game_mode")]
public string GameMode { get; internal set; }
[JsonPropertyName("country")]
public string Country { get; internal set; }
[JsonPropertyName("latency")]
public ushort Latency { get; internal set; }
[JsonPropertyName("players_current")]
public byte CurrentPlayers { get; internal set; }
[JsonPropertyName("players_max")]
public byte MaxPlayers { get; internal set; }
[JsonPropertyName("last_updated")]
public uint LastUpdated { get; internal set; }
[JsonPropertyName("game_version")]
public string GameVersion { get; internal set; }
}
}
using System.IO;
using System.Net.Http;
using System.Text;
using System.Text.Json;
using System.Threading.Tasks;
using AceOfSpadesServersList;
namespace AceOfSpadesServersList
{
public class AceOfSpadesServersList
{
private const string BuildAndShootServersList = "http://services.buildandshoot.com/serverlist.json";
private readonly HttpClient _httpClient;
public AceOfSpadesServersList()
{
if (_httpClient is null)
this._httpClient = new HttpClient();
}
public async Task GetAllServersAsync()
{
var json = string.Empty;
var streamHttpResponse = await this._httpClient.GetStreamAsync(BuildAndShootServersList);
using (var sr = new StreamReader(streamHttpResponse, Encoding.UTF8))
json = await sr.ReadToEndAsync();
var serverList = JsonSerializer.Deserialize<ServerList[]>(json);
}
}
}
I don't know what's wrong, but I tested the exact same code using Newtonsoft.Json, and I just changed the JsonPropertyName attribute to JsonProperty and JsonSerializer.Deserialize<ServerList[]>(json); to JsonConvert.DeserializeObject<ServerList[]>(json); and works normally, it just doesn't work in the standard C# library
System.Text.Json respects the visibility constraints you impose on the class. Remove the internal on your setters.
i have the same problem and fix it by add PropertyNamingPolicy
var serverList = JsonSerializer.Deserialize<ServerList[]>(json, new
JsonSerializerOptions
{
PropertyNamingPolicy = JsonNamingPolicy.CamelCase
});
I am trying to parse a JSON using NewtonSoft (my JSON is an array type). Below is what it looks like...
{
"steps": [
{
"stepsType": "runWizard",
"wizardType": "demoServer",
"config": {
"mode": "add",
"resourceName": "demo server 1",
"activeDbPrimaryServer": {
"serverName": "abc",
"serverAddress": "abc.demo.local"
},
"activeDbCatalog": "demoActiveDB",
"activeDBUserId": "sa",
"activeDBPassword": "xyz",
}
}
]
}
I have created a class using JsonToC# convertor....
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace ServerSetupWizardConsoleApp
{
// Root myDeserializedClass = JsonConvert.DeserializeObject<Root>(myJsonResponse);
public class ActiveDbPrimaryServer
{
public string serverName { get; set; }
public string serverAddress { get; set; }
}
public class Config
{
public string mode { get; set; }
public string resourceName { get; set; }
public ActiveDbPrimaryServer activeDbPrimaryServer { get; set; }
public string activeDbCatalog { get; set; }
public string activeDBUserId { get; set; }
public string activeDBPassword { get; set; }
}
public class Step
{
public string stepsType { get; set; }
public string wizardType { get; set; }
public Config config { get; set; }
}
public class Root
{
public List<Step> steps { get; set; }
}
}
And now when I am trying to deserialise it in another class, I am getting a empty response in console...
I have a TEXT from where I am reading the JSON and then storing it in a string type variable and the using the string to deserialise it.
public Object ReadJson()
{
string jsonText = File.ReadAllText("C:\\Desktop\\demo.json");
var rootObject = (Root) JsonConvert.DeserializeObject(jsonText, typeof(Root));
var activeDbAttr = (ActiveDbPrimaryServer)JsonConvert.DeserializeObject(jsonText, typeof(ActiveDbPrimaryServer));
Console.WriteLine("Value : " + activeDbAttr.serverAddress);
}
This activeDbAttr.serverAddress is giving me nothing in CONSOLE
It print --> value : (nothing after ":" like blank)
Can someone tell me what is wrong here. I followed some old answers, not getting to a point where I can fix it.
Live demo : https://dotnetfiddle.net/PuO8F8
It's a simple navigation issue.
You deserialize to a Root object, and instead of navigating with the property you deserialize again into an other thing, hopping to land in the right place.
var rootObject = JsonConvert.DeserializeObject<Root>(jsonText);
var activesDBs = json.steps.Select( x=> x.config.activeDbPrimaryServer).ToList();
Result:
Dumping object(System.Linq.SelectListIterator`2[Step,ActiveDbPrimaryServer])
[
{
serverAddress : abc.demo.local
serverName : abc
}
]
Why did it failed ?
var jsonDB = JsonConvert.DeserializeObject<ActiveDbPrimaryServer>(GetJson());
Tell the Parser that the Json is of type ActiveDbPrimaryServer.
The parser open the Json and find the first object:
{
"steps": [..]
}
Look for the property of the expected type ActiveDbPrimaryServer
public class ActiveDbPrimaryServer
{
public string serverName { get; set; }
public string serverAddress { get; set; }
}
And find nothing. It end there and give you an object of the right type with no property initialized
Partial deserialization:
If you want to deserialize only the part you need, refer to this documentation from Newtonsoft:
Deserializing Partial JSON Fragments
JObject rootObj = JObject.Parse(GetJson());
// get JSON result objects into a list
IList<JToken> results = rootObj["steps"].Children() // Step is a list so we use `.Children()`
["config"]["activeDbPrimaryServer"].ToList();
IList<ActiveDbPrimaryServer> dbResults = new List<ActiveDbPrimaryServer>();
foreach (JToken result in results)
{
// JToken.ToObject is a helper method that uses JsonSerializer internally
ActiveDbPrimaryServer dbResult = result.ToObject<ActiveDbPrimaryServer>();
dbResults.Add(dbResult);
}
you are deserializing
var rootObject = (Root) JsonConvert.DeserializeObject(jsonText, typeof(Root));
var activeDbAttr = (ActiveDbPrimaryServer)JsonConvert.DeserializeObject(jsonText, typeof(ActiveDbPrimaryServer));
[the second one yields null ... and null cast to ActiveDbPrimaryServer yields null.]
the same string twice into 2 different types of objects that do not share common base class (of course other than object).
your text represents either a Root or a ActiveDbPrimaryServer not both. the ActiveDbPrimaryServer is a member of the config class and that in turn of Step. ....
step through your graph root.Step.Config .. if all the instances are set you should reach your instance of ActibeDdPrimaryServer
I am attempting to return the Json response deserialized from the following https://www.supremenewyork.com/mobile_stock.json
Below is my code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Text;
using System.Drawing;
using System.IO;
using System.Net.Http;
using Newtonsoft.Json;
using Newtonsoft.Json.Converters;
using RestSharp;
using System.Threading.Tasks;
using System.Globalization;
namespace SupremeMobileMonitor
{
class Program
{
static async Task Main(string[] args)
{
var program = new Program();
await program.GetItem();
}
// Declaring variables in the list
static List<ItemDetail> ProductList = new List<ItemDetail>();
List<string> productDesc = new List<string> { "new_item", "price", "category", "imageurl", "itemURL" };
List<string> category = new List<string> { "jackets", "shirts", "tops_sweaters", "pants", "hats", "accessories", "shoes", "skate" };
//creating a class for intializing Json Deserializer
public class MobileStockResponse
{
[JsonProperty("unique_image_url_prefixes")]
public List<object> UniqueImageUrlPrefixes { get; set; }
[JsonProperty("products_and_categories")]
public Dictionary<string, List<ProductsAndCategory>> ProductsAndCategories { get; set; }
[JsonProperty("release_date")]
public string ReleaseDate { get; set; }
[JsonProperty("release_week")]
public string ReleaseWeek { get; set; }
}
public partial class ProductsAndCategory
{
[JsonProperty("name")]
public string Name { get; set; }
[JsonProperty("id")]
public long Id { get; set; }
[JsonProperty("image_url")]
public string ImageUrl { get; set; }
[JsonProperty("image_url_hi")]
public string ImageUrlHi { get; set; }
[JsonProperty("price")]
public long Price { get; set; }
[JsonProperty("sale_price")]
public long SalePrice { get; set; }
[JsonProperty("new_item")]
public bool NewItem { get; set; }
[JsonProperty("position")]
public long Position { get; set; }
[JsonProperty("category_name")]
public string CategoryName { get; set; }
}
//Initializing HttpClient for Requests and po
public async Task GetItem()
{
var client = new HttpClient();
var request = new HttpRequestMessage
{
RequestUri = new Uri("https://www.supremenewyork.com/mobile_stock.json"),
Method = HttpMethod.Get
};
var response = await client.SendAsync(request);
var responseContent = await response.Content.ReadAsStringAsync();
var responseObject = JsonConvert.DeserializeObject<ProductsAndCategory>(responseContent);
Console.WriteLine(responseObject);
}
}
}
When I try to return a value (example: responseObject.id) It will just return '0'
When I try to return the complete response, it returns "SupremeMobileMonitor.Program+ProductsAndCategory"
Any idea why I can't get it returned and what I'm messing up my deserialization? Unfortunately the category known as "new" on the endpoint interferes with the C# keywords therefore removing my option to use dynamic deserialization.
jslowik has the correct answer for your question, how to deserialize it.
To Print the object, you can either create an override ToString() method and print out only the things you are interested in, or you can print the object by serializing it to a string.
Console.WriteLine(JsonConvert.SerializeObject(productsAndCategories, Formatting.Indented);
This will give you a json representation again of the object and show you all the values.
SideNote: If you want to get specific items from, Say, Bags, you can use Linq to get it...
Console.WriteLine(JsonConvert.SerializeObject(responseObject.ProductsAndCategories["Bags"].Select(x => x.Id)));
// Output:
[173389,172978,173019,172974,173018,173001]
Off hand I would just deserialize the full object. Example:
// Full model represented by your class
var responseObject = JsonConvert.DeserializeObject<MobileStockResponse>(responseContent);
// The products and categories dictionaries you're looking for
var productsAndCategories = responseObject.ProductsAndCategories;
Sorry for the somewhat basic question, but what can I say. I can't figure it out. The problem is that there's a foreach loop that's supposed to iterate through the rows (sections) and while it works for the first section, the second time through the loop it doesn't seem to read the second section. The same data is stored in version. BTW, the way the method is called I would be passing in ProductName as a parameter (There will be multiple products represented here and also a version number (e.g. v2.0.0) that I'll need to filter the results for too.
So I have an XML file that looks like this:
<Products>
<ProductName1>
<v2.0.0>
<GUID>"{B5ECEC43-5406-4E4D-96D9-456823100313}"</GUID>
<VersionNameToUninstall>"2.0.0 - 2.0.2"</VersionNameToUninstall>
<UninstallResponseFile>"GVQC-Client-2.0.0-Uninst.iss"</UninstallResponseFile>
</v2.0.0>
<v2.0.3>
<GUID>"{1D6C02D7-8E87-43BE-8AB2-1FF0E5ACD410}"</GUID>
<VersionNameToUninstall>"2.0.3"</VersionNameToUninstall>
<UninstallResponseFile>"GVQC-Client-2.0.3-Uninst.iss"</UninstallResponseFile>
</v2.0.3>
</ProductName1>
<ProductName2>
<v3.0.0>
<GUID>"{ABCDEC43-5406-4E4D-96D9-456823101234}"</GUID>
<VersionNameToUninstall>"2.2.0 - 2.2.2"</VersionNameToUninstall>
<UninstallResponseFile>"GVQC-Client-2.2.0-Uninst.iss"</UninstallResponseFile>
</v3.0.0>
<v4.0.0>
<GUID>"{5D6C02D7-8E87-43BE-8AB2-1FF0E5ACD589}"</GUID>
<VersionNameToUninstall>"4.0.0"</VersionNameToUninstall>
<UninstallResponseFile>"GVQC-Client-4.0.0-Uninst.iss"</UninstallResponseFile>
</v4.0.0>
</ProductName2>
</Products>
There will only be 10 or so versions (e.g. v2.x.x) so there's not a lot of data here. So I created a multidimensional (nested) class/struct to hold the data and when I try my code to read the data it's not working.
Here are the classes/stucts (I've tried both and neither works) that I'm trying to populate:
public class TopLevelObject
{
public string Version { get; set; }
public RowLevelObject Row {get;set;}
}
public struct RowLevelObject
{
public string Guid { get; set; }
public string VersionName { get; set; }
public string UninstallFileName { get; set; }
}
So here's my code. Please just ignore the Stream - that's so I can embed this XML file in the .exe and not have it be a separate file:
public static List<TopLevelObject> GetGUIDSFromFile(string GUIDKey)
List<InstallScriptMSIXMLTopLevelObject> installScriptMSIXMLTopLevelObjectList = new List<InstallScriptMSIXMLTopLevelObject>();
Stream GUIDXmlFileStream = typeof(PGCommonCA).Assembly.GetManifestResourceStream("PGCommonCA.ProductGUIDs.xml");
XElement xElement = XElement.Load(GUIDXmlFileStream);
var versions = xElement.Elements(GUIDKey).Descendants();
foreach (var version in versions)
{
TopLevelObject topLevelObject = new TopLevelObject();
RowLevelObject rowLevelObject = new RowLevelObject();
TopLevelObject.Version = version.Name.LocalName;
RowLevelObject.Guid = version.Element("GUID").Value;
RowLevelObject.VersionName = version.Element("VersionNameToUninstall").Value;
RowLevelObject.UninstallFileName = version.Element("UninstallResponseFile").Value;
TopLevelObjectList.Add(topLevelObject);
}
return TopLevelObjectList;
}
I know there are many ways to read XML and my choice doesn't work so I'm looking for another simple solution.
The following works :
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml;
using System.Xml.Linq;
using System.IO;
namespace ConsoleApplication1
{
class Program
{
const string FILENAME = #"c:\temp\test.xml";
static void Main(string[] args)
{
XDocument doc = XDocument.Load(FILENAME);
XElement productName = doc.Root;
List<TopLevelObject> top = productName.Elements().Select(x => new TopLevelObject() {
Version = x.Name.LocalName,
Row = new RowLevelObject() {
Guid = (string)x.Element("GUID"),
VersionName = (string)x.Element("VersionNameToUninstall"),
UninstallFileName = (string)x.Element("UninstallResponseFile")
}
}).ToList();
}
}
public class TopLevelObject
{
public string Version { get; set; }
public RowLevelObject Row { get; set; }
}
public struct RowLevelObject
{
public string Guid { get; set; }
public string VersionName { get; set; }
public string UninstallFileName { get; set; }
}
}
I figured it out (many thanks to jdweng!!). Here's the final solution based on the revised XML at the top:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml;
using System.Xml.Linq;
using System.IO;
namespace ConsoleApplication1
{
class Program
{
const string FILENAME = #"c:\temp\test.xml";
static TopLevelObject GetInfo(string xmlKey)
{
XDocument doc = XDocument.Load(FILENAME);
XElement productName = doc.Root;
List<TopLevelObject> top = productName.Descendants(xmlKey).Elements().Select(x => new TopLevelObject() {
Version = x.Name.LocalName,
Row = new RowLevelObject() {
Guid = (string)x.Element("GUID"),
VersionName = (string)x.Element("VersionNameToUninstall"),
UninstallFileName = (string)x.Element("UninstallResponseFile")
}
}).ToList();
}
}
public class TopLevelObject
{
public string Version { get; set; }
public RowLevelObject Row { get; set; }
}
public struct RowLevelObject
{
public string Guid { get; set; }
public string VersionName { get; set; }
public string UninstallFileName { get; set; }
}
}
My json string is as follows:
{
"data": {
"order_reference": "7000016543",
"package_data": [
{
"tracking_no": "34144582723408",
"package_reference": "7000016543"
}
]
},
"success": true
}
How do I get tracking_no from this json.
I tried using
dynamic jsonObj = JsonConvert.DeserializeObject(bytesAsString);
and then
foreach (var obj in jsonObj.items)
{
}
but it only yields order reference.
You can create Types for your JSON:
namespace Example
{
using System;
using System.Net;
using System.Collections.Generic;
using Newtonsoft.Json;
public class MyObject
{
[JsonProperty("data")]
public Data Data { get; set; }
[JsonProperty("success")]
public bool Success { get; set; }
}
public class Data
{
[JsonProperty("order_reference")]
public string OrderReference { get; set; }
[JsonProperty("package_data")]
public PackageDatum[] PackageData { get; set; }
}
public class PackageDatum
{
[JsonProperty("package_reference")]
public string PackageReference { get; set; }
[JsonProperty("tracking_no")]
public string TrackingNo { get; set; }
}
}
then you can deserialize using JsonConvert.Deserialize<MyObject>(input);
the problem about your code is: there is no collection on your root json object. I mean it is not an object. Since it is just an object and it's child you can access is directly. And then you have a collection package_data. So you can go for loop in it.
Here is the schema of your json data.
foreach(dynamic item in jsonObj.item.package_data)
{
//item.tracking_no;
//item.package_reference
}