Reading from JSON API with C# in SSIS - c#

I want to read some data from a JSON API in SSIS and write it to a table in SQL Server. I've solved the task using a 3rd party, but the solution isn't that elegant, so now I'm trying to script it myself in Visual Studio using SSIS's script component.
I've researched around the web for solutions, and ended with this result. So far, I'm fairly confident about what is going on, but I lack the final direction for this. I know I need to somehow map the output to the columns I've created in SSIS.
I guess I have to do something around CreateNewOutputRows(), but I'm not sure what. Can someone please help me out on this? Also, since this is more or less my first ever c# script, I would also appreciate it, if there's a way easier solution OR if it is in some way inappropriate etc.
First of all, the output from the API looks like this (API documentation here):
"data":[
{
"created_at":"2016-03-12 09:45:00",
"created_at_unix":1457772300,
"shop_name":"DK - Some name",
"location_id":1111,
"custom_location_id":"2222",
"custom_shop_id":"2222",
"shop_id":3333,
"count_in":"1",
"count_out":"1",
"timekey":3
}
The script I've got so far is
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using System.Linq;
using System.Net;
using System.Collections.Generic;
using Newtonsoft.Json;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
public static void Main()
{
//Test api
var url = "https://login.vemcount.com/api/fetch_data/?data={"api_key":"xxxxxxx","type":"company_id","id":"10","date_from":"2019-01-01 00:00","interval":"15min", "group_by":"shop_id"}";
var json = new WebClient().DownloadString(url);
var root = JsonConvert.DeserializeObject<RootObject>(json);
//Printing last record
Console.WriteLine(root.data.Last().created_at);
}
public class data
{
public string created_at { get; set; }
public int created_at_unix { get; set; }
public string shop_name { get; set; }
public int location_id { get; set; }
public string custom_location_id { get; set; }
public string custom_shop_id { get; set; }
public int shop_id { get; set; }
public string count_in { get; set; }
public string count_out { get; set; }
public int timekey { get; set; }
}
public class RootObject
{
public List<data> data { get; set; }
}
public override void CreateNewOutputRows()
{
}
}

You'll put all that code in CreateNewOutputRows().
First you have to manually add all the columns to the script component. In the example here I only added 2 columns:
The code goes in CreateNewOutPutRows(). I don't have Newtonsoft, just using the JavaScriptSerializer here to show a working example so you can see how to hook it up:
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using System.Web.Script.Serialization;
using System.Collections.Generic;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
public override void CreateNewOutputRows()
{
string json = #"{""data"":[{""created_at"":""2016-03-12 09:45:00"",""created_at_unix"":1457772300,""shop_name"":""DK - Some name"",""location_id"":1111,""custom_location_id"":""2222"",""custom_shop_id"":""2222"",""shop_id"":3333,""count_in"":""1"",""count_out"":""1"",""timekey"":3},{""created_at"":""2016-03-12 09:45:00"",""created_at_unix"":1457772300,""shop_name"":""test2"",""location_id"":1111,""custom_location_id"":""2222"",""custom_shop_id"":""2222"",""shop_id"":3333,""count_in"":""1"",""count_out"":""1"",""timekey"":3}]}";
RootObject Test = new JavaScriptSerializer().Deserialize<RootObject>(json);
/*
* This is where data gets added to the output buffer.
* After AddRow() you are basically mapping the column you manually added(on the left) to the data(on the right).
* using a foreach loop to loop through the deserialize json
*/
foreach (var item in Test.data)
{
Output0Buffer.AddRow();
Output0Buffer.createdat = item.created_at;
Output0Buffer.shopname = item.shop_name;
}
}
public class RootObject
{
public List<data> data { get; set; }
}
public class data
{
public string created_at { get; set; }
public int created_at_unix { get; set; }
public string shop_name { get; set; }
public int location_id { get; set; }
public string custom_location_id { get; set; }
public string custom_shop_id { get; set; }
public int shop_id { get; set; }
public string count_in { get; set; }
public string count_out { get; set; }
public int timekey { get; set; }
}
}
Then in this example, just using a record set destination and enabled data viewer so you can see the individuals rows come back out:

Related

Class with Collection Properties are not binding properly when posted as XML to ASP.Net Core 3.1 Web API

I'm trying to post my data as XML to my asp.net core 3.1 web api. However Collection properties are not getting binded in my model.
Here is my class,
public class Test
{
public int Usrno { get; set; }
public string PCname { get; set; }
public List<Best> Best { get; set; }
}
public class Best
{
public string Hello { get; set; }
public Worst[] Worst { get; set; }
}
public class Worst
{
public int Ko { get; set; }
public Win[] Win { get; set; }
}
public class Win
{
public string Kiss { get; set; }
}
Here is my POST end point,
[HttpPost]
[Consumes("application/xml")]
[Produces("application/xml")]
public IActionResult Create([FromBody]Test data)
{
return Created("", data);
}
Here is my XML input,
<?xml version="1.0" encoding="UTF-8"?>
<Test>
<Usrno>0</Usrno>
<PCname>string</PCname>
<Best>
<Hello>string</Hello>
<Worst>
<Ko>0</Ko>
<Win>
<Kiss>string</Kiss>
</Win>
</Worst>
</Best>
</Test>
Here is the screen print of the POST method in API,
Here is my ConfigureServices in Startup.cs,
services
.AddControllers()
.AddJsonOptions(options => { options.JsonSerializerOptions.PropertyNamingPolicy = null; })
.AddXmlSerializerFormatters()
.AddXmlDataContractSerializerFormatters();
I couldn't figure out what I'm missing. Please assist
Try using a XmlElementAttribute on Best element.
XmlElement attribute indicates that a public field or property represents an XML element when the XmlSerializer serializes or deserializes the object that contains it.
C#
public class Test
{
public int Usrno { get; set; }
public string PCname { get; set; }
[XmlElement("Best")]
public List<Best> Best { get; set; }
}
The MVC's XmlSerializerInputFormatter calls XmlSerializer to deserialize the body of the request and the formetter uses this attribute to mark XML elements.
An array or List in xml serialization expects two tags like "Names" and "Name". You only have one tag so you need to add the attribute XmlElement. This issue occurs in multiple places in your classes. I fixed all the issue. See classes below
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml;
using System.Xml.Serialization;
namespace ConsoleApplication1
{
class Program
{
const string FILENAME = #"c:\temp\test.xml";
static void Main(string[] args)
{
XmlReader reader = XmlReader.Create(FILENAME);
XmlSerializer serializer = new XmlSerializer(typeof(Test));
Test test = (Test)serializer.Deserialize(reader);
}
}
public class Test
{
public int Usrno { get; set; }
public string PCname { get; set; }
[XmlElement("Best")]
public List<Best> Best { get; set; }
}
public class Best
{
public string Hello { get; set; }
[XmlElement("Worst")]
public Worst[] Worst { get; set; }
}
public class Worst
{
public int Ko { get; set; }
[XmlElement("Win")]
public Win[] Win { get; set; }
}
public class Win
{
public string Kiss { get; set; }
}
}

Nested JSON parsing in C#

Hi I have a simple nested json and am trying to parse it with Javascriptserializer(). I created a class with properties and one more class for another nested data. But am not able to access full properties
I have tried the below code
This works well and below is my class where am saving this content
when i do
Please try below code.
I have tried with Newtonsoft.Json
using System;
using System.Collections.Generic;
using Newtonsoft.Json;
public class Program
{
public static void Main()
{
var jsonData ="{\"Command\":\"te\",\"Data\":{\"Image\":\"/6D/ogARAP8\",\"Imagetype\":\"FLS\",\"Imageformat\":\"bmp\",\"MissingFingers\":[\"FLIF\",\"FLMF\"]}}";
var jsonRootObject = JsonConvert.DeserializeObject<RootObject>(jsonData);
Console.WriteLine(jsonRootObject.Data.MissingFingers[0]);
Console.WriteLine(jsonRootObject.Data.Imagetype);
Console.WriteLine(jsonRootObject.Data.Image);
}
public class Data
{
public string Image { get; set; }
public string Imagetype { get; set; }
public string Imageformat { get; set; }
public List<string> MissingFingers { get; set; }
}
public class RootObject
{
public string Command { get; set; }
public Data Data { get; set; }
}
}
Like demo here

Convert JSON Object to C# Object

I have a json object and I am trying to convert it to my c# object. Here is my JSON:
{"GuvenlikNoktaArray": {"GuvenlikNoktası": [{"Id": 1,"GuvenlikNoktası1":"SANTIYE","KartNo":"000001889174217","Sira": 1},{"Id": 2,"GuvenlikNoktası1":"INSAAT","KartNo":"000000803567858","Sira": 2},{"Id": 3,"GuvenlikNoktası1":"ÇALISMA","KartNo":"000003417926233","Sira": 3},{"Id": 4,"GuvenlikNoktası1":"GÜVENLIK","KartNo":"000001888909897","Sira": 4}]}}
And my c# class:
public partial class GuvenlikNoktası
{
public GuvenlikNoktası()
{
this.GüvenlikNoktasıOlay = new HashSet<GüvenlikNoktasıOlay>();
this.PanikButonuAlarmlari = new HashSet<PanikButonuAlarmlari>();
}
public int Id { get; set; }
public string GuvenlikNoktası1 { get; set; }
public string KartNo { get; set; }
public string Sira { get; set; }
public virtual ICollection<GüvenlikNoktasıOlay> GüvenlikNoktasıOlay { get; set; }
public virtual ICollection<PanikButonuAlarmlari> PanikButonuAlarmlari { get; set; }
}
And last, my convert try:
public void AddIstasyon(string json_string)
{
GuvenlikNoktası result = new JavaScriptSerializer().Deserialize<GuvenlikNoktası>(json_string);
}
I don't get any errors but when I debuged, I see that all attributes inside 'result' are null. It seems like an empty object. How can I get a correct 'GuvenlikNoktası' object ? (Btw I am pretty sure I am getting the json object correctly).
If you must keep this JSON structure as-is you may use JObject to navigate inside your JSON properties until you reach your target objects to deserizlize. Please can you try the code below;
PS: This code uses Newtonsoft.Json;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace SO_39847703
{
class Program
{
static void Main(string[] args)
{
string json = "{\"GuvenlikNoktaArray\": {\"GuvenlikNoktası\": [{\"Id\": 1,\"GuvenlikNoktası1\":\"SANTIYE\",\"KartNo\":\"000001889174217\",\"Sira\": 1},{\"Id\": 2,\"GuvenlikNoktası1\":\"INSAAT\",\"KartNo\":\"000000803567858\",\"Sira\": 2},{\"Id\": 3,\"GuvenlikNoktası1\":\"ÇALISMA\",\"KartNo\":\"000003417926233\",\"Sira\": 3},{\"Id\": 4,\"GuvenlikNoktası1\":\"GÜVENLIK\",\"KartNo\":\"000001888909897\",\"Sira\": 4}]}}";
AddIstasyon(json);
}
public static void AddIstasyon(string json_string)
{
dynamic jsonObject = JObject.Parse(json_string);
string jsonToDeserializeStrongType = jsonObject["GuvenlikNoktaArray"]["GuvenlikNoktası"].ToString();
List<GuvenlikNoktası> result = JsonConvert.DeserializeObject<List<GuvenlikNoktası>>(jsonToDeserializeStrongType); ;
}
}
public partial class GuvenlikNoktası
{
public GuvenlikNoktası()
{
this.GüvenlikNoktasıOlay = new HashSet<GüvenlikNoktasıOlay>();
this.PanikButonuAlarmlari = new HashSet<PanikButonuAlarmlari>();
}
public int Id { get; set; }
public string GuvenlikNoktası1 { get; set; }
public string KartNo { get; set; }
public string Sira { get; set; }
public virtual ICollection<GüvenlikNoktasıOlay> GüvenlikNoktasıOlay { get; set; }
public virtual ICollection<PanikButonuAlarmlari> PanikButonuAlarmlari { get; set; }
}
public class GüvenlikNoktasıOlay
{
}
public class PanikButonuAlarmlari
{
}
}
Hope this helps
Your JSON data and your class definition do not fit together. Therefore the default values (NULL) are provided by the serializer.
In order to deserialize the given JSON data you need a class structure like:
public class Root
{
public LevelOne GuvenlikNoktaArray {get; set;}
}
public class LevelOne {
public IEnumerable<GuvenlikNoktası> GuvenlikNoktası {get; set;}
}
You can use this class.
public class GuvenlikNoktası
{
public int Id { get; set; }
public string GuvenlikNoktası1 { get; set; }
public string KartNo { get; set; }
public int Sira { get; set; }
}
public class GuvenlikNoktaArray
{
public IList<GuvenlikNoktası> GuvenlikNoktası { get; set; }
}
public class Example
{
public GuvenlikNoktaArray GuvenlikNoktaArray { get; set; }
}
You can use this link For your referencehttp://jsonutils.com/.

Foreach tag to loop through c# object and insert to database using Entity Framework

Example of Input Json Format how it looks like:
{
Reservations:
[
{
FacilityReservationID:"....",
FacilityID:"....",
Description:"...."
},
........
]
}
Return Result format:
{
Result:"OK"/Error",
Message:"...."
}
I created a asp.net webform to accept a webservice call from a console app program in Json Restful.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using Newtonsoft.Json;
using System.Data.Entity;
namespace ReservationListWebService
{
public partial class UpdateFacility : System.Web.UI.Page
{
I created a context to call the EF Database.
public class KioskContext : DbContext
{
//using the FacilityReservationKioskEntities Connection string
public KioskContext()
: base("name=FacilityReservationKioskEntities")
{
}
public DbSet<Department> Departments { get; set; }
public DbSet<Facility> Facilitys { get; set; }
public DbSet<FacilityReservation> Reservations { get; set; }
}
public class ReservationList
{
public string facilityReservationID { get; set; }
public string facilityID { get; set; }
public DateTime startDateTime { get; set; }
public DateTime endDateTime { get; set; }
public string useShortDescription { get; set; }
public string useDescription { get; set; }
}
The console app would call my webform and hence pass me the input of a string. After deserialized into a c# object and i would eventually insert it into the database using Entity framework format. I am not sure what i should input inside the foreach() tag to loop through the c# objects and insert it into the database. I am stuck with the foreach loop.
And also how am i going to return the output result to the console app that pass me the Json data? Please help and guide! thank you!
protected void Page_Load(object sender, EventArgs e)
{
string departmentID = Request.QueryString["DepartmentID"];
string json = Request.Form["Json"];
ReservationList list = JsonConvert.DeserializeObject<ReservationList>(json);
using (var db = new KioskContext())
{
db.Database.ExecuteSqlCommand(
"DELETE FacilityReservation FROM Department INNER JOIN Facility ON Department.DepartmentID = Facility.DepartmentID" +
"INNER JOIN FacilityReservation ON Facility.FacilityID = FacilityReservation.FacilityID WHERE Department.DepartmentID = '" + departmentID + "'");
foreach(....)
{
FacilityReservation res = new FacilityReservation();
//set all fields here
res.FacilityReservationID = list.facilityReservationID;
db.Reservations.Add(res);
}
db.SaveChanges();
}
}
}
}
I believe your JSon De-Serialization is not exactly correct. You should de-serialize the JSon array to a list of reservations. Then you can loop through each of the reservation in your foreach loop.
public class FacilityReservation
{
public string facilityReservationID { get; set; }
public string facilityID { get; set; }
public DateTime startDateTime { get; set; }
public DateTime endDateTime { get; set; }
public string useShortDescription { get; set; }
public string useDescription { get; set; }
}
public class ReservationList
{
public List<FacilityReservation> Reservations {get; set;}
}
....
ReservationList list = JsonConvert.DeserializeObject<ReservationList>(json);
foreach(FacilityReservation res in list.Reservations)
{
FacilityReservation res = new FacilityReservation();
//set all fields here
res.FacilityReservationID = list.facilityReservationID;
db.Reservations.Add(res);
}
I don't see any problem with your existing code.
You only insert Reservations to database once you call db.SaveChanges(), so it's totally safe when you call db.Reservations.Add(res); inside your foreach

C# MongoDB driver save command creating duplicates

I've just started working with MongoDB for c# and I'm trying to upsert some documents in a collection. I'm using the 'save' command (and even tried update with upsert flag on) but every time I run my code it keeps inserting duplicates: the new records have new object ids (randomly generated) but all the rest of the data is the same. What am I doing wrong? Any suggestion is greatly appreciated.
Here's my code:
List<LatestDataReduced> latestdata = new List<LatestDataReduced>();
//LatestDataReduced is the model of my documents
foreach (var dep in lrd)
{
foreach (var rec in dep.record)
{
var entity = new LatestDataReduced();
entity.PlatformID = platid;
//fill up data in entity...
latestdata.Add(entity);
}
}
var connectionString = "mongodb://localhost";
var client = new MongoClient(connectionString);
var server = client.GetServer();
var database = server.GetDatabase("emodnet2");
var collection = database.GetCollection<LatestDataReduced>("latestdata");
foreach (var ld in latestdata)
{
/*var query = Query.And(Query.EQ("Id", ld.Id), Query.EQ("Date", ld.Date), Query.EQ("Depth", ld.Depth), Query.EQ("PlatformID", ld.PlatformID), Query.EQ("Pressure", ld.Pressure));
var update = Update.Replace(ld);
collection.Update(query, update, UpdateFlags.Upsert);*/
collection.Save(ld);
}
LatestDataReduced class:
using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace MongoDB.Models
{
public class CodeValuePair
{
public string Code { get; set; }
public double Value { get; set; }
}
public class LatestDataReduced
{
//[BsonIgnoreIfDefault]
public ObjectId Id { get; set; }
public int PlatformID { get; set; }
public DateTime Date { get; set; }
public double Depth { get; set; }
public List<CodeValuePair> ParamValue { get; set; }
public double Pressure { get; set; }
public List<CodeValuePair> ParamValueInfo { get; set; }
public string Roos { get; set; }
}
}
I found a way to make this work!
I thought it was mandatory to use the class "ObjectId" for Ids in mongo and I was not able to give a real identifier to my documents, but then I found out you can simply use a String Id:
public class LatestDataReduced
{
//[BsonIgnoreIfDefault]
public String Id { get; set; }
public int PlatformID { get; set; }
public DateTime Date { get; set; }
public double Depth { get; set; }
public List<CodeValuePair> ParamValue { get; set; }
public double Pressure { get; set; }
public List<CodeValuePair> ParamValueInfo { get; set; }
public string Roos { get; set; }
}
so I gave my documents a valid identifier and mongo is now able to recognize equal documents correctly and update them instead of inserting them.
Thank you very much for your attention!
Have a nice monday

Categories

Resources