I've just started working with MongoDB for c# and I'm trying to upsert some documents in a collection. I'm using the 'save' command (and even tried update with upsert flag on) but every time I run my code it keeps inserting duplicates: the new records have new object ids (randomly generated) but all the rest of the data is the same. What am I doing wrong? Any suggestion is greatly appreciated.
Here's my code:
List<LatestDataReduced> latestdata = new List<LatestDataReduced>();
//LatestDataReduced is the model of my documents
foreach (var dep in lrd)
{
foreach (var rec in dep.record)
{
var entity = new LatestDataReduced();
entity.PlatformID = platid;
//fill up data in entity...
latestdata.Add(entity);
}
}
var connectionString = "mongodb://localhost";
var client = new MongoClient(connectionString);
var server = client.GetServer();
var database = server.GetDatabase("emodnet2");
var collection = database.GetCollection<LatestDataReduced>("latestdata");
foreach (var ld in latestdata)
{
/*var query = Query.And(Query.EQ("Id", ld.Id), Query.EQ("Date", ld.Date), Query.EQ("Depth", ld.Depth), Query.EQ("PlatformID", ld.PlatformID), Query.EQ("Pressure", ld.Pressure));
var update = Update.Replace(ld);
collection.Update(query, update, UpdateFlags.Upsert);*/
collection.Save(ld);
}
LatestDataReduced class:
using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace MongoDB.Models
{
public class CodeValuePair
{
public string Code { get; set; }
public double Value { get; set; }
}
public class LatestDataReduced
{
//[BsonIgnoreIfDefault]
public ObjectId Id { get; set; }
public int PlatformID { get; set; }
public DateTime Date { get; set; }
public double Depth { get; set; }
public List<CodeValuePair> ParamValue { get; set; }
public double Pressure { get; set; }
public List<CodeValuePair> ParamValueInfo { get; set; }
public string Roos { get; set; }
}
}
I found a way to make this work!
I thought it was mandatory to use the class "ObjectId" for Ids in mongo and I was not able to give a real identifier to my documents, but then I found out you can simply use a String Id:
public class LatestDataReduced
{
//[BsonIgnoreIfDefault]
public String Id { get; set; }
public int PlatformID { get; set; }
public DateTime Date { get; set; }
public double Depth { get; set; }
public List<CodeValuePair> ParamValue { get; set; }
public double Pressure { get; set; }
public List<CodeValuePair> ParamValueInfo { get; set; }
public string Roos { get; set; }
}
so I gave my documents a valid identifier and mongo is now able to recognize equal documents correctly and update them instead of inserting them.
Thank you very much for your attention!
Have a nice monday
Related
I want to read some data from a JSON API in SSIS and write it to a table in SQL Server. I've solved the task using a 3rd party, but the solution isn't that elegant, so now I'm trying to script it myself in Visual Studio using SSIS's script component.
I've researched around the web for solutions, and ended with this result. So far, I'm fairly confident about what is going on, but I lack the final direction for this. I know I need to somehow map the output to the columns I've created in SSIS.
I guess I have to do something around CreateNewOutputRows(), but I'm not sure what. Can someone please help me out on this? Also, since this is more or less my first ever c# script, I would also appreciate it, if there's a way easier solution OR if it is in some way inappropriate etc.
First of all, the output from the API looks like this (API documentation here):
"data":[
{
"created_at":"2016-03-12 09:45:00",
"created_at_unix":1457772300,
"shop_name":"DK - Some name",
"location_id":1111,
"custom_location_id":"2222",
"custom_shop_id":"2222",
"shop_id":3333,
"count_in":"1",
"count_out":"1",
"timekey":3
}
The script I've got so far is
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using System.Linq;
using System.Net;
using System.Collections.Generic;
using Newtonsoft.Json;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
public static void Main()
{
//Test api
var url = "https://login.vemcount.com/api/fetch_data/?data={"api_key":"xxxxxxx","type":"company_id","id":"10","date_from":"2019-01-01 00:00","interval":"15min", "group_by":"shop_id"}";
var json = new WebClient().DownloadString(url);
var root = JsonConvert.DeserializeObject<RootObject>(json);
//Printing last record
Console.WriteLine(root.data.Last().created_at);
}
public class data
{
public string created_at { get; set; }
public int created_at_unix { get; set; }
public string shop_name { get; set; }
public int location_id { get; set; }
public string custom_location_id { get; set; }
public string custom_shop_id { get; set; }
public int shop_id { get; set; }
public string count_in { get; set; }
public string count_out { get; set; }
public int timekey { get; set; }
}
public class RootObject
{
public List<data> data { get; set; }
}
public override void CreateNewOutputRows()
{
}
}
You'll put all that code in CreateNewOutputRows().
First you have to manually add all the columns to the script component. In the example here I only added 2 columns:
The code goes in CreateNewOutPutRows(). I don't have Newtonsoft, just using the JavaScriptSerializer here to show a working example so you can see how to hook it up:
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using System.Web.Script.Serialization;
using System.Collections.Generic;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
public override void CreateNewOutputRows()
{
string json = #"{""data"":[{""created_at"":""2016-03-12 09:45:00"",""created_at_unix"":1457772300,""shop_name"":""DK - Some name"",""location_id"":1111,""custom_location_id"":""2222"",""custom_shop_id"":""2222"",""shop_id"":3333,""count_in"":""1"",""count_out"":""1"",""timekey"":3},{""created_at"":""2016-03-12 09:45:00"",""created_at_unix"":1457772300,""shop_name"":""test2"",""location_id"":1111,""custom_location_id"":""2222"",""custom_shop_id"":""2222"",""shop_id"":3333,""count_in"":""1"",""count_out"":""1"",""timekey"":3}]}";
RootObject Test = new JavaScriptSerializer().Deserialize<RootObject>(json);
/*
* This is where data gets added to the output buffer.
* After AddRow() you are basically mapping the column you manually added(on the left) to the data(on the right).
* using a foreach loop to loop through the deserialize json
*/
foreach (var item in Test.data)
{
Output0Buffer.AddRow();
Output0Buffer.createdat = item.created_at;
Output0Buffer.shopname = item.shop_name;
}
}
public class RootObject
{
public List<data> data { get; set; }
}
public class data
{
public string created_at { get; set; }
public int created_at_unix { get; set; }
public string shop_name { get; set; }
public int location_id { get; set; }
public string custom_location_id { get; set; }
public string custom_shop_id { get; set; }
public int shop_id { get; set; }
public string count_in { get; set; }
public string count_out { get; set; }
public int timekey { get; set; }
}
}
Then in this example, just using a record set destination and enabled data viewer so you can see the individuals rows come back out:
Using the EF, a Rates table was created based on the Rate model below:
public class Rate
{
[Key]
public int IdNo { get; set; }
public string RateCode { get; set; }
public string RateClass { get; set; }
public string Basis { get; set; }
public bool RateIsActive { get; set; }
public DateTime? ValidFrom { get; set; }
public DateTime? ValidTo { get; set; }
public bool IsReservRate { get; set; }
public decimal? CostDay { get; set; }
public decimal? CostWeek { get; set; }
public decimal? CostMonth { get; set; }
public decimal? CostHour { get; set; }
}
I have no problem retrieving all columns from the Rates table using the code below:
using (AppDbContext db = new AppDbContext())
{
DbSqlQuery<Rate> data = db.Rates.SqlQuery("select * from Rates");
}
but what I need is to select only two columns from the table
DbSqlQuery<Rate> data = db.Rates.SqlQuery("select Rate, Class from Rates");
and it gives me the following error:
The data reader is incompatible with the specified 'RPManager.Models.Rate'. A member of the type, 'RateIsActive', does not have a corresponding column in the data reader with the same name.
Is there a way to active what is needed?
NOTE!!!! I know how to do it using Linq but for this specific case it has to be done, if possible, using db.Rates.SqlQuery()
Is there a way to active what is needed?
Not with the code provided. You have a few options.
Your best option:
Create a class that represents what you need instead of reusing a class just because:
public class RateInfo
{
public x Rate { get; set; }
public x Class { get; set; }
}
var data = db.Rates.SqlQuery<RateInfo>("select Rate, Class from Rates");
or not quite as good; set default values via the sql alias:
// assumming you have an Id column
var data = db.Rates.SqlQuery<Rate>("select 0 as ID, Rate, Class from Rates");
using ServiceStack;
using ServiceStack.OrmLite;
public static string SqliteFileDb = "~/App_Data/db.sqlite".MapHostAbsolutePath();
private static void CreateX(Message msg)
{
//Using Sqlite DB- improved
var dbFactory = new OrmLiteConnectionFactory(SqliteFileDb, SqliteDialect.Provider);
// Wrap all code in using statement to not forget about using db.Close()
using (var db = dbFactory.Open())
{
db.CreateTableIfNotExists<Message>();
Message notex = new Message();
notex.Content = msg.Content;
notex.Datestamp = msg.Datestamp;
notex.Facility = msg.Facility;
notex.Hostname = msg.Hostname;
notex.LocalDate = msg.LocalDate;
notex.RemoteIP = msg.RemoteIP;
notex.Severity = msg.Severity;
db.Save(notex))
db.Close();
}
}
public class Message
{
public FacilityType Facility { get; set; }
public SeverityType Severity { get; set; }
public DateTime Datestamp { get; set; }
public string Hostname { get; set; }
public string Content { get; set; }
public string RemoteIP{ get; set; }
public DateTime LocalDate { get; set; }
}
Can someone advice how to address this
case where I am saving a syslog message
to an sqlite db using servicestack orm.
Seems only one object is always available
and gets updated.Hence no new record getting
created.
If you don't provide a Primary Key in OrmLite, OrmLite will assume the primary key is the first property in the table which is not what you want in this case. You either need to tell OrmLite which property it should use for the Primary Key by annotating it with the [PrimaryKey] attribute, or just add an auto incrementing primary key which the database will populate itself, e.g:
public class Message
{
[AutoIncrement]
public in Id { get; set; }
public FacilityType Facility { get; set; }
public SeverityType Severity { get; set; }
public DateTime Datestamp { get; set; }
public string Hostname { get; set; }
public string Content { get; set; }
public string RemoteIP{ get; set; }
public DateTime LocalDate { get; set; }
}
Also db.Close() is redundant in a using statement and there's no feature you want to use with OrmLite's high-level Save() API in this case so you should just have:
using (var db = dbFactory.Open())
{
db.CreateTableIfNotExists<Message>();
Message notex = new Message();
notex.Content = msg.Content;
notex.Datestamp = msg.Datestamp;
notex.Facility = msg.Facility;
notex.Hostname = msg.Hostname;
notex.LocalDate = msg.LocalDate;
notex.RemoteIP = msg.RemoteIP;
notex.Severity = msg.Severity;
db.Insert(notex);
}
Example of Input Json Format how it looks like:
{
Reservations:
[
{
FacilityReservationID:"....",
FacilityID:"....",
Description:"...."
},
........
]
}
Return Result format:
{
Result:"OK"/Error",
Message:"...."
}
I created a asp.net webform to accept a webservice call from a console app program in Json Restful.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using Newtonsoft.Json;
using System.Data.Entity;
namespace ReservationListWebService
{
public partial class UpdateFacility : System.Web.UI.Page
{
I created a context to call the EF Database.
public class KioskContext : DbContext
{
//using the FacilityReservationKioskEntities Connection string
public KioskContext()
: base("name=FacilityReservationKioskEntities")
{
}
public DbSet<Department> Departments { get; set; }
public DbSet<Facility> Facilitys { get; set; }
public DbSet<FacilityReservation> Reservations { get; set; }
}
public class ReservationList
{
public string facilityReservationID { get; set; }
public string facilityID { get; set; }
public DateTime startDateTime { get; set; }
public DateTime endDateTime { get; set; }
public string useShortDescription { get; set; }
public string useDescription { get; set; }
}
The console app would call my webform and hence pass me the input of a string. After deserialized into a c# object and i would eventually insert it into the database using Entity framework format. I am not sure what i should input inside the foreach() tag to loop through the c# objects and insert it into the database. I am stuck with the foreach loop.
And also how am i going to return the output result to the console app that pass me the Json data? Please help and guide! thank you!
protected void Page_Load(object sender, EventArgs e)
{
string departmentID = Request.QueryString["DepartmentID"];
string json = Request.Form["Json"];
ReservationList list = JsonConvert.DeserializeObject<ReservationList>(json);
using (var db = new KioskContext())
{
db.Database.ExecuteSqlCommand(
"DELETE FacilityReservation FROM Department INNER JOIN Facility ON Department.DepartmentID = Facility.DepartmentID" +
"INNER JOIN FacilityReservation ON Facility.FacilityID = FacilityReservation.FacilityID WHERE Department.DepartmentID = '" + departmentID + "'");
foreach(....)
{
FacilityReservation res = new FacilityReservation();
//set all fields here
res.FacilityReservationID = list.facilityReservationID;
db.Reservations.Add(res);
}
db.SaveChanges();
}
}
}
}
I believe your JSon De-Serialization is not exactly correct. You should de-serialize the JSon array to a list of reservations. Then you can loop through each of the reservation in your foreach loop.
public class FacilityReservation
{
public string facilityReservationID { get; set; }
public string facilityID { get; set; }
public DateTime startDateTime { get; set; }
public DateTime endDateTime { get; set; }
public string useShortDescription { get; set; }
public string useDescription { get; set; }
}
public class ReservationList
{
public List<FacilityReservation> Reservations {get; set;}
}
....
ReservationList list = JsonConvert.DeserializeObject<ReservationList>(json);
foreach(FacilityReservation res in list.Reservations)
{
FacilityReservation res = new FacilityReservation();
//set all fields here
res.FacilityReservationID = list.facilityReservationID;
db.Reservations.Add(res);
}
I don't see any problem with your existing code.
You only insert Reservations to database once you call db.SaveChanges(), so it's totally safe when you call db.Reservations.Add(res); inside your foreach
I need after loading the data from a model, automatically insert an information on a property of this type which do not come from the DB.
Exemple:
namespace project.Models
{
using System;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations;
using System.ComponentModel.DataAnnotations.Schema;
using System.Data.Entity;
public partial class MessageSource
{
public MessageSource()
{
this.messages = new HashSet<Message>();
}
public int id { get; set; }
public string category { get; set; }
public string message { get; set; }
public string message_translation { get; set; }
public virtual ICollection<Message> messages { get; set; }
}
}
id, category and message are DB table properties, message_translation no, after loading this model, always I will need to put automatically information in message_translation. How?
I do this:
public partial class MessageSource
{
public MessageSource()
{
this.messages = new HashSet<Message>();
}
public int id { get; set; }
public string category { get; set; }
public string message { get; set; }
public string message_translation { get; set; }
public virtual ICollection<Message> messages { get; set; }
public string getTranslate()
{
if(message_translation == null)
{
Message msg = messages.FirstOrDefault(m => m.language.Equals(Translate.Instance.getLanguage()));
if(msg != null)
{
message_translation = msg.translation;
}
else
{
message_translation = String.Empty;
}
}
return message_translation;
}
}
works good
public partial class MessageSource
{
private string _defaultLanguage;
public MessageSource(string defaultLang)
{
this.messages = new HashSet<Message>();
this._defaultLanguage = defaultLanguage ;
}
public int id { get; set; }
public string category { get; set; }
public string message { get; set; }
public string message_translation { get;
set{ value = getTranslation(_defaultLnag); }
public virtual ICollection<Message> messages { get; set; }
public string getTranslate(string lang)
{
var Translation = dbcontext.messages.Where(m=> m.Id == this.Id).FirstOrDefault().Translation.Where(t=>t.lang == lang).FirstOrDefault();
return Translation;
}
}
this solution assume that you are saving the translations in a separate table linked to the messages tables
messageSource m = new messageSource();
var m2 = // query your db using sql or entity framework
Then
m.id= m2.id;
m.category= m.category;
m.message= m2.message
m.message_translation= "vvvvvvv whatever"//do add thing here
m.messages = m2.messages;
Edit:
Another solution
Public class translation {
public int Id { get; set;}
public string language { get; set;}
public string text { get; set;}
}
Then you add in your messageSource model the following after removing message_translation
Public Icollection<translation> transMsg { get; set;}
Now suppose you have a msg that is translated into 13 languages; then the ICollection<transMsg> will have 13 item, and we need only the chinese translation
Do as follow:
First get your message
var myMsg = dbContext.messageSources.Where(m=>m.id== 6).FirstOrDefault();
This will not load any translation bec. We didnt use Include() , i.e. lazy loading
However, when we want the translation we pull it via another Linq query as follow:
var myMsgTranslation = myMsg.transMsgs.Where(t=>t.lang == "chinese").FirstOrDefault();
Or you load all the translations as follow
var myMsg = dbContext.messageSources.Where(m=>m.id== 6).Include(m=>m.msgTrans).FirstOrDefault();
And in yor View page u pull the default language as follow
#html.textBoxFor(model=>model.msgTrans.Where(t=>t.lang == "chinese"))
Desclaimer: not tested
Hope that help