Develop Reference

Best way of bulk inserting a list of records in DuckDB using DuckDB.NET

Context
I'm working with the awesome DuckDB.NET library and C# for
this example. I'm specifically working with the ADO.NET provider.
Problem
My database contains the following table:
CREATE TABLE tbl01 (
Id INTEGER,
TextListTest TEXT[],
DateTest DATE
);
In the program each record is encapsulated by a class:
class Record
{
public int Id { get; set; }
public List<string> TextListTest { get; set; };
public DateTime TextListTest { get; set; };
}
and is appended to a List<Record>. This list gets
very large so I would like to avoid the per-row overhead of INSERT statements in a
loop. The documentation says that if I absolutely must use
inserts in this manner I should also wrap them in calls of BEGIN TRANSACTION and COMMIT. I
really don't want to miss out on insert performance here. Is there another approach I can use with
the library I'm using?
I noticed in the DuckDB.NET
sample with the
LowLevelBindingsSample() method that I could use prepared statements but I'm not sure that if that would confer any performance benefits.
Is there an approach I'm missing - perhaps the appender? If someone could provide an example using the 3 specific data
types in the above table that would be greatly appreciated (I'm having trouble figuring out the
LIST column).
using DuckDB.NET.Data;
namespace DuckTest;
class Record
{
public int Id { get; set; }
public List<string> TextListTest { get; set; }
public DateTime DateTest { get; set; }
}
class Program
{
public static void Main(string[] args)
{
// pretend this is a really big list!
List<Record> recordList = new List<Record>
{
new Record { Id = 1, TextListTest = new List<string> { "Ball", "Horse" }, DateTest = new DateTime(1994, 12, 3) },
new Record { Id = 2, TextListTest = new List<string> { "Matthew", "Giorgi is cool!" }, DateTest = new DateTime(1998, 11, 28) },
new Record { Id = 3, TextListTest = new List<string> { "Red", "Black", "Purple" }, DateTest = new DateTime(1999, 9, 13) },
new Record { Id = 4, TextListTest = new List<string> { "Cat" }, DateTest = new DateTime(1990, 2, 5) },
};
using (var duckDBConnection = new DuckDBConnection("Data Source=db01.duckdb"))
{
duckDBConnection.Open();
var command = duckDBConnection.CreateCommand();
command.CommandText = "CREATE TABLE tbl01 ( Id INTEGER, TextListTest TEXT[], DateTest DATE );";
var executeNonQuery = command.ExecuteNonQuery();
// I could do this in a loop but there's probably a better way...
command.CommandText = "INSERT INTO tbl01 VALUES (1, ['Ball', 'Horse'], '1994-12-03');";
executeNonQuery = command.ExecuteNonQuery();
}
}
}
I would be willing to use the low level bindings library if needed.

The library has support for appenders so that should be the fastest way to import data.

Error on class having definiton errors

InitializeComponent();
var connectionString = "mongodb://localhost:27017";
var client = new MongoClient(connectionString);
var database = client.GetDatabase("test");
var collection = database.GetCollection<Rootobject>("container");
var res = collection.Aggregate().Group(x => x._event.NodeId, x => new { Name = x.Key, WTs = x.Select(r => r._event.WT).ToList() }).ToList();
cartesianChart1.Series = new SeriesCollection
{
new LineSeries
{
Title = res["event"]["NodeId"] ,
Values = new ChartValues<double> {4.0, 3.9, 3.9, 3.85, 3.8, 3.8, 3.75}
},
}
The code above , i am trying to use my mongodb for my graphs input . But first i need to be able to query from MongoDB , i have come up with a var record but the 'NodeId' still has errors. it has Event[] does not contain a definition for 'NodeID' and no extension method 'NodeId' accepting a first argument of type Event[]. Below are my public classes
public class Rootobject
{
public string _id { get; set; }
public Header[] header { get; set; }
public Event[] _event { get; set; }
}

The problem is in this part of your code:
.Group(x => x._event.NodeId
Your _event property is of type Event[], which is an array of Event. You can get one item of your array and access the NodeId property (e.g. the from the first element with _event[0].NodeId, which will work if this element exists), but not from the array itself.
I can not tell you with what you should replace this part with, because that depends entirely on what you want to achieve.

Obtaining Child element via LookUp/Something better

I am trying to obtain a child element of a parent element. I have a basic database as so:
public class list
{
public int id { get; set; }
public List<Users> UsersList{ get; set; }
public class Users
{
[Key]
public int Users_id{ get; set; }
public string UserId { get; set; }
}
}
If I was wanting to obtain all of the of the elements in Users that had the specific UserId how would I do that? I am trying to refrain from using a nested for loop and iterating through all my entries of List and Users in the database. I was looking up the LookUp(), but am a bit confused on how to apply it in this case. Any help would be great!

Since you're a bit confused, I'll provide more detail than my original answer. Let's take your code and create a very basic and crude sample program:
using System;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations;
using System.Linq;
namespace SOSample
{
public class list
{
public int id { get; set; }
public List<Users> UsersList { get; set; }
public class Users
{
[Key]
public int Users_id { get; set; }
public string UserId { get; set; }
}
}
class Program
{
static void Main(string[] args)
{
// Instantiate and initialize with sample data.
var sampleList = new list()
{
id = 12345,
UsersList = new List<list.Users>()
{
new list.Users() { Users_id = 1, UserId = "0042" },
new list.Users() { Users_id = 2, UserId = "0019" },
new list.Users() { Users_id = 3, UserId = "0036" },
new list.Users() { Users_id = 4, UserId = "0214" },
new list.Users() { Users_id = 5, UserId = "0042" },
new list.Users() { Users_id = 6, UserId = "0042" },
new list.Users() { Users_id = 7, UserId = "0019" }
}
};
// Linq search.
var someId = "0042";
var linqQuery = sampleList.UsersList.Where(user => user.UserId == someId);
Console.WriteLine("Linq query results:");
foreach (var r in linqQuery)
{
Console.WriteLine($"Users_id: {r.Users_id}, UserId: {r.UserId}");
}
// Lookup search (using same someId as for Linq).
var lookup = sampleList.UsersList.ToLookup(user => user.UserId);
var lookupQuery = lookup[someId];
Console.WriteLine("\nLookup query results:");
foreach (var r in lookupQuery)
{
Console.WriteLine($"Users_id: {r.Users_id}, UserId: {r.UserId}");
}
}
}
}
Output:
Linq query results:
Users_id: 1, UserId: 0042
Users_id: 5, UserId: 0042
Users_id: 6, UserId: 0042
Lookup query results:
Users_id: 1, UserId: 0042
Users_id: 5, UserId: 0042
Users_id: 6, UserId: 0042
Hope that clarifies things. The major issue I see with your question and comments is that it's possible that you're mistaking nested classes for properties. When you instantiate an outer class, the inner class does not get instantiated and it's not some sort of property of an outer class.
Old answer (provides individual details):
I like using Linq. So, assuming sampleList is of type list:
var query = sampleList.UsersList.Where(user => user.UserId == someId);
That's going to give you IEnumerable<list.Users>. You can always use ToList(), ToArray(), ToDictionary() to get the desired collection type:
var results = sampleList.UsersList.Where(user => user.UserId == someId).ToArray();
As far as Lookup, I've seen a few ways it being used, but the most familiar way for me is this:
var lookup = sampleList.UsersList.ToLookup(user => user.UserId);
var query = lookup[someId];
Once again, that'll give you IEnumerable<list.Users>. Alternatively, you can get the collection type of your choice from that query:
var results = lookup[someId].ToArray();
Basically, you're specifying what the key will represent in that lookup (it's the UserId in this case) and then when the time comes, you search by a key.

Converting datatable into hierarchical data structure (JSON) using C#

I am executing a SQL query into datatable. The query can return multiple columns. The result is in key value format and represent hierarchical data. See the below screenshot.
The image shows 3 parts. First the data, then the hierarchical representation of data and JSON equivalent.
Currently the image shows 4 level of data, but we can have 6-7 levels of data. The format will remain same but the number of columns can change.
How can i get the desired result using C#? ? I know it is basic programming but I am having hard time with it.

With a fixed-level hierarchy like you are showing in your example, you can use a LINQ query with grouping to generate the tree structure for your JSON. Here is an example with a three-level hierarchy:
static void Main(string[] args)
{
var data = new List<Data>
{
new Data("Food", "1_g", "beverage", "2_b", "hot", "3_h"),
new Data("Food", "1_g", "beverage", "2_b", "cold", "3_c"),
new Data("Food", "1_g", "fruit", "2_f", "juicy", "3_j"),
new Data("Food", "1_g", "fruit", "2_f", "solid", "3_s"),
new Data("Food", "1_g", "cookie", "2_c", "chocolate", "3_cho"),
};
var tree = from l1 in data
group l1 by new { key = l1.Key_L1, name = l1.L1 } into group1
select new
{
key = group1.Key.key,
name = group1.Key.name,
children = from l2 in group1
group l2 by new { key = l2.Key_L2, name = l2.L2 } into group2
select new
{
key = group2.Key.key,
name = group2.Key.name,
children = from l3 in group2
select new { key = l3.Key_L3, name = l3.L3 }
}
};
var serializer = new JavaScriptSerializer();
Console.WriteLine(serializer.Serialize(tree));
Console.ReadLine();
}
class Data
{
public Data(string l1, string k1, string l2, string k2, string l3, string k3)
{
L1 = l1; Key_L1 = k1;
L2 = l2; Key_L2 = k2;
L3 = l3; Key_L3 = k3;
}
public string L1 { get; set; }
public string Key_L1 { get; set; }
public string L2 { get; set; }
public string Key_L2 { get; set; }
public string L3 { get; set; }
public string Key_L3 { get; set; }
}
The above is a working example of the technique using POCOs.
You mention "datatable"; I assume this is referring to the .NET DataTable class? If so, you can use LINQ to query a DataTable. You just need to convert it to an enumerable using the DataSetExtensions. See: LINQ query on a DataTable
Then in the LINQ statement, you replace the list with your data table .AsEnumerable() and replace the property references with .Field<string>(""). Like so:
DataTable dt = // load data table
var tree = from l1 in dt.AsEnumerable()
group l1 by new { key = l1.Field<string>("Key_L1"), name = l1.Field<string>("L1") } into group1
select new
{
// etc.
};
For a variable number of columns, you have to use a recursive approach, like so:
var tree = Descend(dt.AsEnumerable(), 1, 3);
private static System.Collections.IEnumerable Descend(IEnumerable<DataRow> data, int currentLevel, int maxLevel)
{
if (currentLevel > maxLevel)
{
return Enumerable.Empty<object>();
}
return from item in data
group item by new
{
key = item.Field<string>("Key_L" + currentLevel),
name = item.Field<string>("L" + currentLevel)
} into rowGroup
select new
{
key = rowGroup.Key.key,
name = rowGroup.Key.name,
children = Descend(rowGroup, currentLevel + 1, maxLevel)
};
}
One thing to note is that this approach creates an empty children collection on the leaf nodes.

How do I convert an IEnumerable to JSON?

I have a method that returns an IEnumberable containing 1..n records. How do I convert the results to a JSON string?
Thanks!

IEnumerable<int> sequenceOfInts = new int[] { 1, 2, 3 };
IEnumerable<Foo> sequenceOfFoos = new Foo[] { new Foo() { Bar = "A" }, new Foo() { Bar = "B" } };
var serializer = new System.Web.Script.Serialization.JavaScriptSerializer();
string outputOfInts = serializer.Serialize(sequenceOfInts);
string outputOfFoos = serializer.Serialize(sequenceOfFoos);
Which produces the output
[1,2,3]
[{"Bar":"A"},{"Bar":"B"}]
And then you can get your sequence back
IEnumerable<Foo> foos = serializer.Deserialize<IEnumerable<Foo>>(outputOfFoos);

Maybe you can try this:
var categories = from c in db.tableone
select new { key = c.tableoneID, value = c.tableoneName };
JsonResult categoryJson = new JsonResult();
categoryJson.Data = categories;
return categoryJson;

You can do it like this using .NET Framework itself and without using any 3rd party tools
using System.Web.Script.Serialization;
public class Json
{
public string getJson()
{
// some code //
var products = // IEnumerable object //
string json = new JavaScriptSerializer().Serialize(products);
// some code //
return json;
}
}

When using MVC you can use the "System.Web.Helpers.Json class". I needed a few items rendered with the page in json:
public class Product
{
public int Id { get; set; }
public string Name { get; set; }
public int Categorie { get; set; }
}
In the view:
#{
var products = new List<Product> {
new Product{ Id = 1, Name = "test product", Categorie = 1},
new Product{ Id = 2, Name = "another product",Categorie = 1},
new Product{ Id = 3, Name = "more stuff",Categorie = 1},
new Product{ Id = 4, Name = "even more",Categorie = 2},
new Product{ Id = 5, Name = "and the last",Categorie = 2}
};
}
//fill the javascript variable with products
var products= #(Html.Raw(Json.Encode(products) ));
Note the Html.Raw...
While this can be helpfull, don't over use it. Rendering large sections of data into your page makes the page large and may cause performance issues when the browser can not cache your results. If you need more data, use a REST call, so the browser can cache the results.

I would look into a 3rd party tool to do the conversion of your objects to JSON. Here is a good one: http://json.codeplex.com/