Develop Reference

DynamoDB Data Modeling - Hierarchical Data Structures as items

The access patterns that I'm interested in is the last item for a given exchange and an account name.
+-------------------------------+-------------------------------------------+---------+--------------------------------------------+------------+----------+------------+---------+-----------+----------------+--------------------------------------------------------------------+---------------------------+----------------------------------+--------------+------------------------------+
| PK | SK | Account | Address | AddressTag | Exchange | Instrument | Network | Quantity | TransactionFee | TransactionId | TransferDate | TransferId | TransferType | UpdatedAt |
+-------------------------------+-------------------------------------------+---------+--------------------------------------------+------------+----------+------------+---------+-----------+----------------+--------------------------------------------------------------------+---------------------------+----------------------------------+--------------+------------------------------+
| Exchange#Binance#Account#main | TransferDate#12/17/2022 4:59:12 PM +02:00 | main | 0xF76d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null | Binance | USDT | ETH | 97.500139 | 3.2 | 0x46d28f7d0e1e5b1d074a65dcfbb9d90b3bcdc7e6fca6b1f1f7abb5ab219feb24 | 2022-12-17T16:59:12+02:00 | 1b56485f6a3446c3b883f4f485039260 | 0 | 2023-01-28T20:19:59.9181573Z |
| Exchange#Binance#Account#main | TransferDate#12/17/2022 5:38:23 PM +02:00 | main | 0xF76d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null | Binance | USDT | ETH | 3107.4889 | 3.2 | 0xbb2b92030b988a0184ba02e2e754b7a7f0f963c496c4e3473509c6fe6b54a41d | 2022-12-17T17:38:23+02:00 | 4747f6ecc74f4dd8a4b565e0f15bcf79 | 0 | 2023-01-28T20:20:00.4536839Z |
| Exchange#FTX#Account#main | TransferDate#12/17/2021 5:38:23 PM +02:00 | main | 0x476d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null | FTX | USDT | ETH | 20 | 3.2 | 0xaa2b92030b988a0184ba02e2e754b7a7f0f963c496c4e3473509c6fe6b54a41d | 2021-12-17T17:38:23+02:00 | 4747f6ecc74f4dd8a4b565e0f15bcf79 | 0 | 2023-01-28T20:20:00.5723855Z |
| Exchange#FTX#Account#main | TransferDate#12/19/2022 4:59:12 PM +02:00 | main | 0xc46d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null | FTX | USDT | ETH | 15 | 3.2 | 0xddd28f7d0e1e5b1d074a65dcfbb9d90b3bcdc7e6fca6b1f1f7abb5ab219feb24 | 2022-12-19T16:59:12+02:00 | 1b56485f6a3446c3b883f4f485039260 | 0 | 2023-01-28T20:20:00.5207119Z |
+-------------------------------+-------------------------------------------+---------+--------------------------------------------+------------+----------+------------+---------+-----------+----------------+--------------------------------------------------------------------+---------------------------+----------------------------------+--------------+------------------------------+
First of all, it seems to be working as expected but as I'm still learning I'm not so sure whether the partition key and the sort key I chose are good enough or not. This is important as "Uneven distribution of data due to the wrong choice of partition key" can cause reading/writing above the limit issues.
There was a similar example in the documentation and what they say about TransactionId being a partition key is as following:
In most cases you won’t use TransactionID for any query purposes, so you lose the ability to use the partition key to perform a fast lookup of data. To expand this reasoning, consider the traditional order history view on an e-commerce site. Normally orders are retrieved by customer ID or Order ID, not a UID such as a transaction ID that was synthetically generated during checkout. It’s better to choose a natural partition key than generate a synthetic one that won’t be used for querying.
Another interesting part of the documentation is about the composite key
Composite sort keys let you define hierarchical (one-to-many) relationships in your data that you can query at any level of the hierarchy
[country]#[region]#[state]#[county]#[city]#[neighborhood]
This would let you make efficient range queries for a list of locations at any one of these levels of aggregation, from country, to a neighborhood, and everything in between.
I'm also interested in the "Get all user transfers by date range" access pattern but I'm not sure how I could achieve it. So here we are.
C# implementation
public async Task<UserTransferDto?> GetLastAsync(string exchange, string account)
{
var queryRequest = new QueryRequest
{
TableName = TableName,
KeyConditionExpression = "#pk = :pk",
ExpressionAttributeNames = new Dictionary<string, string>
{
{ "#pk", "PK" }
},
ExpressionAttributeValues = new Dictionary<string, AttributeValue>
{
{ ":pk", new AttributeValue { S = $"Exchange#{exchange}#Account#{account}" } }
},
ScanIndexForward = false,
Limit = 1
};
var response = await _dynamoDb.QueryAsync(queryRequest);
if (response.Items.Count == 0)
{
return null;
}
var itemAsDocument = Document.FromAttributeMap(response.Items[0]);
return JsonSerializer.Deserialize<UserTransferDto>(itemAsDocument.ToJson());;
}
public class UserTransferDto
{
[JsonPropertyName("PK")]
public string Pk => $"Exchange#{Exchange}#Account#{Account}";
[JsonPropertyName("SK")]
public string Sk => $"TransferDate#{TransferDate}";
public required string Exchange { get; init; }
public required string Account { get; init; }
public required DateTimeOffset TransferDate { get; init; }
public required string TransferId { get; init; }
public required TransferType TransferType { get; init; }
public required string Instrument { get; init; }
public required string Network { get; init; }
public required decimal Quantity { get; init; }
public required string Address { get; init; }
public string? AddressTag { get; init; }
public decimal? TransactionFee { get; init; }
public string? TransactionId { get; init; }
public DateTime UpdatedAt { get; set; }
}
public enum TransferType
{
Withdraw = 0,
Deposit = 1
}
Sources:
https://youtu.be/HaEPXoXVf2k?t=720
https://youtu.be/HaEPXoXVf2k?t=798
Hierarchical Data Structures as Items https://youtu.be/HaEPXoXVf2k?t=2775
Access Patterns https://youtu.be/HaEPXoXVf2k?t=2903
https://aws.amazon.com/blogs/database/choosing-the-right-dynamodb-partition-key/
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-sort-keys.html

Your base table design works well for getting the latest item for a given exchange and account (via a Query with that as the PK and getting the last sortable from the SK), except that you’re using non-sortable human time stamps instead of sortable time stamps. You should use 2023-01-28 12:56:08 and so on so that the times sort right as strings.
For the other query to find the latest across all exchanges and accounts, you can create a GSI which has a singular PK and the times as the SK. Just beware that you’re limited in how many writes per second you can do to the same PK. Above 1,000 write units per second you’ll need to shard it and then do a query for each shard to get the latest per shard and then the latest overall.
This is a pattern described in https://youtu.be/0iGR8GnIItQ

Join and sort 2 not related tables

I have 2 entities: Post and Article and I need to select both of them at once and sort by common property. I'm searching for some "join" possibilities, but all I'm finding are examples ending with something like ...where a.id == b.a_id which is not relevant in my case.
That's my models
class BaseEntity {
public int Id;
public DateTime AddedDate;
}
class Post extends BaseEntity {
public string Title;
}
class Article extends BaseEntity {
public string Name;
}
For that data:
Articles:
Id | Name | AddedDate
1 | AAA | 15/05/2020 18:00:00
2 | BBB | 17/05/2020 18:00:00
3 | CCC | 19/05/2020 18:00:00
Posts:
Id | Title | AddedDate
1 | DDD | 16/05/2020 18:00:00
2 | EEE | 18/05/2020 18:00:00
3 | FFF | 20/05/2020 18:00:00
when sorting by date DESC and limit it to 4 i need to end up with something like that, or something similar:
Article Id | PostId | Name | Title | AddedDate
NULL | 3 | NULL | FFF | 20/05/2020 18:00:00
3 | NULL | CCC | NULL | 19/05/2020 18:00:00
NULL | 2 | NULL | EEE | 18/05/2020 18:00:00
2 | NULL | BBB | NULL | 17/05/2020 18:00:00
Could someone advice me somehow? I need to do this in EF Core.
I could construct whole list similarly to this and manipulate on this, but this is very far from being optimal:
var results = (new List<BaseEntity>()).Concat(_context.Posts.ToList());
results = results.Concat(_context.Articles.ToList()).ToList();
//order
//take(x)

To achieve the result you need you will have to project both Post and Article to a shared typed, so first I created a ResultDto - which is representation of the type that you want to get:
public class ResultDto
{
public int? ArticleId { get; set; }
public int? PostId { get; set; }
public string Name { get; set; }
public string Title { get; set; }
public DateTime AddedDate { get; set; }
}
Then you have to project Articles to ResultDto, as well as Post to ResultDto, concat the results and sort like this:
var result = _dbContext
.Articles.Select(a => new ResultDto()
{
ArticleId = a.Id,
AddedDate = a.AddedDate,
Name = a.Name,
PostId = null,
Title = null
}).Concat(_dbContext.Posts.Select(p => new ResultDto()
{
PostId = p.Id,
AddedDate = p.AddedDate,
Title = p.Title,
ArticleId = null,
Name = null
}))
.OrderByDescending(r => r.AddedDate)
.ToList();
Btw since this Select is invoked on an IQueryable, you will have to select all of the properties (even if they would be NULL by default).
This results in following SQL query:
SELECT [t].[ArticleId], [t].[AddedDate], [t].[Name], [t].[PostId], [t].[Title]
FROM (
SELECT [a].[Id] AS [ArticleId], [a].[AddedDate], [a].[Name], NULL AS [PostId], NULL AS [Title]
FROM [Articles] AS [a]
UNION ALL
SELECT NULL AS [ArticleId], [p].[AddedDate], NULL AS [Name], [p].[Id] AS [PostId], [p].[Title]
FROM [Posts] AS [p]
) AS [t]
ORDER BY [t].[AddedDate] DESC

Populate object from dataset when having list in list C#

I have this data available from the database:
|----------------|--------------|----------------|----------------|----------------|----------------|
| entity_name | provider_id | provider_name | product_id | product_name | country_name |
|----------------|--------------|----------------|----------------|----------------|----------------|
| test | 123 | Provider1 | 1 | Product1 | Russia |
|----------------|--------------|----------------|----------------|----------------|----------------|
| test | 123 | Provider1 | 2 | Product2 | Spain |
|----------------|--------------|----------------|----------------|----------------|----------------|
| test | 123 | Provider1 | 3 | Product3 | France |
|----------------|--------------|----------------|----------------|----------------|----------------|
| test | 456 | Provider2 | 3 | Product3 | France |
|----------------|--------------|----------------|----------------|----------------|----------------|
| test | 123 | Provider1 | 4 | Product4 | France |
And I have to map it to this model in C#:
public class EntityModel
{
public string EntityName { get; set; }
public List<ProviderModel> Providers { get; set; }
}
public class ProviderModel
{
public int ProviderID { get; set; }
public string ProviderName { get; set; }
public List<ProductModel> Products { get; set; }
public ProviderModel() { }
}
public class ProductModel
{
public int ProductID { get; set; }
public string ProductName { get; set; }
public string CountryName { get; set; }
public ProductModel() { }
}
So basically I have to group by providers, for every provider I have to show the products:
{
"entityName": "string",
"providers": [
{
"ProviderID": 0,
"ProviderName": "string",
"products": [
{
"productId": 0,
"productName": "string",
"countryName": "string"
}
]
}
]
}
The sp that returns the data is like this:
SELECT DISTINCT
e.entity_name
,pro.provider_id
,pro.provider_name
,p.product_id
,p.product_name
,pc.country_name
FROM provider_product_table ppt
INNER JOIN product p ON ppt.product_id = p.product_id
INNER JOIN product_parent pp on pp.product_parent_id=p.product_parent_id
INNER JOIN provider pro ON pro.provider_id = ppt.provider_id
INNER JOIN product_country pc on pc.product_id=p.product_id
INNER JOIN entity e on e.product_parent_id=pp.product_parent_id
WHERE p.product_parent_id = #product_parent_id
ORDER BY p.product_id ASC
I tried a lot of groupBy versions but I get stuck at mapping the second list, the products one.
How can I achieve this? Thank you !

let's start with saying that your database is not well-structured or not created for your need, it will be more than good if you try to seprate your data into three table (entity,provider,product) with relationship between entity and provider , and relationship between your provider and the product table.
however, maybe you are developing new feature that it was not thinking at first to make your code or even your database more extensible.
in this case i need to see your group query request (the groupby ) that you created already.
meanwhile i can imagine your need, so you have several solution.
the first one is what are you trying to do , create several query that fill your classes ,i can also imagine with Redundancy, that because there are some id who are unfortunately duplicated due to your database structure, in this case you need to use DISTINCT or INTERSECT (even if its not your need here ) .
seconds one you can deal with temporary table using the keyword INTO#tmpTable if you are using sqlserver, so basically is to crate a new tables in memory to perform your query ( those tables is what i propose above ).
try to edit your question with those query you describe.

C# Core 2.0 - get data from one model based on data in another model?

I am using .NET Core 2.0
I do have 2 tables:
Movies
ID | Name | AuthorId
----------------------
1 | AAAA | 1
2 | BBBB | 1
3 | CCCC | 1
4 | DDDD | 2
Reviews
ID | FilmId | AuthorId | SomeText
------------------------------------
1 | 1 | 1 | this is horror
2 | 1 | 2 | this is comendy
3 | 3 | 1 | it may be horror
4 | 3 | 2 | it is definitely comedy, not horror
My Models are:
class Movie {
[Key]
public int ID { get; set; }
public string Name { get; set; }
public int AuthorId { get; set; }
public List<Review> Reviews { get; set; }
}
class Review {
[Key]
public int ID { get; set; }
public string SomeText { get; set; }
public int AuthorId { get; set; }
public int FilmId { get; set; }
public Movie Film { get; set; }
}
My Context is:
public class MyContext : DbContext
{
public DbSet<Movie> Movie { get; set; }
public DbSet<Review> Review { get; set; }
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<Movie>().HasMany(m => m.Reviews).WithOne(r => r.Film);
}
}
Now I want to make such query:
SELECT * FROM Movies m JOIN Reviews r ON (m.ID == r.FilmId) WHERE
m.AuthorId = 1 AND (r.AuthorID = 3 OR r.SomeText LIKE '%definitely%');
The expected result should be CCCC.
Currently I do have:
List<Movie> movies = _context.Movie.Include(m => m.Reviews.Select(r => r.AuthorId == 3 || r.SomeText.Contains("definitely"))).ToList();
Howto express above query with this notation?
However I do get:
InvalidOperationException: The property expression 'm => {from Review
r in m.Reviews select (([r].AuthorId == __user_ID_0) OrElse
[r].SomeText.Contains("definitely"))}' is not valid. The expression
should represent a property access: 'r => r.MyProperty'. For more
information on including related data, see
http://go.microsoft.com/fwlink/?LinkID=746393.
I have read this article but it gave me nothing.

I think this linq query returns the correct result (since I cannot test it)
List<Movie> movies = _context.Movie.where(m => m.AuthorId == 1 && m.Reviews.Any(r =>
r.AuthorId == 3 || r.SomeText.Contains("definitely"))).ToList();

Bad performance ASP.NET MVC ViewModel

I need advice about my application.
First of all, I have two models which represent two tables (there are not the real tables)
CUSTOMERS (135K rows | 40 columns)
+-------------+--------+------------+
| CUSTOMER_ID | NAME | FIRST_NAME |
+-------------+--------+------------+
| 1234 | DUPONT | JEAN |
+-------------+--------+------------+
ORDERS
+-------------+----------+-------+
| CUSTOMER_ID | ORDER_ID | TYPE |
+-------------+----------+-------+
| 1234 | 5678 | MEET |
| 1234 | 9105 | CANDY |
| 2568 | 7523 | CANDY |
+-------------+----------+-------+
I want to get a customer with his list of orders.
So I created a Viewmodel :
public class ViewModel
{
public string CustomerID { get; set; }
public string Name { get; set; }
public string FirstName { get; set; }
public List<OrdersViewModel> Orders { get; set; }
public ViewModel(){Orders = new List<OrdersViewModel>();
}
And now the query :
var query = from c in northwind.CUSTOMERS
select new ViewModel()
{
CustomerID = c.CustomerID,
Name = c.Name,
FirstName = c.FirstName
};
var CustomersModels = query.ToList();
var queryOrders = from c in northwind.CUSTOMERS
join o in northwind.ORDERS
on c.CustomerID equals o.CustomerID
select new OrdersViewModel()
{
CustomerID = d.CustomerID,
OrderId= d.OrderId,
Type= d.Type
};
var modelsOrders = queryOrders .ToList();
foreach (ViewModel item in modelsOrders )
{
var listModels = modelsOrders .Where(e => e.PMRQTOTM == item.PMRQTOTM).ToList();
item.Orders = listModels;
}
Is there a better way to realise this?
I get the CustomerModel in less than 30 seconds but for the order it takes several minutes...

If Customer and Order has a relationship, you can use Include in first query.
Even then you should not query 135K all at once. Instead, you want to use Skip and Take based on Page Size and Page Index.

Oy, just create the appropriate indexes on your tables and your queries will fly:
CREATE UNIQUE INDEX CUSTOMERS_CUSTOMER_ID
ON CUSTOMERS (CUSTOMER_ID)
CREATE INDEX ORDERS_CUSTOMER_ID
ON ORDERS (CUSTOMER_ID)