Develop Reference

DynamoDB Data Modeling - Hierarchical Data Structures as items

The access patterns that I'm interested in is the last item for a given exchange and an account name.
+-------------------------------+-------------------------------------------+---------+--------------------------------------------+------------+----------+------------+---------+-----------+----------------+--------------------------------------------------------------------+---------------------------+----------------------------------+--------------+------------------------------+
| PK | SK | Account | Address | AddressTag | Exchange | Instrument | Network | Quantity | TransactionFee | TransactionId | TransferDate | TransferId | TransferType | UpdatedAt |
+-------------------------------+-------------------------------------------+---------+--------------------------------------------+------------+----------+------------+---------+-----------+----------------+--------------------------------------------------------------------+---------------------------+----------------------------------+--------------+------------------------------+
| Exchange#Binance#Account#main | TransferDate#12/17/2022 4:59:12 PM +02:00 | main | 0xF76d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null | Binance | USDT | ETH | 97.500139 | 3.2 | 0x46d28f7d0e1e5b1d074a65dcfbb9d90b3bcdc7e6fca6b1f1f7abb5ab219feb24 | 2022-12-17T16:59:12+02:00 | 1b56485f6a3446c3b883f4f485039260 | 0 | 2023-01-28T20:19:59.9181573Z |
| Exchange#Binance#Account#main | TransferDate#12/17/2022 5:38:23 PM +02:00 | main | 0xF76d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null | Binance | USDT | ETH | 3107.4889 | 3.2 | 0xbb2b92030b988a0184ba02e2e754b7a7f0f963c496c4e3473509c6fe6b54a41d | 2022-12-17T17:38:23+02:00 | 4747f6ecc74f4dd8a4b565e0f15bcf79 | 0 | 2023-01-28T20:20:00.4536839Z |
| Exchange#FTX#Account#main | TransferDate#12/17/2021 5:38:23 PM +02:00 | main | 0x476d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null | FTX | USDT | ETH | 20 | 3.2 | 0xaa2b92030b988a0184ba02e2e754b7a7f0f963c496c4e3473509c6fe6b54a41d | 2021-12-17T17:38:23+02:00 | 4747f6ecc74f4dd8a4b565e0f15bcf79 | 0 | 2023-01-28T20:20:00.5723855Z |
| Exchange#FTX#Account#main | TransferDate#12/19/2022 4:59:12 PM +02:00 | main | 0xc46d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null | FTX | USDT | ETH | 15 | 3.2 | 0xddd28f7d0e1e5b1d074a65dcfbb9d90b3bcdc7e6fca6b1f1f7abb5ab219feb24 | 2022-12-19T16:59:12+02:00 | 1b56485f6a3446c3b883f4f485039260 | 0 | 2023-01-28T20:20:00.5207119Z |
+-------------------------------+-------------------------------------------+---------+--------------------------------------------+------------+----------+------------+---------+-----------+----------------+--------------------------------------------------------------------+---------------------------+----------------------------------+--------------+------------------------------+
First of all, it seems to be working as expected but as I'm still learning I'm not so sure whether the partition key and the sort key I chose are good enough or not. This is important as "Uneven distribution of data due to the wrong choice of partition key" can cause reading/writing above the limit issues.
There was a similar example in the documentation and what they say about TransactionId being a partition key is as following:
In most cases you won’t use TransactionID for any query purposes, so you lose the ability to use the partition key to perform a fast lookup of data. To expand this reasoning, consider the traditional order history view on an e-commerce site. Normally orders are retrieved by customer ID or Order ID, not a UID such as a transaction ID that was synthetically generated during checkout. It’s better to choose a natural partition key than generate a synthetic one that won’t be used for querying.
Another interesting part of the documentation is about the composite key
Composite sort keys let you define hierarchical (one-to-many) relationships in your data that you can query at any level of the hierarchy
[country]#[region]#[state]#[county]#[city]#[neighborhood]
This would let you make efficient range queries for a list of locations at any one of these levels of aggregation, from country, to a neighborhood, and everything in between.
I'm also interested in the "Get all user transfers by date range" access pattern but I'm not sure how I could achieve it. So here we are.
C# implementation
public async Task<UserTransferDto?> GetLastAsync(string exchange, string account)
{
var queryRequest = new QueryRequest
{
TableName = TableName,
KeyConditionExpression = "#pk = :pk",
ExpressionAttributeNames = new Dictionary<string, string>
{
{ "#pk", "PK" }
},
ExpressionAttributeValues = new Dictionary<string, AttributeValue>
{
{ ":pk", new AttributeValue { S = $"Exchange#{exchange}#Account#{account}" } }
},
ScanIndexForward = false,
Limit = 1
};
var response = await _dynamoDb.QueryAsync(queryRequest);
if (response.Items.Count == 0)
{
return null;
}
var itemAsDocument = Document.FromAttributeMap(response.Items[0]);
return JsonSerializer.Deserialize<UserTransferDto>(itemAsDocument.ToJson());;
}
public class UserTransferDto
{
[JsonPropertyName("PK")]
public string Pk => $"Exchange#{Exchange}#Account#{Account}";
[JsonPropertyName("SK")]
public string Sk => $"TransferDate#{TransferDate}";
public required string Exchange { get; init; }
public required string Account { get; init; }
public required DateTimeOffset TransferDate { get; init; }
public required string TransferId { get; init; }
public required TransferType TransferType { get; init; }
public required string Instrument { get; init; }
public required string Network { get; init; }
public required decimal Quantity { get; init; }
public required string Address { get; init; }
public string? AddressTag { get; init; }
public decimal? TransactionFee { get; init; }
public string? TransactionId { get; init; }
public DateTime UpdatedAt { get; set; }
}
public enum TransferType
{
Withdraw = 0,
Deposit = 1
}
Sources:
https://youtu.be/HaEPXoXVf2k?t=720
https://youtu.be/HaEPXoXVf2k?t=798
Hierarchical Data Structures as Items https://youtu.be/HaEPXoXVf2k?t=2775
Access Patterns https://youtu.be/HaEPXoXVf2k?t=2903
https://aws.amazon.com/blogs/database/choosing-the-right-dynamodb-partition-key/
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-sort-keys.html

Your base table design works well for getting the latest item for a given exchange and account (via a Query with that as the PK and getting the last sortable from the SK), except that you’re using non-sortable human time stamps instead of sortable time stamps. You should use 2023-01-28 12:56:08 and so on so that the times sort right as strings.
For the other query to find the latest across all exchanges and accounts, you can create a GSI which has a singular PK and the times as the SK. Just beware that you’re limited in how many writes per second you can do to the same PK. Above 1,000 write units per second you’ll need to shard it and then do a query for each shard to get the latest per shard and then the latest overall.
This is a pattern described in https://youtu.be/0iGR8GnIItQ

How to sort my IQueryable hierarchically in a flat list?

I have some flat list that I want to sort in a pseudo-hierarchical order. Without materialization using .ToList(), only IQueryable.
I do the following:
public class DictionaryElement
{
[Key]
public Guid ID { get; set; }
[ForeignKey("ParentDictionaryElement")]
public Guid? ParentDictionaryElementID { get; set; }
public DictionaryElement ParentDictionaryElement { get; set; }
public ICollection<DictionaryElement> DictionaryElements { get; set; }
public String Value { get; set; }
}
public IQueryable<DictionaryElement> Foo(DbContext db)
{
IQueryable<DictionaryElement> allElementsQuery = db.DictionaryElements;
//... allElementsQuery.OrderBy(); ?
return allElementsQuery;
}
and data:
+----+---------------------------+
| ID | ParentDictionaryElementID |
+----+---------------------------+
| 1 | null |
| 2 | null |
| 3 | 1 |
| 4 | null |
| 6 | 3 |
| 5 | 2 |
+----+---------------------------+
I want to get IQueryable which will receive such data:
+----+---------------------------+
| ID | ParentDictionaryElementID |
+----+---------------------------+
| 1 | null |
| 3 | 1 |
| 6 | 3 |
| 2 | null |
| 5 | 2 |
| 4 | null |
+----+---------------------------+
The difference is that the second table is a hierarchy but in a flat, flat view, as it would have looked at normal display.
That is, the children immediately follow the parent
ID 1
ID 3 (parentID 1)
ID 6 (parentID 3)
ID 2
ID 5 (parentID 2)
ID 4

try this?
return allElementsQuery.OrderBy(m => m).AsQueryable();

LINQ GroupBy for multiple Column

I am stuck in logic to group by. I have a model that has all information but I have to group information according to CustomerBuildingMapping
public class TicketsDataModel
{
public string BuildingID { get; set; }
public string Ticket { get; set; }
public string Amount { get; set; }
public string CustomerID { get; set; }
public string BuildingName { get; set; }
}
Current Data
BuildingID | Ticket | Amount |CustomerID | BuildingName
10 | 001 | 50 | 1 | JP Building
11 | 002 | 45 | 1 | Tiskon
52 | 452 | 35 | 2 | Lalit
65 | 568 | 78 | 2 | Tuilp
41 | 121 | 12 | 1 | BK Trp
-
public class CustomerBuildingMapping
{
public long LeadID { get; set; }
public string CustomerID { get; set; }
public List<BuildingInfo> BuildingInfo{ get; set; }
}
public class BuildingInfo
{
public string BuildingID { get; set; }
public string TicketNumber { get; set; }
public long Amount { get; set; }
public string BuildingName { get; set; }
}
Expected Data after group by
LeadID 1001
CustomerID 1
BuildingInfo
BuildingID | Ticket | Amount | BuildingName
10 | 001 | 50 | JP Building
11 | 002 | 45 | Tiskon
41 | 121 | 12 | BK Trp
LeadID 1002
CustomerID 2
BuildingInfo
BuildingID | Ticket | Amount | BuildingName
52 | 452 | 35 | Lalit
65 | 568 | 78 | Tulip
I have written this code but not able to group by for multiple columns.
List<CustomerBuildingMapping> objCustomerBuildingMappingResult = objTicketsForTheDayInfo.TicketsForTheDay.GroupBy(l => l.CustomerID).Select(grp => new CustomerBuildingMapping
{
CustomerID = grp.Key,
//BuildingInfo = grp.Select(l => l.BuildingID).ToList(),
}).ToList();

You do not need to group by multiple columns. Based on sample data you are only grouping by one field, CustomerID.
var objCustomerBuildingMappingResult = objTicketsForTheDayInfo.TicketsForTheDay
.GroupBy(l => l.CustomerID)
.Select(grp => new CustomerBuildingMapping
{
CustomerID = grp.Key,
LeadId = long.Parse(grp.Key) + 1000,
BuildingInfo = grp.Select(l => new BuildingInfo {
BuildingID = l.BuildingID,
TicketNumber = l.Ticket,
Amount = l.Amount,
BuildingName = l.BuildingName
}).ToList(),
}).ToList();

As Nkosi has pointed out, in OP's example, because LeadId is an automatically generated surrogate key, there's no need to GroupBy on anything other than a single key field (CustomerID), and make a function to generate the surrogate LeadId.
However, in the more general case, if a composite key needs to be constructed for a GroupBy, then both types of Tuples (System.Tuple on older versions of C#, and System.ValueTuple in C#7 and later) make for good transient grouping keys when used with .GroupBy and .ToDictionary. This is because Tuples internally build the tuple instance's HashCode by combining the underlying Hashcodes of the contained types.
For the older System.Tuple, you need to deal with the ugly Itemx properties:
var objCustomerBuildingMappingResult = objTicketsForTheDayInfo
.TicketsForTheDay
.GroupBy(l => Tuple.Create(l.CustomerID, l.BuildingID))
.Select(grp => new CustomerBuildingMapping
{
CustomerID = grp.Key.Item1,
BuildingId = grp.Key.Item2,
// ...
})
.ToList();
But this is more readable (and performant) with System.ValueTuple:
var objCustomerBuildingMappingResult = objTicketsForTheDayInfo
.TicketsForTheDay
.GroupBy(l => (CustomerId: l.CustomerID, BuildingId: l.BuildingID))
.Select(grp => new CustomerBuildingMapping
{
CustomerID = grp.Key.CustomerId,
BuildingId = grp.Key.BuildingID,
// ...

WPF Datagrid dynamic columns from model property

Using WPF datagrid with MVVM structure (no code behind). I was wondering if it was possible to take a model property that normally shows up in a single column and use that to dynamically generate new columns.
Example
public Class1
{
public int Property1 { get; set; }
public string Property2 { get; set; }
public double Property3 { get; set; }
}
Mock up standard datagrid
|Property1|Property2|Property3|
| 1 | A | 1.25 |
| 2 | B | 2 |
| 3 | B | 3 |
Mock up desired datagrid
|Property1| A | B |
| 1 | 1.25 | |
| 2 | | 2 |
| 3 | | 3 |
There is no limit on what value Property2 can be.

Entity / Code First / SQL Database - Same parameter but differents values

Context
I'm a beginner in Entity / CodeFirst and SQL database.
I would to build a table with the same parameter:
-----------------------------------------
| ID | F1 | F2 | F3 | F4 | F5 | F6 | F7 |
-----------------------------------------
| 1 |1000| 500| 250| 100| 50 | | |
| 2 | 500| 250| 125| | | | |
| 3 | 250| 125| 100| | | | |
| 4 | 200| 100| | | | | |
| 5 | 100| 50 | | | | | |
-----------------------------------------
Fs1 to 7 is a list the same parameter with different value.
Questions
What's the correct way to describe it in taking into Code First consideration?
I suppose this way isn't correct...
public class FsTable
{
public int Id { get; set; }
public double F1 { get; set; }
public double F2 { get; set; }
public double F3 { get; set; }
public double F4 { get; set; }
public double F5 { get; set; }
public double F6 { get; set; }
public double F7 { get; set; }
}
What's the correct way to fill this table with values above?

First of all think about colums without value. Should they be nullable or not? You can seed your Db from the Seed method.
If you are using console aplication you need to invoke some operation over the database so you can create it
Ps. Check out Repository Pattern