I have this data available from the database:
|----------------|--------------|----------------|----------------|----------------|----------------|
| entity_name | provider_id | provider_name | product_id | product_name | country_name |
|----------------|--------------|----------------|----------------|----------------|----------------|
| test | 123 | Provider1 | 1 | Product1 | Russia |
|----------------|--------------|----------------|----------------|----------------|----------------|
| test | 123 | Provider1 | 2 | Product2 | Spain |
|----------------|--------------|----------------|----------------|----------------|----------------|
| test | 123 | Provider1 | 3 | Product3 | France |
|----------------|--------------|----------------|----------------|----------------|----------------|
| test | 456 | Provider2 | 3 | Product3 | France |
|----------------|--------------|----------------|----------------|----------------|----------------|
| test | 123 | Provider1 | 4 | Product4 | France |
And I have to map it to this model in C#:
public class EntityModel
{
public string EntityName { get; set; }
public List<ProviderModel> Providers { get; set; }
}
public class ProviderModel
{
public int ProviderID { get; set; }
public string ProviderName { get; set; }
public List<ProductModel> Products { get; set; }
public ProviderModel() { }
}
public class ProductModel
{
public int ProductID { get; set; }
public string ProductName { get; set; }
public string CountryName { get; set; }
public ProductModel() { }
}
So basically I have to group by providers, for every provider I have to show the products:
{
"entityName": "string",
"providers": [
{
"ProviderID": 0,
"ProviderName": "string",
"products": [
{
"productId": 0,
"productName": "string",
"countryName": "string"
}
]
}
]
}
The sp that returns the data is like this:
SELECT DISTINCT
e.entity_name
,pro.provider_id
,pro.provider_name
,p.product_id
,p.product_name
,pc.country_name
FROM provider_product_table ppt
INNER JOIN product p ON ppt.product_id = p.product_id
INNER JOIN product_parent pp on pp.product_parent_id=p.product_parent_id
INNER JOIN provider pro ON pro.provider_id = ppt.provider_id
INNER JOIN product_country pc on pc.product_id=p.product_id
INNER JOIN entity e on e.product_parent_id=pp.product_parent_id
WHERE p.product_parent_id = #product_parent_id
ORDER BY p.product_id ASC
I tried a lot of groupBy versions but I get stuck at mapping the second list, the products one.
How can I achieve this? Thank you !
let's start with saying that your database is not well-structured or not created for your need, it will be more than good if you try to seprate your data into three table (entity,provider,product) with relationship between entity and provider , and relationship between your provider and the product table.
however, maybe you are developing new feature that it was not thinking at first to make your code or even your database more extensible.
in this case i need to see your group query request (the groupby ) that you created already.
meanwhile i can imagine your need, so you have several solution.
the first one is what are you trying to do , create several query that fill your classes ,i can also imagine with Redundancy, that because there are some id who are unfortunately duplicated due to your database structure, in this case you need to use DISTINCT or INTERSECT (even if its not your need here ) .
seconds one you can deal with temporary table using the keyword INTO#tmpTable if you are using sqlserver, so basically is to crate a new tables in memory to perform your query ( those tables is what i propose above ).
try to edit your question with those query you describe.
Related
The access patterns that I'm interested in is the last item for a given exchange and an account name.
+-------------------------------+-------------------------------------------+---------+--------------------------------------------+------------+----------+------------+---------+-----------+----------------+--------------------------------------------------------------------+---------------------------+----------------------------------+--------------+------------------------------+
| PK | SK | Account | Address | AddressTag | Exchange | Instrument | Network | Quantity | TransactionFee | TransactionId | TransferDate | TransferId | TransferType | UpdatedAt |
+-------------------------------+-------------------------------------------+---------+--------------------------------------------+------------+----------+------------+---------+-----------+----------------+--------------------------------------------------------------------+---------------------------+----------------------------------+--------------+------------------------------+
| Exchange#Binance#Account#main | TransferDate#12/17/2022 4:59:12 PM +02:00 | main | 0xF76d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null | Binance | USDT | ETH | 97.500139 | 3.2 | 0x46d28f7d0e1e5b1d074a65dcfbb9d90b3bcdc7e6fca6b1f1f7abb5ab219feb24 | 2022-12-17T16:59:12+02:00 | 1b56485f6a3446c3b883f4f485039260 | 0 | 2023-01-28T20:19:59.9181573Z |
| Exchange#Binance#Account#main | TransferDate#12/17/2022 5:38:23 PM +02:00 | main | 0xF76d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null | Binance | USDT | ETH | 3107.4889 | 3.2 | 0xbb2b92030b988a0184ba02e2e754b7a7f0f963c496c4e3473509c6fe6b54a41d | 2022-12-17T17:38:23+02:00 | 4747f6ecc74f4dd8a4b565e0f15bcf79 | 0 | 2023-01-28T20:20:00.4536839Z |
| Exchange#FTX#Account#main | TransferDate#12/17/2021 5:38:23 PM +02:00 | main | 0x476d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null | FTX | USDT | ETH | 20 | 3.2 | 0xaa2b92030b988a0184ba02e2e754b7a7f0f963c496c4e3473509c6fe6b54a41d | 2021-12-17T17:38:23+02:00 | 4747f6ecc74f4dd8a4b565e0f15bcf79 | 0 | 2023-01-28T20:20:00.5723855Z |
| Exchange#FTX#Account#main | TransferDate#12/19/2022 4:59:12 PM +02:00 | main | 0xc46d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null | FTX | USDT | ETH | 15 | 3.2 | 0xddd28f7d0e1e5b1d074a65dcfbb9d90b3bcdc7e6fca6b1f1f7abb5ab219feb24 | 2022-12-19T16:59:12+02:00 | 1b56485f6a3446c3b883f4f485039260 | 0 | 2023-01-28T20:20:00.5207119Z |
+-------------------------------+-------------------------------------------+---------+--------------------------------------------+------------+----------+------------+---------+-----------+----------------+--------------------------------------------------------------------+---------------------------+----------------------------------+--------------+------------------------------+
First of all, it seems to be working as expected but as I'm still learning I'm not so sure whether the partition key and the sort key I chose are good enough or not. This is important as "Uneven distribution of data due to the wrong choice of partition key" can cause reading/writing above the limit issues.
There was a similar example in the documentation and what they say about TransactionId being a partition key is as following:
In most cases you won’t use TransactionID for any query purposes, so you lose the ability to use the partition key to perform a fast lookup of data. To expand this reasoning, consider the traditional order history view on an e-commerce site. Normally orders are retrieved by customer ID or Order ID, not a UID such as a transaction ID that was synthetically generated during checkout. It’s better to choose a natural partition key than generate a synthetic one that won’t be used for querying.
Another interesting part of the documentation is about the composite key
Composite sort keys let you define hierarchical (one-to-many) relationships in your data that you can query at any level of the hierarchy
[country]#[region]#[state]#[county]#[city]#[neighborhood]
This would let you make efficient range queries for a list of locations at any one of these levels of aggregation, from country, to a neighborhood, and everything in between.
I'm also interested in the "Get all user transfers by date range" access pattern but I'm not sure how I could achieve it. So here we are.
C# implementation
public async Task<UserTransferDto?> GetLastAsync(string exchange, string account)
{
var queryRequest = new QueryRequest
{
TableName = TableName,
KeyConditionExpression = "#pk = :pk",
ExpressionAttributeNames = new Dictionary<string, string>
{
{ "#pk", "PK" }
},
ExpressionAttributeValues = new Dictionary<string, AttributeValue>
{
{ ":pk", new AttributeValue { S = $"Exchange#{exchange}#Account#{account}" } }
},
ScanIndexForward = false,
Limit = 1
};
var response = await _dynamoDb.QueryAsync(queryRequest);
if (response.Items.Count == 0)
{
return null;
}
var itemAsDocument = Document.FromAttributeMap(response.Items[0]);
return JsonSerializer.Deserialize<UserTransferDto>(itemAsDocument.ToJson());;
}
public class UserTransferDto
{
[JsonPropertyName("PK")]
public string Pk => $"Exchange#{Exchange}#Account#{Account}";
[JsonPropertyName("SK")]
public string Sk => $"TransferDate#{TransferDate}";
public required string Exchange { get; init; }
public required string Account { get; init; }
public required DateTimeOffset TransferDate { get; init; }
public required string TransferId { get; init; }
public required TransferType TransferType { get; init; }
public required string Instrument { get; init; }
public required string Network { get; init; }
public required decimal Quantity { get; init; }
public required string Address { get; init; }
public string? AddressTag { get; init; }
public decimal? TransactionFee { get; init; }
public string? TransactionId { get; init; }
public DateTime UpdatedAt { get; set; }
}
public enum TransferType
{
Withdraw = 0,
Deposit = 1
}
Sources:
https://youtu.be/HaEPXoXVf2k?t=720
https://youtu.be/HaEPXoXVf2k?t=798
Hierarchical Data Structures as Items https://youtu.be/HaEPXoXVf2k?t=2775
Access Patterns https://youtu.be/HaEPXoXVf2k?t=2903
https://aws.amazon.com/blogs/database/choosing-the-right-dynamodb-partition-key/
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-sort-keys.html
Your base table design works well for getting the latest item for a given exchange and account (via a Query with that as the PK and getting the last sortable from the SK), except that you’re using non-sortable human time stamps instead of sortable time stamps. You should use 2023-01-28 12:56:08 and so on so that the times sort right as strings.
For the other query to find the latest across all exchanges and accounts, you can create a GSI which has a singular PK and the times as the SK. Just beware that you’re limited in how many writes per second you can do to the same PK. Above 1,000 write units per second you’ll need to shard it and then do a query for each shard to get the latest per shard and then the latest overall.
This is a pattern described in https://youtu.be/0iGR8GnIItQ
I have 2 entities: Post and Article and I need to select both of them at once and sort by common property. I'm searching for some "join" possibilities, but all I'm finding are examples ending with something like ...where a.id == b.a_id which is not relevant in my case.
That's my models
class BaseEntity {
public int Id;
public DateTime AddedDate;
}
class Post extends BaseEntity {
public string Title;
}
class Article extends BaseEntity {
public string Name;
}
For that data:
Articles:
Id | Name | AddedDate
1 | AAA | 15/05/2020 18:00:00
2 | BBB | 17/05/2020 18:00:00
3 | CCC | 19/05/2020 18:00:00
Posts:
Id | Title | AddedDate
1 | DDD | 16/05/2020 18:00:00
2 | EEE | 18/05/2020 18:00:00
3 | FFF | 20/05/2020 18:00:00
when sorting by date DESC and limit it to 4 i need to end up with something like that, or something similar:
Article Id | PostId | Name | Title | AddedDate
NULL | 3 | NULL | FFF | 20/05/2020 18:00:00
3 | NULL | CCC | NULL | 19/05/2020 18:00:00
NULL | 2 | NULL | EEE | 18/05/2020 18:00:00
2 | NULL | BBB | NULL | 17/05/2020 18:00:00
Could someone advice me somehow? I need to do this in EF Core.
I could construct whole list similarly to this and manipulate on this, but this is very far from being optimal:
var results = (new List<BaseEntity>()).Concat(_context.Posts.ToList());
results = results.Concat(_context.Articles.ToList()).ToList();
//order
//take(x)
To achieve the result you need you will have to project both Post and Article to a shared typed, so first I created a ResultDto - which is representation of the type that you want to get:
public class ResultDto
{
public int? ArticleId { get; set; }
public int? PostId { get; set; }
public string Name { get; set; }
public string Title { get; set; }
public DateTime AddedDate { get; set; }
}
Then you have to project Articles to ResultDto, as well as Post to ResultDto, concat the results and sort like this:
var result = _dbContext
.Articles.Select(a => new ResultDto()
{
ArticleId = a.Id,
AddedDate = a.AddedDate,
Name = a.Name,
PostId = null,
Title = null
}).Concat(_dbContext.Posts.Select(p => new ResultDto()
{
PostId = p.Id,
AddedDate = p.AddedDate,
Title = p.Title,
ArticleId = null,
Name = null
}))
.OrderByDescending(r => r.AddedDate)
.ToList();
Btw since this Select is invoked on an IQueryable, you will have to select all of the properties (even if they would be NULL by default).
This results in following SQL query:
SELECT [t].[ArticleId], [t].[AddedDate], [t].[Name], [t].[PostId], [t].[Title]
FROM (
SELECT [a].[Id] AS [ArticleId], [a].[AddedDate], [a].[Name], NULL AS [PostId], NULL AS [Title]
FROM [Articles] AS [a]
UNION ALL
SELECT NULL AS [ArticleId], [p].[AddedDate], NULL AS [Name], [p].[Id] AS [PostId], [p].[Title]
FROM [Posts] AS [p]
) AS [t]
ORDER BY [t].[AddedDate] DESC
I have a SQL Server four tables 'mapping' table MappingACVP as:
Author_ID | CoAuthor_ID | Venue_ID | Paper_ID | Year
----------------------------------------------------
677 | 42700 | 64309 | 812229 | 2005
677 | 42700 | 64309 | 812486 | 2005
677 | 42700 | 64309 | 818273 | 2005
677 | 42700 | 65182 | 812229 | 2005
... | ... | ... | ... | ...
... | ... | ... | ... | ...
By using Entity Framework, I got auto-generated code for all four tables included in Entity Framework Model i.e. Authors.cs, CoAuthors_New.cs, Venues_New.cs and Papers_New.cs as well, whereas tables other than Authors all have Author_ID as foreign key as a relationship.
The auto-generated code for class Authors.cs is as:
public partial class Authors
{
public Authors()
{
this.CoAuthors_New = new HashSet<CoAuthors_New>();
this.Papers_New = new HashSet<Papers_New>();
this.Venues_New = new HashSet<Venues_New>();
}
public int id { get; set; }
public int Author_ID { get; set; }
public string Author_Name { get; set; }
public virtual ICollection<CoAuthors_New> CoAuthors_New { get; set; }
public virtual ICollection<Papers_New> Papers_New { get; set; }
public virtual ICollection<Venues_New> Venues_New { get; set; }
}
Now if I want to declare a list as:
List<Authors> _eAthors = new List<Authors>();
and want to fetch values for filling this list from database (using MappingACVP table), whereas there is no Year property in Authors.cs class. So,
How will I fill eAuthors list from database?
I need advice about my application.
First of all, I have two models which represent two tables (there are not the real tables)
CUSTOMERS (135K rows | 40 columns)
+-------------+--------+------------+
| CUSTOMER_ID | NAME | FIRST_NAME |
+-------------+--------+------------+
| 1234 | DUPONT | JEAN |
+-------------+--------+------------+
ORDERS
+-------------+----------+-------+
| CUSTOMER_ID | ORDER_ID | TYPE |
+-------------+----------+-------+
| 1234 | 5678 | MEET |
| 1234 | 9105 | CANDY |
| 2568 | 7523 | CANDY |
+-------------+----------+-------+
I want to get a customer with his list of orders.
So I created a Viewmodel :
public class ViewModel
{
public string CustomerID { get; set; }
public string Name { get; set; }
public string FirstName { get; set; }
public List<OrdersViewModel> Orders { get; set; }
public ViewModel(){Orders = new List<OrdersViewModel>();
}
And now the query :
var query = from c in northwind.CUSTOMERS
select new ViewModel()
{
CustomerID = c.CustomerID,
Name = c.Name,
FirstName = c.FirstName
};
var CustomersModels = query.ToList();
var queryOrders = from c in northwind.CUSTOMERS
join o in northwind.ORDERS
on c.CustomerID equals o.CustomerID
select new OrdersViewModel()
{
CustomerID = d.CustomerID,
OrderId= d.OrderId,
Type= d.Type
};
var modelsOrders = queryOrders .ToList();
foreach (ViewModel item in modelsOrders )
{
var listModels = modelsOrders .Where(e => e.PMRQTOTM == item.PMRQTOTM).ToList();
item.Orders = listModels;
}
Is there a better way to realise this?
I get the CustomerModel in less than 30 seconds but for the order it takes several minutes...
If Customer and Order has a relationship, you can use Include in first query.
Even then you should not query 135K all at once. Instead, you want to use Skip and Take based on Page Size and Page Index.
Oy, just create the appropriate indexes on your tables and your queries will fly:
CREATE UNIQUE INDEX CUSTOMERS_CUSTOMER_ID
ON CUSTOMERS (CUSTOMER_ID)
CREATE INDEX ORDERS_CUSTOMER_ID
ON ORDERS (CUSTOMER_ID)
In my C# application (an ASP.NET MVC5 website) I am using Entity Framework 6 as my ORM and I'm using code-first with an existing database. The database is SQL Server 2012 Standard.
In my database I have a view (dbo.vBoardReportLayouts) that functions as a joining table for a number of different elements. I've created an entity for this view and configured the navigation properties.
When I attempt to use this entity with LINQ entity framework generates a SQL query that tries to use a table that doesn't actually exist. Specifically it is looking for the BoardReportLayoutPages table.
Now for all the gory details...
The vBoardReportLayouts view looks like this:
| ReportID | PageTypeId | PageId | LayoutID |
+----------+------------+--------+----------+
| 1 | 1 | 1 | 1 |
| 1 | 1 | 1 | 2 |
| 1 | 1 | 2 | 3 |
| 1 | 1 | 3 | 4 |
| 1 | 2 | 4 | 5 |
| 1 | 2 | 4 | 6 |
| 2 | 1 | 1 | 1 |
| 2 | 1 | 1 | 2 |
| 2 | 1 | 2 | 3 |
| 2 | 1 | 3 | 4 |
| 2 | 2 | 4 | 5 |
| 2 | 2 | 4 | 6 |
The BoardReportLayouts entity class:
[Table("vBoardReportLayouts")]
public partial class BoardReportLayout {
public BoardReportLayout() {
BoardReports = new HashSet<BoardReport>();
PageTypes = new HashSet<PageType>();
Pages = new HashSet<Page>();
PageLayouts = new HashSet<PageLayout>();
}
[Key]
public int BoardReportLayoutId { get; set; }
public int BoardReportId { get; set; }
public int PageTypeId { get; set; }
public int PageId { get; set; }
public int PageLayoutId { get; set; }
public virtual ICollection<BoardReport> BoardReports { get; set; }
public virtual ICollection<PageType> PageTypes { get; set; }
public virtual ICollection<Page> Pages { get; set; }
public virtual ICollection<PageLayout> PageLayouts { get; set; }
}
From the OnModelCreating() method of the dbContext:
modelBuilder.Entity<BoardReportLayout>()
.HasMany(e => e.BoardReports)
.WithMany(e => e.BoardReportLayouts);
modelBuilder.Entity<BoardReportLayout>()
.HasMany(e => e.PageTypes)
.WithMany(e => e.BoardReportLayouts);
modelBuilder.Entity<BoardReportLayout>()
.HasMany(e => e.Pages)
.WithMany(e => e.BoardReportLayouts);
modelBuilder.Entity<BoardReportLayout>()
.HasMany(e => e.PageLayouts)
.WithMany(e => e.BoardReportLayouts);
The BoardReport, PageType, Page, and PageLayout entities all have the following property defined:
public virtual ICollection<BoardReportLayout> BoardReportLayouts { get; set; }
Executing this LINQ query:
var test = (from l in ppdb.BoardReportLayouts
where l.BoardReportId == 8
select l.Pages).SelectMany(p => p).ToList();
Generates this SQL:
SELECT [Join1].[PageId] AS [PageId],
[Join1].[PageTypeId] AS [PageTypeId],
[Join1].[Name] AS [Name],
[Join1].[ParameterValue] AS [ParameterValue],
[Join1].[SsrsPageParamterValue] AS [SsrsPageParamterValue],
[Join1].[TableOfContentsOrder] AS [TableOfContentsOrder]
FROM [dbo].[vBoardReportLayouts] AS [Extent1]
INNER JOIN (
SELECT [Extent2].[BoardReportLayout_BoardReportLayoutId] AS [BoardReportLayout_BoardReportLayoutId],
[Extent3].[PageId] AS [PageId],
[Extent3].[PageTypeId] AS [PageTypeId],
[Extent3].[Name] AS [Name],
[Extent3].[ParameterValue] AS [ParameterValue],
[Extent3].[SsrsPageParamterValue] AS [SsrsPageParamterValue],
[Extent3].[TableOfContentsOrder] AS [TableOfContentsOrder]
FROM [dbo].[BoardReportLayoutPages] AS [Extent2]
INNER JOIN [dbo].[Pages] AS [Extent3] ON [Extent3].[PageId] = [Extent2].[Page_PageId]
) AS [Join1] ON [Extent1].[BoardReportLayoutId] = [Join1].[BoardReportLayout_BoardReportLayoutId]
WHERE 8 = [Extent1].[BoardReportId]
The problem is in the FROM clause of the [Join1] table expression. I'm not sure why it is trying to use the table dbo.BoardReportLayoutPages since it could join directly from the view to the Pages table. The table dbo.BoardReportLayoutPages doesn't exist so this throws an exception.
This bit of code is specifying that the BoardReportLayout has a many to many relationship with pages:
modelBuilder.Entity<BoardReportLayout>()
.HasMany(e => e.Pages)
.WithMany(e => e.BoardReportLayouts);
That means there should be a BoardReportLayoutPages table to model that relationship. The BoardReportLayout is supported by a view, so you had to write the sql to create it yourself, because code first would have generated tables to support the model you have defined here. If you do have a many-to-many relationship here, then you should have created the BoardReportLayoutPages as a table or view also.
But you also have a PageId property in the BoardReportLayout. This means that there is a second one-to-many relationship between a Page and BoardReportLayouts. If you want to get to a BoardReportLayout's Page via this relationship then you should add a public Page Page { get; set; } navigation property.
Only you can decide whether you have two relationships here, or just one, and what type of relationship(s) you have.