Develop Reference

Getting multiple entities using Azure TableStorage over multiple partitions

I'm using Azure.Data.Tables (12.6.1) and I need to query a single record from multiple partitions of a single table (so the result would be multiple records, 1 from each partition). Each entity needs to be looked up by its partition key and row key - for a single TableClient.GetEntity() call this would be a point query.
After reading the documentation I'm confused if it's efficient or not to call TableClient.QueryAsync() with multiple partition key / row key pairs and the search results I found provide contradicting suggestions.
Is it efficient to do this (for a number of partition key / row key combinations, up to ~50) or is it just better to call GetEntity() one by one, for each entity?
var filter = "(PartitionKey eq 'p1' And RowKey eq 'r1') Or " +
"(PartitionKey eq 'p2' And RowKey eq 'r2') Or ...";
var results = await tableClient.QueryAsync(filter, 500, null, cancelToken);

I don't know if there is a definitive answer here as it probably depends on your specific requirements. I would suggest testing different options and tune accordingly.
Just for reference, here is a general reference about query performance for tables https://learn.microsoft.com/azure/storage/tables/table-storage-design-for-query

I settled on parallelizing point queries for this scenario, and has given good results. I have heavy-burst read scenarios, I may have 10's/100's of 1000's of lookups to do against 100's of millions of records). I prefer that over a query with a series of OR's, as those were tending to give worse throughput (I don't have any stats to hand now....)
For me parallelization happens through 2 means:
lower level: awaiting a batch of Tasks, each making an individual point query
higher level: architecting a particularly heavy workload to scale out over multiple instances, each making parallel queries via 1)

Firestore order of ORDER, LIMIT, WHERE clauses

Assume we have a dataset like this:
Name
Population
Capital
London
8,799,800
true
Barcelona
1,620,343
false
Luxembourg City
128,512
true
When working with SQL databases, the order in which you typed your ORDER, LIMIT and WHERE clauses is important. For instance:
SELECT *
FROM Cities
WHERE Capital = true
ORDER BY Population DESC
LIMIT 2
would return London and Luxembourg City. Whereas
SELECT *
FROM Cities
ORDER BY Population DESC
LIMIT 2
WHERE Capital = true
would return London
(the second statement is probably illegal but that's a different thing)
In Firestore (I'm using the C# SDK), we can set the query arguments and then get the snapshot. We could do:
Query query = citiesRef
.WhereEqualTo("Capital", true)
.OrderByDescending("Population")
.Limit(2);
QuerySnapshot querySnapshot = await query.GetSnapshotAsync();
We could also run this:
Query query = citiesRef
.OrderByDescending("Population")
.Limit(2)
.WhereEqualTo("Capital", true);
QuerySnapshot querySnapshot = await query.GetSnapshotAsync();
If Firestore was to behave in the way that a SQL database does, these 2 statements would potentially return different results. However, when we run them, they seem to return the same results.
Are the two Firestore queries equivalent?

Yes, they are the same. When you specify a limit, you are always limiting the final set of results, after all filters and orders, that would be returned to the client so that the results can be paginated by the client. It is never limiting the data set before any other operations are applied.

LINQ or SQL: Group by with a sum of distinct values

I have a database table, on SQL Server 2019, containing a time series of prices collected with multiple frequencies (daily, weekly or monthly) which I query using EF Core 3.1
I'm trying to extract these prices, aggregated by month, but without losing the information of the collection frequency.
From the following set of data:
I'm trying to get this one, which contains the aggregate average value of the prices, grouped by Month, and with the frequencies of the raw records.
These could be easily solved by using
string.Join(",",s.Select(innerSel=>innerSel.OriginalFrequency).Distinct())
but unfortunately, I can't use as I need to work on IQueryable objects and run the execution of the LINQ query only at the end when I take a subset of data, based on the page-size, because converting to a List the query before grouping means to get several thousands of records from the DB.
I was trying to use a combination of SUM and COUNT of the frequencies in order to easily understand which is the original combination by multiplication these two values (see the schema below) but the COUNT and SUM should count only distinct values, otherwise, it doesn't work.
Is there a way to not lose this information in some way, without overloading the database server requesting unnecessary data, or making multiple requests?
This is the code where I'm stuck:
var aggregatedMonthlyPrices = prices.GroupBy(g => new
{
g.DateMonth,
g.DateYear
}).Select(s => new
{
DateMonth = s.Key.DateMonth,
DateYear = s.Key.DateYear
Price=s.Average(avg=>avg.Price),
FrequencySum= s.Sum(sum=>sum.DataCollectionFrequencyId),
FrequencyCount = s.Count(),
});

Entity Framework DbContext filtered query for count is extremely slow using a variable

Using an ADO.NET entity data model I've constructed two queries below against a table containing 1800 records that has just over 30 fields that yield staggering results.
// Executes slowly, over 6000 ms
int count = context.viewCustomers.AsNoTracking()
.Where(c => c.Cust_ID == _custID).Count();
// Executes instantly, under 20 ms
int count = context.viewCustomers.AsNoTracking()
.Where(c => c.Cust_ID == 625).Count();
I see from the database log that Entity Framework provides that the queries are almost identical except that the filter portion uses a parameter. Copying this query into SSMS and declaring & setting this parameter there results in a near instant query so it doesn't appear to be on the database end of things.
Has anyone encountered this that can explain what's happening? I'm at the mercy of a third party control that adds this command to the query in an attempt to limit the number of rows returned, getting the count is a must. This is used for several queries so a generic solution is needed. It is unfortunate it doesn't work as advertised, it seems to only make the query take 5-10 times as long as it would if I just loaded the entire view into memory. When no filter is used however, it works like a dream.
Use of these components includes the source code so I can change this behavior but need to consider which approaches can be used to provide a reusable solution.

You did not mention about design details of your model but if you only want to have count of records based on condition, then this can be optimized by only counting the result set based on one column. For example,
int count = context.viewCustomers.AsNoTracking().Where(c => c.Cust_ID == _custID).Count();
If you design have 10 columns, and based on above statement let say 100 records have been returned, then against every record result set contains 10 columns' data which is of not use.
You can optimize this by only counting result set based on single column.
int count = context.viewCustomers.AsNoTracking().Where(c => c.Cust_ID == _custID).Select(x=>new {x.column}).Count();
Other optimization methods, like using async variants of count CountAsync can be used.

Performance issues to iterate results with C# SQLite DataReader and attached database

I am using System.Data.SQLite and SQLiteDataReader in my C# project. I am facing performance issues when getting the results of a query with attached databases.
Here is an example of a query to search text into two databases :
ATTACH "db2.db" as db2;
SELECT MainRecord.RecordID,
((LENGTH(MainRecord.Value) - LENGTH(REPLACE(UPPER(MainRecord.Value), UPPER("FirstValueToSearch"), ""))) / 18) AS "FirstResultNumber",
((LENGTH(DB2Record.Value) - LENGTH(REPLACE(UPPER(DB2Record.Value), UPPER("SecondValueToSearch"), ""))) / 19) AS "SecondResultNumber"
FROM main.Record MainRecord
JOIN db2.Record DB2Record ON DB2Record.RecordID BETWEEN (MainRecord.PositionMin) AND (MainRecord.PositionMax)
WHERE FirstResultNumber > 0 AND SecondResultNumber > 0;
DETACH db2;
When executing this query with SQLiteStudio or SQLiteAdmin, this works fine, I am getting the results in a few seconds (the Record table can contain hundreds of thousands of records, the query returns 36000 records).
When executing this query in my C# project, the execution takes a few seconds too, but it takes hours to run through all the results.
Here is my code :
// Attach databases
SQLiteDataReader data = null;
using (SQLiteCommand command = this.m_connection.CreateCommand())
{
command.CommandText = "SELECT...";
data = command.ExecuteReader();
}
if (data.HasRows)
{
while (data.Read())
{
// Do nothing, just iterate all results
}
}
data.Close();
// Detach databases
Calling the Read method of the SQLiteDataReader once can take more than 10 seconds ! I guess this is because the SQLiteDataReader is lazy loaded (and so it doesn't return the whole rowset before reading the results), am I right ?
EDIT 1 :
I don't know if this has something to do with lazy loading, like I said initially, but all I want is being able to get ALL the results as soon as the query is ended. Isn't it possible ? In my opinion, this is really strange that it takes hours to get results of a query executed in few seconds...
EDIT 2 :
I just added a COUNT(*) in my select query in order to see if I could get the total number of results at the first data.Read(), just to be sure that it was only the iteration of the results that was taking so long. And I was wrong : this new request executes in few seconds in SQLiteAdmin / SQLiteStudio, but takes hours to execute in my C# project. Any idea why the same query is so much longer to execute in my C# project?
EDIT 3 :
Thanks to EXPLAIN QUERY PLAN, I noticed that there was a slight difference in the execution plan for the same query between SQLiteAdmin / SQLiteStudio and my C# project. In the second case, it is using an AUTOMATIC PARTIAL COVERING INDEX on DB2Record instead of using the primary key index. Is there a way to ignore / disable the use of automatic partial covering indexes? I know it is used to speed up the queries, but in my case, it's rather the opposite that happens...
Thank you.

Besides finding matching records, it seems that you're also counting the number of times the strings matched. The result of this count is also used in the WHERE clause.
You want the number of matches, but the number of matches does not matter in the WHERE clause - you could try change the WHERE clause to:
WHERE MainRecord.Value LIKE '%FirstValueToSearch%' AND DB2Record.Value LIKE '%SecondValueToSearch%'
It might not result in any difference though - especially if there's no index on the Value columns - but worth a shot. Indexes on text columns require alot of space, so I wouldn't blindly recommend that.
If you haven't done so yet, place an index on the DB2's RecordID column.
You can use EXPLAIN QUERY PLAN SELECT ... to make SQLite spit out what it does to try to make your query perform, the output of that might help diagnose the problem.

Are you sure you use the same version of sqlite in System.Data.SQLite, SQLiteStudio and SQLiteAdmin ?
You can have huge differences.

One more typical reason why SQL query can take different amount of time when executed with ADO.NET and from native utility (like SQLiteAdmin) are command parameters used in CommandText (it is not clear from your code whether parameters are used or not). Depending on ADO.NET provider implementation the following identical CommandText values:
SELECT * FROM sometable WHERE somefield = ? // assume parameter is '2'
and
SELECT * FROM sometable WHERE somefield='2'
may lead to absolutely different execution plan and query performance.
Another suggestion: you may disable journal (specifying "Journal Mode=off;" in the connection string) and synchronous mode ("Synchronous=off;") as these options also may affect query performance in some cases.