I'm working on a small ASP.NET Core project for tagging images using Entity Framework Core on a Sqlite database, mainly just for learning. There are two tables (and POCOs), Tags and Images, where multiple tags are related to each image. I'm trying to get a count of all Images that have tags associated with them.
In plain SQL I'd write SELECT COUNT(DISTINCT ImageId) FROM Tags to get the count, and in LINQ I came up with _context.Tags.Select(t => t.Image).Distinct().Count(). But that LINQ query appears to cause EF-Core to join the two tables, return all of the rows, and then do the Distinct and Count in code.
I tried to do _context.Tags.FromSql("SELECT COUNT(DISTINCT ImageId) FROM Tags"), but because that query only returns the count the call fails because EF can't map the result to a Tag. I also tried to use _context.Database.FromSql<int>, but wasn't able to find any real documentation on it and there doesn't seem to be IntelliSense for it.
What I have done for now is what's detailed in the "ADO.NET" section of this blog post from Eric Anderson:
int count;
using (var connection = _context.Database.GetDbConnection())
{
connection.Open();
using (var command = connection.CreateCommand())
{
command.CommandText = "SELECT COUNT(DISTINCT ImageId) FROM Tags";
string result = command.ExecuteScalar().ToString();
int.TryParse(result, out count);
}
}
But is that the best way to go about getting the count efficiently?
Edit: Here's the query that EF is putting in the Debug output:
SELECT "t"."TagId", "t"."Content", "t"."ImageId", "t.Image"."ImageId", "t.Image"."FileName", "t.Image"."Path", "t.Image"."Url"
FROM "Tags" AS "t"
LEFT JOIN "Images" AS "t.Image" ON "t"."ImageId" = "t.Image"."ImageId"
ORDER BY "t"."ImageId"
As of now, you can't define an ad-hoc result.
Good news is that it's currently on the backlog: https://github.com/aspnet/EntityFramework/issues/1862
In the meantime, here's an extension method that would work:
public static int IntFromSQL(this ApplicationDbContext context, string sql )
{
int count;
using (var connection = context.Database.GetDbConnection())
{
connection.Open();
using (var command = connection.CreateCommand())
{
command.CommandText = sql;
string result = command.ExecuteScalar().ToString();
int.TryParse(result, out count);
}
}
return count;
}
Usage:
int result = _context.IntFromSQL("SELECT COUNT(DISTINCT ImageId) FROM Tags");
Your original line of code should have done exactly what you wanted. It would also be recommended over inline SQL.
_context.Tags.Select(t => t.Image).Distinct().Count()
Are you sure that this called the database for the two tables and then queried them in memory? If you observed this behaviour while debugging then it's possible that your inspection caused the IQueryable to enumerate which would call the database using a different query than it would have otherwise.
A way to check the actual query, without breaking into the running code, is by using the MyLoggerProvider from the Entity Framework Core documentation.
https://docs.efproject.net/en/latest/miscellaneous/logging.html?highlight=logging
Once the logger is registered in the code then any SQL query ran against the server will be displayed in the console window and/or in the file c:\temp\log.txt.
The following log message was generated when using a Distinct() and a Count() on the database tables of the website example.
SELECT COUNT(*)
FROM (
SELECT DISTINCT [a.Blog].[BlogId], [a.Blog].[Url]
FROM [Posts] AS [a]
INNER JOIN [Blogs] AS [a.Blog] ON [a].[BlogId] = [a.Blog].[BlogId]
) AS [t]Closing connection to database '***' on server 'tcp:**************'.
Finally, since you do not need any of the properties on the t.Image then it seems that you should be using a Where() rather than a Select().
Related
I have the following raw query being executed through Entity Framework via a SqlQuery call on the DbSet:
public IEnumerable<TEntity> GetComplicatedData<TEntity>()
{
return database
.Set<TEntity>()
.SqlQuery("SELECT * FROM <Complicated Query Here>");
}
...
var count = GetComplicatedData<Item>()
.Where(f => f.OwnerID == 5)
.Count();
This works, but is very slow due to the fact that SqlQuery executes immediately without the Where getting applied.
Is there any way to call SqlQuery in such a way that the Where gets applied server-side?
Essentially, I want Entity Framework to generate a store query like:
SELECT
<Columns>
FROM
(
SELECT * FROM <Complicated Query Here>
) a
WHERE a.OwnerID = 5
Or, is there a way to translate my where expression into a query that I can append manually (a.k.a, without manually writing a SQL query, the where clause is not always that simple)?
This cannot be done with LINQ methods, because "raw" queries are of type DbRawSqlQuery<TElement>, which lacks support required for "composing" dynamic queries with Where.
If you are interested in counting items, you could work around the issue by taking a condition:
public int CountComplicatedData<TEntity>(Func<TEntity,bool> condition) {
return database
.Set<TEntity>()
.SqlQuery("SELECT * FROM <Complicated Query Here>")
.Count(condition);
}
SQL Server provides output for inserted and updated record with the 'inserted' keyword.
I have a table representing a processing queue. I use the following query to lock a record and get the ID of the locked record:
UPDATE TOP (1) GlobalTrans
SET LockDateTime = GETUTCDATE()
OUTPUT inserted.ID
WHERE LockDateTime IS NULL
This will output a column named ID with all the updated record IDs (a single ID in my case). How can I translate this into EF in C# to execute the update and get the ID back?
Entity Framework has no way of doing that.
You could do it the ORM way, by selecting all the records, setting their LockDateTime and writing them back. That probably is not safe for what you want to do because by default it's not one single transaction.
You can span your own transactions and use RepeatableRead as isolation level. That should work. Depending on what your database does in the background, it might be overkill though.
You could write the SQL by hand. That defeats the purpose of entity framework, but it should be just as safe as it was before as far as the locking mechanism is concerned.
You could also put it into a stored procedure and call that. It's a little bit better than the above version because at least somebody will compile it and check that the table and column names are correct.
Simple Example #1 to get a data table:
I did this directly against the connection:
Changed the command.ExecuteNonQuery() to command.ExecuteReader()
var connection = DbContext().Database.Connection as SqlConnection;
using (var command = connection.CreateCommand())
{
command.CommandText = sql;
command.CommandTimeout = 120;
command.Parameters.Add(param);
using (var reader = command.ExecuteReader())
{
var resultTable = new DataTable();
resultTable.Load(reader);
return resultTable;
}
}
FYI, If you don't have an OUTPUT clause in your SQL, it will return an empty data table.
Example #2 to return entities:
This is a bit more complicated but does work.
using a SQL statement with a OUTPUT inserted.*
var className = typeof(T).Name;
var container = ObjContext().MetadataWorkspace.GetEntityContainer(UnitOfWork.ObjContext().DefaultContainerName, DataSpace.CSpace);
var setName = (from meta in container.BaseEntitySets where meta.ElementType.Name == className select meta.Name).First();
var results = ObjContext().ExecuteStoreQuery<T>(sql, setName, trackingEnabled ? MergeOption.AppendOnly : MergeOption.NoTracking).ToList();
T being the entity being worked on
I'm using MiniProfiler to profile my sql commands.
One issue I'm dealing with now is repeated INSERT statements generated by linq.
I've converted them into a SqlBulkCopy command, however now it doesn't appear to record it in the sql view in MiniProfiler.
Would there even be an associated command string for a SqlBulkCopy?
Is it possible to get the bulk copy to appear in the list of sql commands?
Can I at least make it counted in the % sql bit?
I'm aware I could use MiniProfiler.Current.Step("Doing Bulk Copy") but that wouldn't count as SQL, and wouldn't show in the listing with any detail.
Current code below:
public static void BulkInsertAll<T>(this DataContext dc, IEnumerable<T> entities)
{
var conn = (dc.Connection as ProfiledDbConnection).InnerConnection as SqlConnection;
conn.Open();
Type t = typeof(T);
var tableAttribute = (TableAttribute)t.GetCustomAttributes(
typeof(TableAttribute), false).Single();
var bulkCopy = new SqlBulkCopy(conn)
{
DestinationTableName = tableAttribute.Name
};
//....
bulkCopy.WriteToServer(table);
}
You should be able to use CustomTimings to profile these. These are included in the new v3 version that is now available on nuget.
You can see some example usages of CustomTiming in the sample project where this is used to record http and redis events.
An example of how you could use it with SqlBulkCopy:
string sql = GetBulkCopySql(); // what should show up for the SqlBulkCopy event?
using (MiniProfiler.Current.CustomTiming("SqlBulkCopy", sql))
{
RunSqlBulkCopy(); // run the actual SqlBulkCopy operation
}
I followed this answer,
How can I supply a List<int> to a SQL parameter?
Please see these questions of mine for understanding scenario,
How can I update Crate IDs of List of Fruits in single SQL query in c#
how can i update SQL table logic
What I am trying and not working
private void relate_fruit_crate(List<string> selectedFruitIDs, int selectedCrateID)
{
string updateStatement = "UPDATE relate_fruit_crate set CrateID = #selectedCrateID where FruitID = #selectedFruitIDs";
using (SqlConnection connection = new SqlConnection(ConnectionString()))
using (SqlCommand cmd = new SqlCommand(updateStatement, connection))
{
connection.Open();
cmd.Parameters.Add(new SqlParameter("#selectedCrateID", selectedCrateID.ToString()));
cmd.Parameters.Add(new SqlParameter("#selectedFruitIDs", String.Join(",",selectedFruitIDs.ToArray())));
cmd.ExecuteNonQuery();
}
}
My code runs without any error,
You need to use the IN keyword in your scenario. The problem is that the SqlCommand.Parameters pattern does not build the query itself, but calls a stored procedure on the database:
exec sp_executesql N'UPDATE relate_fruit_crate set CrateID = #selectedCrateID where FruitID in(''#selectedFruitIDs'')', N'#selectedCrateID nvarchar(1),#selectedFruitIDs nvarchar(5)', #selectedCrateID = N'1', #selectedFruitIDs = N'1,2'
This will not work as the array is escaped.
The workaround would be to either use a normal StringBuilder to create the query. (Warning! SQL Injection) or to call the query for each ID separately.
Maybe there's a way to do this with the SqlCommand.Parameters, but I could not find one.
OLD POST::
string updateStatement = "UPDATE relate_fruit_crate set CrateID IN ('#selectedCrateID') where FruitID = '#selectedFruitIDs'";
[....]
cmd.Parameters.Add(new SqlParameter("#selectedFruitIDs", String.Join("','",selectedFruitIDs.ToArray())));
and equals (=) query will only match a single value.
Multi-value parameter queries are a bit of a pain in TSQL. There are options like table-valued parameters, or "split" UDFs - otherwise... it is a bit tricky. You end up having to add multiple parameters (depending on the data), and change the query to suit. If I may suggest... a library like "dapper" may help you here - it is designed to make scenarios like this easy:
using Dapper; // at the top of your code file, to enable dapper
...
private void relate_fruit_crate(List<string> selectedFruitIDs, int selectedCrateID)
{
// note the slightly unusual "in" here (no paranethesis) - that is because
// dapper is going to do some voodoo...
using (SqlConnection connection = new SqlConnection(ConnectionString()))
{
connection.Open();
connection.Execute(
"UPDATE relate_fruit_crate set CrateID = #selectedCrateID where FruitID in #selectedFruitIDs",
new { selectedFruitIDs, selectedCrateID });
}
}
here "dapper" does all the work of figuring out how to express that in using multiple parameters, adding the correct number of parameters. It is also just much easier (in particular, look at how little work we did with commands and parameters; it handles readers nicely too).
Dapper is freely available from NuGet
I'm using Dapper Extensions for some of my types and it works really well for most use cases. I've run into a case where I have a many-many relationship, and I want to do something like:-
SELECT id,a,b,c FROM Foo WHERE Foo.id in (SELECT foo_id FROM foo-bar WHERE bar-id=#bar_id)
Obviously Dapper Extensions can handle "SELECT id,a,b,c FROM Foo" but not the latter part. I could do a select to get the list of Foo id's I want, and then pass that to Dapper Extensions but that's less efficient.
The part I can't do with plain Dapper is get the SELECT column list automatically, so what I'd really like is a way to:-
Get the SELECT column list from Dapper Extension's internal mechanisms
Get the basic "SELECT id,a,b,c FROM Foo" from Dapper Extension's internal mechanisms
Hook Dapper Extensions Get code so that I can add a custom WHERE clause
I've looked at the code, but I can't spot how to do any of these things. Can anyone help? I've worked around by using plain Dapper and "SELECT * ..." at the moment, but I'm sure there's a better way.
Here is another option:
You can create a View:
select * from Foo
join FooBar b
on a.foo_id = b.foo_id
Then use predicates to select with any where clause:
using (SqlConnection cn = new SqlConnection(_connectionString))
{
cn.Open();
var predicate = Predicates.Field<Foo>(f => f.foo_id, Operator.Eq, 1);
IEnumerable<Foo> list = cn.GetList<Foo>(predicate);
cn.Close();
}
Generated SQL should look something like:
SELECT
[Foo].[foo_id]
, [Foo].[...]
, [Foo].[...]
, [Foo].[...]
, [Foo].[...]
FROM [ViewName]
WHERE ([ViewName].[foo_id] = #foo_id_0)
I did not know this was not supported back in 2012. So about 1.7K views in two years and not much exposure. But in case someone new to Dapper landed here and wondering if it works, the answer is, it is working. Using the latest version of, as of this writing, Dapper v1.42 from nuget:
var sql = "SELECT id,a,b,c FROM Foo WHERE Foo.id in (
SELECT foo_id FROM foo-bar WHERE bar-id=#bar_id)"
using (var cn = new SqlConnection(the_connection_string)) {
cn.Open();
var returnedObject = cn.Query<dynamic>(sql, new { bar_id = some_value });
}