Develop Reference

Execute custom SQL before running FromSqlRaw in Entity Framework Core 6 or above in SQL Server

I only need it to work for SQL Server. This is an example. The question is about a general approach.
There is a nice extension method from https://entityframework-extensions.net called WhereBulkContains. It is, sort of, great, except that the code of the methods in this library is obfuscated and they do not produce valid SQL when .ToQueryString() is called on IQueryable<T> with these extension methods applied.
Subsequently, I can't use such methods in production code as I am not "allowed" to trust such code due to business reasons. Sure, I can write tons of tests to ensure that WhereBulkContains works as expected, except that there are some complicated cases where the performance of WhereBulkContains is well below stellar, whereas properly written SQL works in a blink of an eye. And (read above), since the code of this library is obfuscated, there is no way to figure out what's wrong there without spending a large amount of time. We would've bought the license (as this is not a freeware) if the library weren't obfuscated. All together that basically kills the library for our purposes.
This is where it gets interesting. I can easily create and populate a temporary table, e.g. (I have a table called EFAgents with an int PK called AgentId in the database):
private string GetTmpAgentSql(IEnumerable<int> agentIds) => #$"
drop table if exists #tmp_Agents;
create table #tmp_Agents (AgentId int not null, primary key clustered (AgentId asc));
{(agentIds
.Chunk(1_000)
.Select(e => $#"
insert into #tmp_Agents (AgentId)
values
({e.JoinStrings("), (")});
")
.JoinStrings(""))}
select 0 as Result
";
private const string AgentSql = #"
select a.* from EFAgents a inner join #tmp_Agents t on a.AgentID = t.AgentId";
where GetContext returns EF Core database context and JoinStrings comes from Unity.Interception.Utilities and then use it as follows:
private async Task<List<EFAgent>> GetAgents(List<int> agentIds)
{
var tmpSql = GetTmpAgentSql(agentIds);
using var ctx = GetContext();
// This creates a temporary table and populates it with the ids.
// This is a proprietary port of EF SqlQuery code, but I can post the whole thing if necessary.
var _ = await ctx.GetDatabase().SqlQuery<int>(tmpSql).FirstOrDefaultAsync();
// There is a DbSet<EFAgent> called Agents.
var query = ctx.Agents
.FromSqlRaw(AgentSql)
.Join(ctx.Agents, t => t.AgentId, a => a.AgentId, (t, a) => a);
var sql = query.ToQueryString() + Environment.NewLine;
// This should provide a valid SQL; https://entityframework-extensions.net does NOT!
// WriteLine - writes to console or as requested. This is irrelevant to the question.
WriteLine(sql);
var result = await query.ToListAsync();
return result;
}
So, basically, I can do what I need in two steps:
using var ctx = GetContext();
// 1. Create a temp table and populate it - call GetTmpAgentSql.
...
// 2. Build the join starting from `FromSqlRaw` as in example above.
This is doable, half-manual, and it is going to work.
The question is how to do that in one step, e.g., call:
.WhereMyBulkContains(aListOfIdConstraints, whateverElseIsneeded, ...)
and that's all.
I am fine if I need to pass more than one parameter in each case in order to specify the constraints.
To clarify the reasons why do I need to go into all these troubles. We have to interact with a third party database. We don't have any control of the schema and data there. The database is large and poorly designed. That resulted in some ugly EFC LINQ queries. To remedy that, some of that ugliness was encapsulated into a method, which takes IQueryable<T> (and some more parameters) and returns IQueryable<T>. Under the hood this method calls WhereBulkContains. I need to replace this WhereBulkContains by, call it, WhereMyBulkContains, which would be able to provide correct ToQueryString representation (for debugging purposes) and be performant. The latter means that SQL should not contain in clause with hundreds (and even sometimes thousands) of elements. Using inner join with a [temp] table with a PK and having an index on the FK field seem to do the trick if I do that in pure SQL. But, ... I need to do that in C# and effectively in between two LINQ method calls. Refactoring everything is also not an option because that method is used in many places.
Thanks a lot!

I think you really want to use a Table Valued Parameter.
Creating an SqlParameter from an enumeration is a little fiddly, but not too difficult to get right;
CREATE TYPE [IntValue] AS TABLE (
Id int NULL
)
private IEnumerable<SqlDataRecord> FromValues(IEnumerable<int> values)
{
var meta = new SqlMetaData(
"Id",
SqlDbType.Int
);
foreach(var value in values)
{
var record = new SqlDataRecord(
meta
);
record.SetInt32(0, value);
yield return record;
}
}
public SqlParameter ToIntTVP(IEnumerable<int> values){
return new SqlParameter()
{
TypeName = "IntValue",
SqlDbType = SqlDbType.Structured,
Value = FromValues(values)
};
}
Personally I would define a query type in EF Core to represent the TVP. Then you can use raw sql to return an IQueryable.
public class IntValue
{
public int Id { get; set; }
}
modelBuilder.Entity<IntValue>(e =>
{
e.HasNoKey();
e.ToView("IntValue");
});
IQueryable<IntValue> ToIntQueryable(DbContext ctx, IEnumerable<int> values)
{
return ctx.Set<IntValue>()
.FromSqlInterpolated($"select * from {ToIntTVP(values)}");
}
Now you can compose the rest of your query using Linq.
var ids = ToIntQueryable(ctx, agentIds);
var query = ctx.Agents
.Where(a => ids.Any(i => i.Id == a.Id));

I would propose to use linq2db.EntityFrameworkCore (note that I'm one of the creators). It has built-in temporary tables support.
We can create simple and reusable function which filters records of any type:
public static class HelperMethods
{
private class KeyHolder<T>
{
[PrimaryKey]
public T Key { get; set; } = default!;
}
public static async Task<List<TEntity>> GetRecordsByIds<TEntity, TKey>(this IQueryable<TEntity> query, IEnumerable<TKey> ids, Expression<Func<TEntity, TKey>> keyFunc)
{
var ctx = LinqToDBForEFTools.GetCurrentContext(query) ??
throw new InvalidOperationException("Query should be EF Core query");
// based on DbContext options, extension retrieves connection information
using var db = ctx.CreateLinqToDbConnection();
// create temporary table and BulkCopy records into that table
using var tempTable = await db.CreateTempTableAsync(ids.Select(id => new KeyHolder<TKey> { Key = id }), tableName: "temporaryIds");
var resultQuery = query.Join(tempTable, keyFunc, t => t.Key, (q, t) => q);
// we use ToListAsyncLinqToDB to avoid collission with EF Core async methods.
return await resultQuery.ToListAsyncLinqToDB();
}
}
Then we can rewrite your function GetAgents to the following:
private async Task<List<EFAgent>> GetAgents(List<int> agentIds)
{
using var ctx = GetContext();
var result = await ctx.Agents.GetRecordsByIds(agentIds, a => a.AgentId);
return result;
}

C# Dynamic LINQ Contains using an array only evaluates the first value

Relatively new to C# and started using Dynamic LINQ to filter a data table adapter using a string. The issue I'm having is that the Where clause only seems to evaluate the first value in the string and none of the others. Here is the code I am using.
string[] ids = new string[] { "12345", "67891", "45878" };
var resultQ = (from pr in table1 select pr).AsQueryable();
var iq = resultQ.Where("#0.Contains(FieldName)", ids);
It works but only for the first value "12345" so the output of iq displays all fields for "12345". I tried using linq.dynamic.core to see if that would help but the still same result (or I haven't used it properly). I know I'm probably missing something minor here but any help would be greatly appreciated.
Also on a separate note: I wanted to convert the end result of iq which is a IQueryable type to EnumerationRowCollection type. Is this possible?
Thanks in advance

Managed to fix both points now. Either set string[] to string for dynamic LINQ to get all values in list as coded below
string ids = "12345,67891,45878";
var resultQ = (from pr in table1 select pr).AsQueryable();
var iq = resultQ.Where("#0.Contains(FieldName)", ids);
or use Syed's suggestion and change the LINQ query and keep the array (thanks again Seyed)
For the conversion of IQueryable type to EnumerationRowCollection I changed EnumerationRowCollection to IEnumerable and this worked for all my LINQ queries
Thanks

Just use this. It will works fine.
string[] ids = new string[] { "12345", "67891", "45878" };
var resultQ = (from pr in table1 select pr).AsQueryable();
var iq = resultQ.Where(w => ids.Contains(w.Id)).ToList();

Get raw sql string in Sql Kata

I am using SqlKata purely to build sql queries in C#. I am wanting to take the output of my built up Query, get the raw (compiled) sql string, and execute it against SQL.
I thought this would do it:
var factory = new QueryFactory(null, new SqlServerCompiler());
var query = new Query();
...
var sqlText = factory.Compiler.Compile(query).Sql;
But this gives this:
SELECT TOP (#p0) [AllStarFull].[GameNumber], [AllStarFull].[LeagueId], [AllStarFull].[PlayedInGame] FROM [AllStarFull]
This throws an exception because (#p0) is a param, and not the actual value.
In the documentation, it mentions to bring in Logger but I don't really need logging functionality (right now).
https://sqlkata.com/docs/execution/logging
var db = new QueryFactory(connection, new SqlServerCompiler());
// Log the compiled query to the console
db.Logger = compiled => {
Console.WriteLine(compiled.ToString());
};
var users = db.Query("Users").Get();
Is there anyway to get the raw sql string from the Query with all params populated?

If you need just to build the SQL there is no need to include the SqlKata.Execution package (which is include the QueryFactory class).
The simplest way is:
using SqlKata;
using SqlKata.Compilers;
// Create an instance of SQLServer
var compiler = new SqlServerCompiler();
var query = new Query("Users").Where("Id", 1).Where("Status", "Active");
SqlResult result = compiler.Compile(query);
string sql = result.Sql;
List<object> bindings = result.Bindings; // [ 1, "Active" ]
as mentioned in the docs you can use the result.ToString() to get the full query
var sql = result.ToString();
but this is not a good practice, the correct way is to use the parameterized query with bindings to execute it.
taken from https://sqlkata.com/docs#compile-only-example

If you are injecting QueryFactory dependency, you can use its compiler:
var query = new Query("Users")... etc
var rawSQL = queryFactory.Compiler.Compile(query).RawSql;

How can I pass array as a sql query param for cosmos DB query

I want to pass array as a param to SqlQuerySpec to be able to use it in the IN expression when building query for azure cosmos db.
What i'm trying to do is something like we do with regular (string, int etc) params:
private SqlQuerySpec BuildQuery(IEnumerable<string> exclTypes)
{
var queryText = "SELECT * FROM root r WHERE r.Type NOT IN (#types)";
var parameters = new SqlParameterCollection{new SqlParameter("#types", exclTypes.ToArray())};
return new SqlQuerySpec()
{QueryText = queryText, Parameters = parameters};
}
But that doesn't work in such way. Any other ways I can pass array as a param?
Thanks.

Your query should look something like this:
SELECT * FROM root r WHERE ARRAY_CONTAINS(#types, r.Type) <> true
then you can pass #types as array and check if that array contains value you have in property r.Type in your document.
refs:
https://learn.microsoft.com/en-us/azure/cosmos-db/sql-api-sql-query-reference#bk_array_contains
https://github.com/Azure/azure-documentdb-node/issues/156

The easiest way to do this tisto set up a table-valued parameter. Your array would be passed as a TVP and since it is then a table it can be used as part of an IN predicate. There's lots of material online about that.

C# Entity Framework text SQL query wrapper

I have a problem similar to this:
How to retrieve multiple columns from non-entity type sql query?
I need to implement the method string[,] DirectQuery(string sqlText, string[] param) which is basically a C# equivalent of SQL Server Management Studio.
The user is supposed to enter a SQL query as string text (+ string parameters to avoid SQL injection) and receive back a string matrix containing the outcome of the query.
Internally, I'm using Entity Framework.
Here's my implementation:
public string[,] DirectQuery(string sqlQuery, string[] param)
{
//discover how many fields are specified in the select clause
string ip = sqlQuery.ToLower().Split(new string[] { "from" }, StringSplitOptions.None)[0];
int cols = ip.Count(y => y == ',') + 1;
//execute the query
DbRawSqlQuery<string> res = param != null ? _context.Database.SqlQuery<string>(sqlQuery, param) : _context.Database.SqlQuery<string>(sqlQuery);
//wrap everything in a matrix to return
return res.ToArray().Array2Matrix(res.ToArray().Length /cols, cols);
}
where
public static T[,] Array2Matrix<T>(this T[] flat, int rows, int cols) where T : class
is my custom method that turns flat arrays into rows x cols matrices.
If in the select clause users specify a single attribute, that works fine, but in case of 2+ fields needed the execution of DirectQuery fires the runtime exception dbrawsqlquery he data reader has more than one field. Multiple fields are not valid for EDM primitive or enumeration types. That's completely reasonable, but since the query can be whatever I can't create a custom class to wrap every possible outcome.
What do you suggest?

The problem is that you're using a method - DbRawSqlQuery which must be told what type to expect, and you're telling it to expect just a string, so it has no idea what to do with more than one returned column.
Maybe it would work if you specified string[] or IEnumerable<string> or something? Alternatively you could define a series of objects with 1, 2, 3, 4 etc values and detect the number of items at runtime and use the correct class... but that seems absurd.
Really though, I'd suggest NOT using EF as someone suggested above. Find something which can return dynmamic objects, OR just use ADO.Net directly.