Develop Reference

Execute custom SQL before running FromSqlRaw in Entity Framework Core 6 or above in SQL Server

I only need it to work for SQL Server. This is an example. The question is about a general approach.
There is a nice extension method from https://entityframework-extensions.net called WhereBulkContains. It is, sort of, great, except that the code of the methods in this library is obfuscated and they do not produce valid SQL when .ToQueryString() is called on IQueryable<T> with these extension methods applied.
Subsequently, I can't use such methods in production code as I am not "allowed" to trust such code due to business reasons. Sure, I can write tons of tests to ensure that WhereBulkContains works as expected, except that there are some complicated cases where the performance of WhereBulkContains is well below stellar, whereas properly written SQL works in a blink of an eye. And (read above), since the code of this library is obfuscated, there is no way to figure out what's wrong there without spending a large amount of time. We would've bought the license (as this is not a freeware) if the library weren't obfuscated. All together that basically kills the library for our purposes.
This is where it gets interesting. I can easily create and populate a temporary table, e.g. (I have a table called EFAgents with an int PK called AgentId in the database):
private string GetTmpAgentSql(IEnumerable<int> agentIds) => #$"
drop table if exists #tmp_Agents;
create table #tmp_Agents (AgentId int not null, primary key clustered (AgentId asc));
{(agentIds
.Chunk(1_000)
.Select(e => $#"
insert into #tmp_Agents (AgentId)
values
({e.JoinStrings("), (")});
")
.JoinStrings(""))}
select 0 as Result
";
private const string AgentSql = #"
select a.* from EFAgents a inner join #tmp_Agents t on a.AgentID = t.AgentId";
where GetContext returns EF Core database context and JoinStrings comes from Unity.Interception.Utilities and then use it as follows:
private async Task<List<EFAgent>> GetAgents(List<int> agentIds)
{
var tmpSql = GetTmpAgentSql(agentIds);
using var ctx = GetContext();
// This creates a temporary table and populates it with the ids.
// This is a proprietary port of EF SqlQuery code, but I can post the whole thing if necessary.
var _ = await ctx.GetDatabase().SqlQuery<int>(tmpSql).FirstOrDefaultAsync();
// There is a DbSet<EFAgent> called Agents.
var query = ctx.Agents
.FromSqlRaw(AgentSql)
.Join(ctx.Agents, t => t.AgentId, a => a.AgentId, (t, a) => a);
var sql = query.ToQueryString() + Environment.NewLine;
// This should provide a valid SQL; https://entityframework-extensions.net does NOT!
// WriteLine - writes to console or as requested. This is irrelevant to the question.
WriteLine(sql);
var result = await query.ToListAsync();
return result;
}
So, basically, I can do what I need in two steps:
using var ctx = GetContext();
// 1. Create a temp table and populate it - call GetTmpAgentSql.
...
// 2. Build the join starting from `FromSqlRaw` as in example above.
This is doable, half-manual, and it is going to work.
The question is how to do that in one step, e.g., call:
.WhereMyBulkContains(aListOfIdConstraints, whateverElseIsneeded, ...)
and that's all.
I am fine if I need to pass more than one parameter in each case in order to specify the constraints.
To clarify the reasons why do I need to go into all these troubles. We have to interact with a third party database. We don't have any control of the schema and data there. The database is large and poorly designed. That resulted in some ugly EFC LINQ queries. To remedy that, some of that ugliness was encapsulated into a method, which takes IQueryable<T> (and some more parameters) and returns IQueryable<T>. Under the hood this method calls WhereBulkContains. I need to replace this WhereBulkContains by, call it, WhereMyBulkContains, which would be able to provide correct ToQueryString representation (for debugging purposes) and be performant. The latter means that SQL should not contain in clause with hundreds (and even sometimes thousands) of elements. Using inner join with a [temp] table with a PK and having an index on the FK field seem to do the trick if I do that in pure SQL. But, ... I need to do that in C# and effectively in between two LINQ method calls. Refactoring everything is also not an option because that method is used in many places.
Thanks a lot!

I think you really want to use a Table Valued Parameter.
Creating an SqlParameter from an enumeration is a little fiddly, but not too difficult to get right;
CREATE TYPE [IntValue] AS TABLE (
Id int NULL
)
private IEnumerable<SqlDataRecord> FromValues(IEnumerable<int> values)
{
var meta = new SqlMetaData(
"Id",
SqlDbType.Int
);
foreach(var value in values)
{
var record = new SqlDataRecord(
meta
);
record.SetInt32(0, value);
yield return record;
}
}
public SqlParameter ToIntTVP(IEnumerable<int> values){
return new SqlParameter()
{
TypeName = "IntValue",
SqlDbType = SqlDbType.Structured,
Value = FromValues(values)
};
}
Personally I would define a query type in EF Core to represent the TVP. Then you can use raw sql to return an IQueryable.
public class IntValue
{
public int Id { get; set; }
}
modelBuilder.Entity<IntValue>(e =>
{
e.HasNoKey();
e.ToView("IntValue");
});
IQueryable<IntValue> ToIntQueryable(DbContext ctx, IEnumerable<int> values)
{
return ctx.Set<IntValue>()
.FromSqlInterpolated($"select * from {ToIntTVP(values)}");
}
Now you can compose the rest of your query using Linq.
var ids = ToIntQueryable(ctx, agentIds);
var query = ctx.Agents
.Where(a => ids.Any(i => i.Id == a.Id));

I would propose to use linq2db.EntityFrameworkCore (note that I'm one of the creators). It has built-in temporary tables support.
We can create simple and reusable function which filters records of any type:
public static class HelperMethods
{
private class KeyHolder<T>
{
[PrimaryKey]
public T Key { get; set; } = default!;
}
public static async Task<List<TEntity>> GetRecordsByIds<TEntity, TKey>(this IQueryable<TEntity> query, IEnumerable<TKey> ids, Expression<Func<TEntity, TKey>> keyFunc)
{
var ctx = LinqToDBForEFTools.GetCurrentContext(query) ??
throw new InvalidOperationException("Query should be EF Core query");
// based on DbContext options, extension retrieves connection information
using var db = ctx.CreateLinqToDbConnection();
// create temporary table and BulkCopy records into that table
using var tempTable = await db.CreateTempTableAsync(ids.Select(id => new KeyHolder<TKey> { Key = id }), tableName: "temporaryIds");
var resultQuery = query.Join(tempTable, keyFunc, t => t.Key, (q, t) => q);
// we use ToListAsyncLinqToDB to avoid collission with EF Core async methods.
return await resultQuery.ToListAsyncLinqToDB();
}
}
Then we can rewrite your function GetAgents to the following:
private async Task<List<EFAgent>> GetAgents(List<int> agentIds)
{
using var ctx = GetContext();
var result = await ctx.Agents.GetRecordsByIds(agentIds, a => a.AgentId);
return result;
}

Entity Framework not populating key

I'm importing data from a delimited file using LINQ and Entity Framework and in one scenario EF is not populating the key property. The class is:
public class BallastType
{
public int BallastTypeId { get; set; }
public string Name { get; set; }
}
The file contains a header row and a row for each entry. The code to import it is:
var baseDir = AppDomain.CurrentDomain.BaseDirectory;
var seedDataPath = Path.Combine(baseDir, "bin", "SeedData");
var ballastTypesFile = Path.Combine(seedDataPath, "BallastTypes.txt");
var ballastTypes = (from l in File.ReadAllLines(ballastTypesFile).Skip(1)
let x = l.Split('\t').ToArray()
select new BallastType()
{
Name = x[0]
});
context.BallastTypes.AddRange(ballastTypes);
context.SaveChanges();
Running this code inserts all the entities but BallastTypeId is uninitialized (0). If I add ToList after the select...
...select new BallastType()
{
Name = x[0]
}).ToList();
it works as expected and BallastTypeId is populated. One note is that I'm checking this at a breakpoint.
Why does adding ToList cause EF to work as expected? I'm using EF 6.1.3.

Without a .ToList(), ballastTypes is an IQueryable, and is only evaluated when you enumerate it. If you check the values of BallastTypeId after SaveChanges, you are in essence re-running your LINQ statement, and re-reading the contents of the file. This means that the entities you read when looking for the inserted ID are not the ones you inserted into the context.
If you add a .ToList(), then ballastTypes becomes a list instead of an IQueryable, and can be evaluated multiple times.

Calling a SQL User-defined function in a LINQ query

I am having a hard time getting this to work. I am trying to do a radius search using the following Filter helper on an IQueryable. There are a set of other filters that get applied before RadiusSearch applies. The order shouldn't really matter since the goal is to get the query to be deferred until a ToList() operation.
public static IQueryable<ApiSearchCommunity> RadiusSearch(this IQueryable<ApiSearchCommunity> communities)
{
var centerLatitude = 30.421278;
var centerLongitude = -97.426261;
var radius = 25;
return communities.Select(c => new ApiSearchCommunity()
{
CommunityId = c.CommunityId,
City = c.City,
//Distance = c.GetArcDistance(centerLatitude, centerLongitude, c.Latitude, c.Longitude, radius)
});
}
Can I somehow write a helper like GetArcDistance above which in turn calls a UDF on SQL? The query I am trying to generate is the following
SELECT
comms.community_id,
comms.city,
comms.distance
FROM (
SELECT
c.community_id,
c.city,
dbo.udf_ArcDistance(
30.421278,-97.426261,
c.community_latitude,
c.community_longitude
) AS distance
FROM communities c) AS comms
WHERE comms.distance <= 25
ORDER BY comms.distance

Ok, I think I understand the question - the gist of it is you want to be able to call a SQL UDF as part of your Linq to Entities query.
This is if you're using database or model first:
This article explains how to do it: http://msdn.microsoft.com/en-us/library/dd456847(VS.100).aspx
To sum it up, you first need to edit your edmx file in an xml editor, in the edmx:StorageModels >> Schema section you need to specify a mapping to your sql udf, eg
<Function Name="SampleFunction" ReturnType="int" Schema="dbo">
<Parameter Name="Param" Mode="In" Type="int" />
</Function>
Then you need to create a static function somewhere with the EdmFunction attribute on it, something like this:
public static class ModelDefinedFunctions
{
[EdmFunction("TestDBModel.Store", "SampleFunction")]
public static int SampleFunction(int param)
{
throw new NotSupportedException("Direct calls are not supported.");
}
}
This method will get mapped to the UDF at query time by entity framework. The first attribute argument is the store namespace - you can find this in your edmx xml file on the Schema element (look for Namespace). The second argument is the name of the udf.
You can then call it something like this:
var result = from s in context.UDFTests
select new
{
TestVal = ModelDefinedFunctions.SampleFunction(22)
};
Hope this helps.

if you use Code-First approach, then you cannot call UDFs as you want (as of EF6) - here is the proof, and another one. You are only limited to calling UDF as a part of your SQL query:
bool result = FooContext.CreateQuery<bool>(
"SELECT VALUE FooModel.Store.UserDefinedFunction(#someParameter) FROM {1}",
new ObjectParameter("someParameter", someParameter)
).First();
which is ugly IMO and error-prone.
Also - this MSDN page says:
The process for calling a custom function requires three basic steps:
Define a function in your conceptual model or declare a function in your storage model.
which essentially means you need to use Model-First approach to call UDFs.

select table in database using a parameter in WCF service

I am still a beginner at writing C# and SQL and was wondering if someone could help me with this basic question. I have looked all over the internet and am still stuck.
I am trying to write a WCF service to access my database. I only need one method:
public PathDto GetPath(string src, string trg)
{
using (var context = new PathsEntities())
{
var p = (
from a
in context.src
where a.Target = trg
select a).Distance, Path;
}
}
where the parameter src is the table name, and the string trg is the entity's primary key.
Visual studio gives me the error: ...pathsService does not contain a definition for src because it is trying to look up the table "src" and not the string contained in the variable.
How can I use my parameter in the lookup statement?

I am going to assume you are using DbContext EF5.0 stuff
public PathDto GetPath(string tableType, string id)
{
using (var context = new PathsEntities())
{
var type = Type.GetType(tableType);
var p = context.Set(type).Find(id);
return (PathDto)p;
}
}
Seems you DON'T use EF 5.0 and have only got EF 4.0 and are using ObjectContext. Try this...no idea if it works since I don't really use EF 4.0. Alternatively download EF 5.0
public PathDto GetPath(string tableType, string id)
{
using (var context = new PathsEntities())
{
var type = Type.GetType(tableType);
var p = context.GetObjectByKey(new EntityKey(tableType, "id", id));
return (PathDto)p;
}
}

Calling user defined functions in Entity Framework 4

I have a user defined function in a SQL Server 2005 database which returns a bit. I would like to call this function via the Entity Framework. I have been searching around and haven't had much luck.
In LINQ to SQL this was obscenely easy, I would just add the function to the Data context Model, and I could call it like this.
bool result = FooContext.UserDefinedFunction(someParameter);
Using the Entity Framework, I have added the function to my Model and it appears under SomeModel.Store\Stored Procedures in the Model Browser.
The model has generated no code for the function, the XML for the .edmx file contains:
<Function Name="UserDefinedFunction" ReturnType="bit" Aggregate="false" BuiltIn="false" NiladicFunction="false" IsComposable="true" ParameterTypeSemantics="AllowImplicitConversion" Schema="dbo">
<Parameter Name="someParameter" Type="int" Mode="In" />
</Function>
The closest I could get was something like this:
bool result = ObjectContext.ExecuteFunction<bool>(
"UserDefinedFunction",
new ObjectParameter("someParameter", someParameter)
).First();
But I got the following error message:
The FunctionImport
'UserDefinedFunction' could not be
found in the container 'FooEntities'.
Names have been changed to protect the innocent.
tldr: How do I call scalar valued user defined functions using Entity Framework 4.0?

I have finally worked it out :D For scalar functions you can append the FROM {1} clause.
bool result = FooContext.CreateQuery<bool>(
"SELECT VALUE FooModel.Store.UserDefinedFunction(#someParameter) FROM {1}",
new ObjectParameter("someParameter", someParameter)
).First();
This is definitely a case for using LINQ to SQL over EF.

If you want to call table-valued function in MS SQL via Entity Framework;
You can use this:
var retval = db.Database.SqlQuery<MyClass>(String.Format(#"select * from dbo.myDatabaseFunction({0})", id));

calling user defined static type SQL function
You can use ExecuteStoreQuery,
first parameter takes the SQL function calling statement in string format
second parameter takes an object of SqlParameter class in which you pass your function parameter name with its value. as shown in the below method
public string GetNumberOFWorkDays(Guid leaveID)
{
using (var ctx = new INTERNAL_IntranetDemoEntities())
{
return ctx.ExecuteStoreQuery<string>(
"SELECT [dbo].[fnCalculateNumberOFWorkDays](#leaveID)",
new SqlParameter { ParameterName = "leaveID", Value = leaveID }
).FirstOrDefault();
}
}

Calling a function in EF4 with ODPNET beta 2 (Oracle), which returns a NUMBER:
using (var db = new FooContext())
{
var queryText = "SELECT FooModel.Store.MY_FUNC(#param) FROM {1}"
var result = db.CreateQuery<DbDataRecord>(queryText,new ObjectParameter("param", paramvalue));
return result.First().GetDecimal(0);
}
I'm adding it here because based on Evil Pigeon's code I could figure it out for Oracle. (Also upvoted his answer).

ObjectResult result = context.ExecuteStoreQuery("select EmployeeName from User where Id = {0}", 15);
http://msdn.microsoft.com/en-us/library/ee358758.aspx

You might find this helpful. It would appear that you can only call them via eSQL directly and not using Linq.

thought id add a note for anybody else that come across this, if your sql function parameter is nullable you have to call the function in sql with parameter value of "default". To call a function like that using Entity Framwork try
bool result = FooContext.CreateQuery<bool>(
"SELECT VALUE FooModel.Store.UserDefinedFunction(null) FROM {1}"
).First();