Develop Reference

C# IEnumerable Return Changes after Function Parameter Changes

I was writing unit tests to compare an original response to a filtered response using a request object as a parameter. In doing so I noticed that if I change the request object after getting a response the IEnumerable list will change - As I type this, my thinking is that because it is an IEnumerable with LINQ, the request.Filter property is a reference in the LINQ query, which is what causes this behavior. If I converted this to a list instead of an IEnumerable, I suspect the behavior would go away because the .ToList() will evaluate the LINQ expressions instead of deferring. Is that the case?
public class VendorResponse {
public IEnumerable<string> Vendors { get; set; }
}
var request = new VendorRequest() {
Filter = ""
};
var response = await _service.GetVendors(request);
int vendorCount = response.Vendors.Count(); // 20
request.Filter = "at&t";
int newCount = response.Vendors.Count(); // 17
public async Task<VendorResponse> GetVendors(VendorRequest request)
{
var vendors = await _dataService.GetVendors();
return new VendorResponse {
Vendors = vendors.Where(v => v.IndexOf(request.Filter) >= 0)
}
}

If deferred execution is preferable, you can capture the current state of request.Filter with a local variable and use that in the Where predicate
public async Task<VendorResponse> GetVendors(VendorRequest request)
{
var filter = request.Filter;
var vendors = await _dataService.GetVendors();
return new VendorResponse {
Vendors = vendors.Where(v => v.IndexOf(filter) >= 0)
}
}

Yes!
This is an example of deferred execution of an IEnumerable, which just encapsulates a query on some data without encapsulating the result of that query.
An IEnumerable can be enumerated (via its IEnumerator), and "knows" how to enumerate the query it encapsulates, but this will not actually happen until something executes the enumeration.
In your case the enumeration is executed by the call to .Count() which needs to know how many items are in the result of the query. The enumeration occurs every time you call .Count(), so changing the filter between the two invocations leads to you getting two different results.
As you have correctly deduced, calling .ToList() and capturing the result in a variable before performing any further operations would lead to you capturing the resulting data rather than the query, and so lead to both counts having the same value.
Try this out yourself. In future, be sure to force the evaluation of the enumerable before passing to other queries, or returning out to unknown code, otherwise you or your users will encounter unexpected behaviour and possible performance issues.
Hope this helps :)
Edit 1:
As Moho has pointed out, and you have also alluded to in your original post, this is also a result of the request.Filter being captured by the IEnumerable as a reference type. If you can capture the value and pass this in instead, the result of the IEnumerable will no longer be modified by changing the filter.

Execute custom SQL before running FromSqlRaw in Entity Framework Core 6 or above in SQL Server

I only need it to work for SQL Server. This is an example. The question is about a general approach.
There is a nice extension method from https://entityframework-extensions.net called WhereBulkContains. It is, sort of, great, except that the code of the methods in this library is obfuscated and they do not produce valid SQL when .ToQueryString() is called on IQueryable<T> with these extension methods applied.
Subsequently, I can't use such methods in production code as I am not "allowed" to trust such code due to business reasons. Sure, I can write tons of tests to ensure that WhereBulkContains works as expected, except that there are some complicated cases where the performance of WhereBulkContains is well below stellar, whereas properly written SQL works in a blink of an eye. And (read above), since the code of this library is obfuscated, there is no way to figure out what's wrong there without spending a large amount of time. We would've bought the license (as this is not a freeware) if the library weren't obfuscated. All together that basically kills the library for our purposes.
This is where it gets interesting. I can easily create and populate a temporary table, e.g. (I have a table called EFAgents with an int PK called AgentId in the database):
private string GetTmpAgentSql(IEnumerable<int> agentIds) => #$"
drop table if exists #tmp_Agents;
create table #tmp_Agents (AgentId int not null, primary key clustered (AgentId asc));
{(agentIds
.Chunk(1_000)
.Select(e => $#"
insert into #tmp_Agents (AgentId)
values
({e.JoinStrings("), (")});
")
.JoinStrings(""))}
select 0 as Result
";
private const string AgentSql = #"
select a.* from EFAgents a inner join #tmp_Agents t on a.AgentID = t.AgentId";
where GetContext returns EF Core database context and JoinStrings comes from Unity.Interception.Utilities and then use it as follows:
private async Task<List<EFAgent>> GetAgents(List<int> agentIds)
{
var tmpSql = GetTmpAgentSql(agentIds);
using var ctx = GetContext();
// This creates a temporary table and populates it with the ids.
// This is a proprietary port of EF SqlQuery code, but I can post the whole thing if necessary.
var _ = await ctx.GetDatabase().SqlQuery<int>(tmpSql).FirstOrDefaultAsync();
// There is a DbSet<EFAgent> called Agents.
var query = ctx.Agents
.FromSqlRaw(AgentSql)
.Join(ctx.Agents, t => t.AgentId, a => a.AgentId, (t, a) => a);
var sql = query.ToQueryString() + Environment.NewLine;
// This should provide a valid SQL; https://entityframework-extensions.net does NOT!
// WriteLine - writes to console or as requested. This is irrelevant to the question.
WriteLine(sql);
var result = await query.ToListAsync();
return result;
}
So, basically, I can do what I need in two steps:
using var ctx = GetContext();
// 1. Create a temp table and populate it - call GetTmpAgentSql.
...
// 2. Build the join starting from `FromSqlRaw` as in example above.
This is doable, half-manual, and it is going to work.
The question is how to do that in one step, e.g., call:
.WhereMyBulkContains(aListOfIdConstraints, whateverElseIsneeded, ...)
and that's all.
I am fine if I need to pass more than one parameter in each case in order to specify the constraints.
To clarify the reasons why do I need to go into all these troubles. We have to interact with a third party database. We don't have any control of the schema and data there. The database is large and poorly designed. That resulted in some ugly EFC LINQ queries. To remedy that, some of that ugliness was encapsulated into a method, which takes IQueryable<T> (and some more parameters) and returns IQueryable<T>. Under the hood this method calls WhereBulkContains. I need to replace this WhereBulkContains by, call it, WhereMyBulkContains, which would be able to provide correct ToQueryString representation (for debugging purposes) and be performant. The latter means that SQL should not contain in clause with hundreds (and even sometimes thousands) of elements. Using inner join with a [temp] table with a PK and having an index on the FK field seem to do the trick if I do that in pure SQL. But, ... I need to do that in C# and effectively in between two LINQ method calls. Refactoring everything is also not an option because that method is used in many places.
Thanks a lot!

I think you really want to use a Table Valued Parameter.
Creating an SqlParameter from an enumeration is a little fiddly, but not too difficult to get right;
CREATE TYPE [IntValue] AS TABLE (
Id int NULL
)
private IEnumerable<SqlDataRecord> FromValues(IEnumerable<int> values)
{
var meta = new SqlMetaData(
"Id",
SqlDbType.Int
);
foreach(var value in values)
{
var record = new SqlDataRecord(
meta
);
record.SetInt32(0, value);
yield return record;
}
}
public SqlParameter ToIntTVP(IEnumerable<int> values){
return new SqlParameter()
{
TypeName = "IntValue",
SqlDbType = SqlDbType.Structured,
Value = FromValues(values)
};
}
Personally I would define a query type in EF Core to represent the TVP. Then you can use raw sql to return an IQueryable.
public class IntValue
{
public int Id { get; set; }
}
modelBuilder.Entity<IntValue>(e =>
{
e.HasNoKey();
e.ToView("IntValue");
});
IQueryable<IntValue> ToIntQueryable(DbContext ctx, IEnumerable<int> values)
{
return ctx.Set<IntValue>()
.FromSqlInterpolated($"select * from {ToIntTVP(values)}");
}
Now you can compose the rest of your query using Linq.
var ids = ToIntQueryable(ctx, agentIds);
var query = ctx.Agents
.Where(a => ids.Any(i => i.Id == a.Id));

I would propose to use linq2db.EntityFrameworkCore (note that I'm one of the creators). It has built-in temporary tables support.
We can create simple and reusable function which filters records of any type:
public static class HelperMethods
{
private class KeyHolder<T>
{
[PrimaryKey]
public T Key { get; set; } = default!;
}
public static async Task<List<TEntity>> GetRecordsByIds<TEntity, TKey>(this IQueryable<TEntity> query, IEnumerable<TKey> ids, Expression<Func<TEntity, TKey>> keyFunc)
{
var ctx = LinqToDBForEFTools.GetCurrentContext(query) ??
throw new InvalidOperationException("Query should be EF Core query");
// based on DbContext options, extension retrieves connection information
using var db = ctx.CreateLinqToDbConnection();
// create temporary table and BulkCopy records into that table
using var tempTable = await db.CreateTempTableAsync(ids.Select(id => new KeyHolder<TKey> { Key = id }), tableName: "temporaryIds");
var resultQuery = query.Join(tempTable, keyFunc, t => t.Key, (q, t) => q);
// we use ToListAsyncLinqToDB to avoid collission with EF Core async methods.
return await resultQuery.ToListAsyncLinqToDB();
}
}
Then we can rewrite your function GetAgents to the following:
private async Task<List<EFAgent>> GetAgents(List<int> agentIds)
{
using var ctx = GetContext();
var result = await ctx.Agents.GetRecordsByIds(agentIds, a => a.AgentId);
return result;
}

How to properly convert a list to IQueryable on a asp net core web API

im trying to deploy a simple search function that uses a simple tag system, probably there are better ways to bt this is the solution i landed on:
public async Task<ActionResult<IEnumerable<t_usuarios_pub>>> Gett_usuarios_pubByTag(string tag)
{
string[] TagList;
TagList = tag.Split(',');
List<t_usuarios_pub> results = new List<t_usuarios_pub>();
var pubs = from m in _context.t_usuarios_pub select m;
if (!String.IsNullOrEmpty(tag))
{
foreach (var Itag in TagList)
{
pubs = pubs.Where(s => (s.tag.Contains(Itag) && s.estatus<2)).OrderBy(x => x.estatus);
foreach(var x in pubs)
{
if (!results.Contains(x))
{
results.Add(x);
}
}
}
}
return await results.AsQueryable().ToListAsync();
}
problem is that the List results cant be converted to IQueryable, this is the error stack.
Any idea of how i can properly implement it ?
System.InvalidOperationException:
The source 'IQueryable' doesn't implement 'IAsyncEnumerable<UserPostAPI.Models.t_usuarios_pub>'.
Only sources that implement 'IAsyncEnumerable' can be used for Entity Framework asynchronous operations.
´´´

Since results is a local variable of List<> type, I don't believe you need to await the last line. It might just work with:
return results.AsQueryable();
In which case, your method may not even need to be an async method. Or, depending on what _context is, perhaps the await needs to be on the pubs filter call:
pubs = await pubs.Where(s => (s.tag.Contains(Itag) && s.estatus<2))
.OrderBy(x => x.estatus)
.ToListAsync();
Furthermore, since your method says it returns an IEnumerable<>, you probably don't need the .AsQueryable(), either:
return result;
There's also a lot of refactoring you could do -- here are just a few things you may want to consider:
Move the String.IsNullOrEmpty(tag) check to the beginning of the method as a guard.
If this is user input, trim the split tags in TagList to avoid any issues with extra whitespace, and then be sure to remove any resulting empty members.
You could put the estatus < 2 check in your query to reduce having to re-check it again for every item in pubs for every ITag.

How to apply unknown query operators to a collection reference on Firebase Client

I have written a GetCollectionAsync method for querying collections on Firestore. As long as I want to get the whole collection documents, it works fine. Now I want to introduce query operators such as: WhereEqualTo, WhereIn, WhereGreaterThanOrEqualTo, etc
I want my method to stay generic and expose a parameter so that the caller can pass one or many query operators. Is there a way to achieve this?
public async Task<IEnumerable<T>> GetCollectionAsync<T>(string path, /* query operators? */)
{
var collectionRef = _firestoreDb.Collection(path);
/* apply query operators to collection reference? */
var snapshot = await collectionRef.GetSnapshotAsync();
return snapshot.ConvertTo<T>();
}

As requested by #FaridShumbar here's my workaround.
The concession I made was to split the method in two. So the caller needs first to call a GetCollectionReference method, apply the filters on it, and pass it to QueryCollectionAsync to execute the query.
Code below:
public CollectionReference GetCollectionReference(string collectionPath)
{
return _firestoreDb.Collection(collectionPath);
}
public async Task<IEnumerable<T>> QueryCollectionAsync<T>(Query collectionReference)
{
var snapshot = await collectionReference.GetSnapshotAsync();
return snapshot.ConvertTo<T>();
}
// Call example
var collectionRef = _firebaseService.GetCollectionReference(path).WhereEqualTo("image", null);
var articles = await _firebaseService.QueryCollectionAsync<MyArticle>(collectionRef);

Creating a completely dynamic query with RavenDB using LuceneQuery

I want one method that can query my entire RavenDB database.
My method signature looks like this:
public static DataTable GetData(string className, int amount, string orderByProperty, string filterByProperty, string filterByOperator, string filterCompare)
I figured I can accomplish all of the above with a dynamic LuceneQuery.
session.Advanced.LuceneQuery<dynamic>();
The problem is: Since I'm using dynamic in the type given, how do I ensure that the query only includes the types matching the className?
I'm looking for something like .WhereType(className) or .Where("type: " + className).
Solution
This returns the results of the correct type:
var type = Type.GetType("Business.Data.DTO." + className);
var tagName = RavenDb.GetTypeTagName(type);
using (var session = RavenDb.OpenSession())
{
var result = session.Advanced
.LuceneQuery<object, RavenDocumentsByEntityName>()
.WhereEquals("Tag", tagName)
.ToList();
}
Note, it is not possible to add additional "WhereEquals" or other filters to this. This is because nothing specific to that document type is included in the "RavenDocumentByEntityName" index.
This means that this solution cannot be used for what I wanted to accomplish.
What I ended up doing
Although it doesn't fulfill my requirement completely, this is what I ended up doing:
public static List<T> GetData<T>(DataQuery query)
{
using (var session = RavenDb.OpenSession())
{
var result = session.Advanced.LuceneQuery<T>();
if (!string.IsNullOrEmpty(query.FilterByProperty))
{
if (query.FilterByOperator == "=")
{
result = result.WhereEquals(query.FilterByProperty, query.FilterCompare);
}
else if (query.FilterByOperator == "StartsWith")
{
result = result.WhereStartsWith(query.FilterByProperty, query.FilterCompare);
}
else if (query.FilterByOperator == "EndsWith")
{
result = result.WhereEndsWith(query.FilterByProperty, query.FilterCompare);
}
}
if (!string.IsNullOrEmpty(query.OrderByProperty))
{
if (query.Descending)
{
result = result.OrderBy(query.OrderByProperty);
}
else
{
result = result.OrderByDescending(query.OrderByProperty);
}
}
result = result.Skip(query.Skip).Take(query.Amount);
return result.ToList();
}
}
Although this is most certainly an anti-pattern, it's a neat way to just look at some data, if that's what you want. It's called very easily like this:
DataQuery query = new DataQuery
{
Amount = int.Parse(txtAmount.Text),
Skip = 0,
FilterByProperty = ddlFilterBy.SelectedValue,
FilterByOperator = ddlOperator.SelectedValue,
FilterCompare = txtCompare.Text,
OrderByProperty = ddlOrderBy.SelectedValue,
Descending = chkDescending.Checked
};
grdData.DataSource = DataService.GetData<Server>(query);
grdData.DataBind();
"Server" is one of the classes/document types I'm working with, so the downside, where it isn't completely dynamic, is that I would have to define a call like that for each type.

I strongly suggest you don't go down this road. You are essentially attempting to hide the RavenDB Session object, which is very powerful and intended to be used directly.
Just looking at the signature of the method you want to create, the parameters are all very restrictive and make a lot of assumptions that might not be true for the data you're working on. And the return type - why would you return a DataTable? Maybe return an object or a dynamic, but nothing in Raven is structured in tables, so DataTable is a bad idea.
To answer the specific question, the type name comes from the Raven-Entity-Name metadata, which you would need to build an index over. This happens automatically when you index using the from docs.YourEntity syntax in an index. Raven does this behind the scenes when you use a dynamic index such as .Query<YourEntity> or .Advanced.LuceneQuery<YourEntity>.
Still, you shouldn't do this.