I am using Dapper with Oracle Managed and trying to use QueryMultiple method. I have a stored procedure in an Oracle package that returns multiple recordsets. I want to reuse my code for multiple package methods and the data coming back is coming from different source tables, so columns and actual data will vary.
To handle this I have a class that uses an IEnumerable<string> for storing FieldNames and an IEnumerable<object[]> for storing the data for each row.
With just a single recordset I am able to use the ExecuteReader method to iterate the results and add them myself. However, I would like to use QueryMultiple to get everything in a single call. Right now I have two recordsets coming back but others may be added.
After a few different attempts I was able to get the following code to work. However, it seems like there should be a better way to get this. Further below is a piece of code found in another SO question that seemed to be what I wanted but I just couldn't get it to work. Does anyone have any suggestions about how I'm getting the FieldNames and Data in the code below and if there is a better way using Dapper API. Loving Dapper by the way. Thanks for any suggestions.
// class to load results into
public class Results {
public IEnumerable<string> FieldNames { get; set; }
public DetailInfo Detail { get; set; }
public IEnumerable<object[]> Data { get; set; }
}
// code to get the results from the database field names and data varies
// method is private so that external callers cannot pass any just any string as methodName
private Results GetTableResults(string methodName) {
var queryparams = new OracleDynamicParameters();
queryparams.Add(name: "l_detail_cursor", dbType: OracleDbType.RefCursor, direction: ParameterDirection.Output);
queryparams.Add(name: "l_data_cursor", dbType: OracleDbType.RefCursor, direction: ParameterDirection.Output);
// ... other parameters go here, removed for example
Results results = new Results();
string sql = string.Format("{0}.GET_{1}_DETAILS", PackageName, methodName);
using (IDbConnection db = new OracleConnection(this.ConnectionString)) {
db.Open();
using (var multi = db.QueryMultiple(sql: sql, param: queryparams, commandType: CommandType.StoredProcedure)) {
// detail in first cursor, no problems here
results.Detail = multi.Read<DetailInfo>().Single();
// --------------------------------------------------
// this is the code I'm trying to see if there is a better way to handle
// --------------------------------------------------
// data in second cursor
var data = multi.Read().Select(dictionary =>
// cast to IDictionary
dictionary as IDictionary<string, object>
);
// pull from Keys
results.FieldNames = data.First().Select(d => d.Key);
// pull from values
results.Data = data.Select(d => d.Values.ToArray());
// --------------------------------------------------
}
}
return results;
}
I was attempting to try to use something like the following but only get an exception at run time about splitOn needing to be specified. I tried using something like ROWNUM to even give the recordset an "id" but that didn't seem to help.
var data = multi.Read<dynamic, dynamic, Tuple<dynamic, dynamic>>(
(a, b) => Tuple.Create((object)a, (object)b)).ToList();
Related
I only need it to work for SQL Server. This is an example. The question is about a general approach.
There is a nice extension method from https://entityframework-extensions.net called WhereBulkContains. It is, sort of, great, except that the code of the methods in this library is obfuscated and they do not produce valid SQL when .ToQueryString() is called on IQueryable<T> with these extension methods applied.
Subsequently, I can't use such methods in production code as I am not "allowed" to trust such code due to business reasons. Sure, I can write tons of tests to ensure that WhereBulkContains works as expected, except that there are some complicated cases where the performance of WhereBulkContains is well below stellar, whereas properly written SQL works in a blink of an eye. And (read above), since the code of this library is obfuscated, there is no way to figure out what's wrong there without spending a large amount of time. We would've bought the license (as this is not a freeware) if the library weren't obfuscated. All together that basically kills the library for our purposes.
This is where it gets interesting. I can easily create and populate a temporary table, e.g. (I have a table called EFAgents with an int PK called AgentId in the database):
private string GetTmpAgentSql(IEnumerable<int> agentIds) => #$"
drop table if exists #tmp_Agents;
create table #tmp_Agents (AgentId int not null, primary key clustered (AgentId asc));
{(agentIds
.Chunk(1_000)
.Select(e => $#"
insert into #tmp_Agents (AgentId)
values
({e.JoinStrings("), (")});
")
.JoinStrings(""))}
select 0 as Result
";
private const string AgentSql = #"
select a.* from EFAgents a inner join #tmp_Agents t on a.AgentID = t.AgentId";
where GetContext returns EF Core database context and JoinStrings comes from Unity.Interception.Utilities and then use it as follows:
private async Task<List<EFAgent>> GetAgents(List<int> agentIds)
{
var tmpSql = GetTmpAgentSql(agentIds);
using var ctx = GetContext();
// This creates a temporary table and populates it with the ids.
// This is a proprietary port of EF SqlQuery code, but I can post the whole thing if necessary.
var _ = await ctx.GetDatabase().SqlQuery<int>(tmpSql).FirstOrDefaultAsync();
// There is a DbSet<EFAgent> called Agents.
var query = ctx.Agents
.FromSqlRaw(AgentSql)
.Join(ctx.Agents, t => t.AgentId, a => a.AgentId, (t, a) => a);
var sql = query.ToQueryString() + Environment.NewLine;
// This should provide a valid SQL; https://entityframework-extensions.net does NOT!
// WriteLine - writes to console or as requested. This is irrelevant to the question.
WriteLine(sql);
var result = await query.ToListAsync();
return result;
}
So, basically, I can do what I need in two steps:
using var ctx = GetContext();
// 1. Create a temp table and populate it - call GetTmpAgentSql.
...
// 2. Build the join starting from `FromSqlRaw` as in example above.
This is doable, half-manual, and it is going to work.
The question is how to do that in one step, e.g., call:
.WhereMyBulkContains(aListOfIdConstraints, whateverElseIsneeded, ...)
and that's all.
I am fine if I need to pass more than one parameter in each case in order to specify the constraints.
To clarify the reasons why do I need to go into all these troubles. We have to interact with a third party database. We don't have any control of the schema and data there. The database is large and poorly designed. That resulted in some ugly EFC LINQ queries. To remedy that, some of that ugliness was encapsulated into a method, which takes IQueryable<T> (and some more parameters) and returns IQueryable<T>. Under the hood this method calls WhereBulkContains. I need to replace this WhereBulkContains by, call it, WhereMyBulkContains, which would be able to provide correct ToQueryString representation (for debugging purposes) and be performant. The latter means that SQL should not contain in clause with hundreds (and even sometimes thousands) of elements. Using inner join with a [temp] table with a PK and having an index on the FK field seem to do the trick if I do that in pure SQL. But, ... I need to do that in C# and effectively in between two LINQ method calls. Refactoring everything is also not an option because that method is used in many places.
Thanks a lot!
I think you really want to use a Table Valued Parameter.
Creating an SqlParameter from an enumeration is a little fiddly, but not too difficult to get right;
CREATE TYPE [IntValue] AS TABLE (
Id int NULL
)
private IEnumerable<SqlDataRecord> FromValues(IEnumerable<int> values)
{
var meta = new SqlMetaData(
"Id",
SqlDbType.Int
);
foreach(var value in values)
{
var record = new SqlDataRecord(
meta
);
record.SetInt32(0, value);
yield return record;
}
}
public SqlParameter ToIntTVP(IEnumerable<int> values){
return new SqlParameter()
{
TypeName = "IntValue",
SqlDbType = SqlDbType.Structured,
Value = FromValues(values)
};
}
Personally I would define a query type in EF Core to represent the TVP. Then you can use raw sql to return an IQueryable.
public class IntValue
{
public int Id { get; set; }
}
modelBuilder.Entity<IntValue>(e =>
{
e.HasNoKey();
e.ToView("IntValue");
});
IQueryable<IntValue> ToIntQueryable(DbContext ctx, IEnumerable<int> values)
{
return ctx.Set<IntValue>()
.FromSqlInterpolated($"select * from {ToIntTVP(values)}");
}
Now you can compose the rest of your query using Linq.
var ids = ToIntQueryable(ctx, agentIds);
var query = ctx.Agents
.Where(a => ids.Any(i => i.Id == a.Id));
I would propose to use linq2db.EntityFrameworkCore (note that I'm one of the creators). It has built-in temporary tables support.
We can create simple and reusable function which filters records of any type:
public static class HelperMethods
{
private class KeyHolder<T>
{
[PrimaryKey]
public T Key { get; set; } = default!;
}
public static async Task<List<TEntity>> GetRecordsByIds<TEntity, TKey>(this IQueryable<TEntity> query, IEnumerable<TKey> ids, Expression<Func<TEntity, TKey>> keyFunc)
{
var ctx = LinqToDBForEFTools.GetCurrentContext(query) ??
throw new InvalidOperationException("Query should be EF Core query");
// based on DbContext options, extension retrieves connection information
using var db = ctx.CreateLinqToDbConnection();
// create temporary table and BulkCopy records into that table
using var tempTable = await db.CreateTempTableAsync(ids.Select(id => new KeyHolder<TKey> { Key = id }), tableName: "temporaryIds");
var resultQuery = query.Join(tempTable, keyFunc, t => t.Key, (q, t) => q);
// we use ToListAsyncLinqToDB to avoid collission with EF Core async methods.
return await resultQuery.ToListAsyncLinqToDB();
}
}
Then we can rewrite your function GetAgents to the following:
private async Task<List<EFAgent>> GetAgents(List<int> agentIds)
{
using var ctx = GetContext();
var result = await ctx.Agents.GetRecordsByIds(agentIds, a => a.AgentId);
return result;
}
In an MVC 5 web app using Entity Framework, I learned how to populate an Index view by using db.Database.SqlQuery<model> to execute a stored procedure and show the results in the Index View.
This is the relevant code in my Index View (and it works).
// supply parameter values required by the stored procedure
object[] parameters = {
new SqlParameter("#campus",SqlDbType.NVarChar,3) {Value=vm.SelectedCampus},
new SqlParameter("#date1",SqlDbType.DateTime) {Value=Convert.ToDateTime(vm.SelectedStartDate)},
new SqlParameter("#date2",SqlDbType.DateTime) {Value=Convert.ToDateTime(vm.SelectedEndDate)}
};
// populate the list by calling the stored procedure and supplying parameters
IEnumerable<PerfOdomoeterDate> query =
db.Database.SqlQuery<PerfOdomoeterDate>("PerfOdomoeterDate #campus, #date1, #date2",
parameters).OrderBy(m => m.StudentName).ToList();
And to put that code into better context, here is the entire Index ActionResult.
private PerformanceContext db = new PerformanceContext();
private static readonly string d1 = DateTime.Now.ToShortDateString();
private static readonly string d2 = DateTime.Now.ToShortDateString();
[HttpGet]
public ActionResult Index(int? page, string SelectedCampus = "CRA", string SelectedStartDate=null, string SelectedEndDate=null)
{
int PageNumber = (page ?? 1);
PerfOdomoeterDateViewModel vm = new PerfOdomoeterDateViewModel();
vm.SelectedCampus = SelectedCampus;
vm.SelectedStartDate = string.IsNullOrEmpty(SelectedStartDate) ? d1 : SelectedStartDate;
vm.SelectedEndDate = string.IsNullOrEmpty(SelectedEndDate) ? d2 :SelectedEndDate;
vm.CampusList = StaticClasses.ListBank.CampusList();
// supply parameter values required by the stored procedure
object[] parameters = {
new SqlParameter("#campus",SqlDbType.NVarChar,3) {Value=vm.SelectedCampus},
new SqlParameter("#date1",SqlDbType.DateTime) {Value=Convert.ToDateTime(vm.SelectedStartDate)},
new SqlParameter("#date2",SqlDbType.DateTime) {Value=Convert.ToDateTime(vm.SelectedEndDate)}
};
// populate the list by calling the stored procedure and supplying parameters
IEnumerable<PerfOdomoeterDate> query =
db.Database.SqlQuery<PerfOdomoeterDate>("PerfOdomoeterDate #campus, #date1, #date2",
parameters).OrderBy(m => m.StudentName).ToList();
vm.CreditTable = query.ToPagedList(PageNumber, 25);
return View(vm);
}
As I stated, this code works perfectly in the Index View. However, in a separate ActionResult, where the user has an option to export the data set to an Excel file, I use the same code, and I get this runtime error:
The SqlParameter is already contained by another SqlParameterCollection.
I was under the impression that each ActionResult is in its own scope, so how is it that I am getting this error when I am calling up a new query from a separate ActionResult?
Intellisense did not give me any clues as to how I could explicitly empty the parameters after executing the stored procedure.
This is the code in the ActionResult that is producing the error.
public ActionResult ExportToExcel(string SelectedCampus, string SelectedStartDate, string SelectedEndDate)
{
object[] parameters2 = {
new SqlParameter("#campus",SqlDbType.NVarChar,3) {Value=SelectedCampus},
new SqlParameter("#date1",SqlDbType.DateTime) {Value=Convert.ToDateTime(SelectedStartDate)},
new SqlParameter("#date2",SqlDbType.DateTime) {Value=Convert.ToDateTime(SelectedEndDate)}
};
IEnumerable<PerfOdomoeterDate> query =
db.Database.SqlQuery<PerfOdomoeterDate>("PerfOdomoeterDate #campus, #date1, #date2",
parameters2).OrderBy(m => m.StudentName).AsEnumerable();
...
The ADO.Net objects (like SqlParameter, SqlCommand etc.) presented to us by the .Net framework are a mere layer on top of the real stuff under the hood that is managed by the .Net connection pool. If we create a new SqlConnection —which is implicitly done by db.Database.SqlQuery— we don't really establish a new connection to the database. That would be far too expensive. In reality, our connection object "plugs" in to an available connection in the connection pool.
Normally, this mechanism is pretty transparent, but it is unveiled in issues like the one you see here. I remember having experienced similar issues (exceptions that persisted longer than met the eye).
The message is: you can't beat it, so join it. The quick solution seems to be renaming the parameters in one of the methods. A better solution, of course, is to factor out the repetitive part of your code into a method that contains the identical parts.
I would say, This is how as per the design.
You need to extract the data right from there .ToArray() or .ToList() etc...
Do not try to re execute the query for further data operations.
I create a simple stored procedure with some joins with the customer table and other related tables, which takes in two parameters. I can execute this SP in SQL and works.
I drag and drop this SP to my DBML file and recompile.
I add the below code in order to call the SP and return it in a List
public IQueryable<Entities.Customer> AllCustomerRanges(int CId, int ItemID)
{
List<Entities.Customer> c = myDataContext.spCustomerRanges(CId, ItemID).ToList();
}
This gives me the error:
Cannot implicitly convert type 'System.Collections.Generic.List< spCustomerRangesResult>' to 'System.Collections.Generic.List< Entities.Customer>'
Now i dont have a class spCustomerRangesResult but after some research I'm puzzled if i have done something wrong or if i need to implement a class with all the properties that the Customer class has (which sounds a little long winded) or if i've just made an error.
Any idea of how i can call a SP which shows the data in a List?
new class spCustomerRangesResult automatically generated based on sp result, you should convert it to Entities.Customer like this:
public IQueryable<Entities.Customer> AllCustomerRanges(int CId, int ItemID)
{
var c = myDataContext.spCustomerRanges(CId, ItemID).ToList();
if (c == null)
return null;
var customers = c.Select(a => new Entities.Customer
{
FirstName=a.spResultFirstName,
LastName = a.spResultLastName
//this just example conversion, change it as needed.
});
return customers;
}
please note, that I return IQueryable even though the approach that you take when using ToList() but yet returning IQuerybale may not be needed. I dont know all details so this only to show how to convert but the whole method may need re-factoring.
I have a fairly agnostic ADO.NET application that connects to a number of databases and is able to extract the necessary information to run. I have hit a snag with DB2 and how it handles named parameters, particularly when I reuse a named parameter in the same query. I know of a couple of ways to get around this by simply adding more parameters, but in theory it should work as it does on other databases that I connect to as the parameter name is the same.
What I'm doing is a bit more complicated and involves subqueries etc, but to demonstrate, take the following query:
select value from test.table where cola=#key1 and colb=#key1;
The named parameter #key1 is used twice.
My code is as follows:
try
{
DbProviderFactory dbfFactory = DbProviderFactories.GetFactory("IBM.Data.DB2.iSeries");
using (DbConnection dbConnection = dbfFactory.CreateConnection())
{
dbConnection.ConnectionString = "DataSource=xxx.xxx.xxx.xxx;UserID=xxxxxxxx;password=xxxxxxxxx";
using (DbCommand dbCommand = dbConnection.CreateCommand())
{
IDbDataParameter iddpParameter1 = dbCommand.CreateParameter();
iddpParameter1.ParameterName = "#key1";
iddpParameter1.DbType = DbType.String;
iddpParameter1.Value = "1";
dbCommand.Parameters.Add(iddpParameter1);
dbCommand.CommandType = CommandType.Text;
dbCommand.CommandText = "select value from test.table where cola=#key1 and colb=#key1";
dbConnection.Open();
using (IDataReader idrReader = dbCommand.ExecuteReader())
{
while (idrReader.Read())
{
...
}
}
}
} // end dbConnection
} // end try
catch (Exception ex)
{
Console.Write(ex.Message);
}
When I run this I get an exception that tells me:
System.InvalidOperationException: Not enough parameters specified. The command requires 2 parameter(s), but only 1 parameter(s) exist in the parameter collection.
I get what it is telling me, but I'm looking for help in figuring out how I can have the provider use the named parameter for both parameters as they are the same. It seems that it is doing a blind count of named parameters and not realizing that they are the same named parameters. SQL Server seems to allow me to do this with the same code above. I'm guessing it's just one of those differences in the providers, but hoping someone has run into this and has a solution for DB2 that doesn't get into specific DB2 code.
Thanks, appreciate the assistance.
well I did a little more digging, and I wonder if it might be the connector that you are using. So I'm doing the following (which is very similar to what you are doing)
in my app config file I have
<connectionStrings>
<add name="AWOLNATION" providerName="Ibm.Data.DB2" connectionString="Server=sail:50000;Database=Remix;" />
</connectionStrings>
in my Databasemanager class I would initialize it like so
public static DatabaseManager Instance(string connectionStringName)
{
var connectionStringSettings = ConfigurationManager.ConnectionStrings[connectionStringName];
if (connectionStringSettings == null) throw new MissingMemberException("[app.config]", string.Format("ConnectionStrings[{0}]", connectionStringName));
return new DatabaseManager(connectionStringSettings);
}
private DatabaseManager(ConnectionStringSettings connectionInformation)
{
_connectionInformation = connectionInformation;
_parameters = new Dictionary<string, object>();
}
private void Initialize()
{
_connection = DbProviderFactories.GetFactory(_connectionInformation.ProviderName).CreateConnection();
_connection.ConnectionString = _connectionInformation.ConnectionString;
_command = _connection.CreateCommand();
}
I add parameters a little different though. I have a Dictionary<string,object> that I add too when setting up my query. To use your example I would have had this
public IEnumerable<object> GetSomething(string key)
{
var sql = "select value from test.table where cola = #key1 and colb = #key1";
_manager.AddParameter("#key1", key);
return _manager.ExecuteReader<object>(sql, ToSomethignUseful);
}
private object ToSomethignUseful(DatabaseManager arg)
{
return new { Value = arg.GetArgument<object>("value") };
}
then reading is where the OP and I have similar code
public IEnumerable<T> ExecuteReader<T>(string sql, Func<DatabaseManager, T> conversionBlock)
{
Initialize();
using (_connection)
{
_connection.Open();
_command.CommandText = sql;
_command.CommandType = CommandType.Text;
if (_parameters.Count > 0)
AddParameters(_command, _parameters);
_parameters.Clear();
using (_reader = _command.ExecuteReader())
{
while (_reader.Read())
{
yield return conversionBlock(this);
}
}
}
}
private static void AddParameters(DbCommand command, Dictionary<string, object> parameters)
{
foreach (var param in parameters)
{
command.Parameters.Add(CreateParameter(command, param.Key, param.Value));
}
}
private static DbParameter CreateParameter(DbCommand command, string key, object value)
{
var parameter = command.CreateParameter();
parameter.ParameterName = key;
parameter.Value = value;
return parameter;
}
running said code is working for me, so I wonder if the difference is in the provider that we are using. I'm using named parameters in production and have been for atleast a year now, possibly closer to 2 years.
I will say that I did get the same error when essentially running the same code twice, as shown in this code
public static void IndendedPrintForEach<T>(this IEnumerable<T> array, string header, Func<T, string> consoleStringConverterMethod)
{
var list = array.ToList();
var color = Console.ForegroundColor;
Console.ForegroundColor = ConsoleColor.Magenta;
Console.WriteLine($"<<<{header}>>>");
Console.ForegroundColor = color;
if (!list.Any())
{
Console.ForegroundColor = ConsoleColor.DarkRed;
Console.WriteLine(" ************NoItemsFound************");
Console.ForegroundColor = color;
}
else
{
foreach (var item in list)
Console.WriteLine($" {consoleStringConverterMethod(item)}");
}
}
on line 3 var list = array.ToList() was the fix to the problem that you were seeing for me. before I had if (!array.Any()) which would run the query and use the parameters (which I clear out before I execture the query) then when I go to enumerate through and print each item in the array I was then getting the error. For me the problem was that it was re-running the query which I had no more parameters. The fix was to enumerate the query with the ToList() and then do my checking and printing on the list.
You answered you own question: "Unfortunately, I have not found a solution. I had to create another named parameter and just assign it the same value"
Oracle/DB2/Sybase especially are difficult with SQL queries and parameters.
Parameters in the SQL query should be in the same order they are added to the C# parameters are added to the C# SQL command (Oracle, Sybase)
Put parenthesis around the SQL query where clause parts using parameters (all)
Make sure the SQL data types are matching the C# parameter data types (all)
Check for overflow/underflow of parameter data so that the SQL query does not error
Pass in null/empty string in the appropriate format for the database. Ideally, create C# SQLParameter create methods to create the parameter in the correct format for the database
Oracle is particularly finicky about this. Take the time to build a C# wrapper library to construct a C# query object correctly, construct C# parameters correctly and add the C# SQL parameters to the query.
Put in notes that the query parameter add order should match the order of "#" parameters in the SQL query.
This wrapper library is you documentation for you and the next developer to avoid the problems you've encountered.
TL;DR I'm using EntityFramework 5.0 with Oracle and need to query a table for two columns only using index with NVL of two columns.
Details after hours of attempts... I'll try to organize it as possible.
The desired SQL query should be:
SELECT t.Code, NVL(t.Local, t.Global) Description
FROM Shows t
Where t.Code = 123
So what is the problem? If I want to use Context.Shows.Parts.SqlQuery(query) I must return the whole row(*), but then I get Table Access Full, so I must return only the desired columns.
The next thing(Actually there were a lot of tries before the following...) that I've tried which gives a very close results was using the null-coalescing operator(??) :
Context.Shows.Where(x => x.Code == 123)
.Select(x => new { x.Code, Description = x.Local ?? x.Global);
But the SQL it's using is complicated using case & when and not using my Index on Code, Nvl(Local, Global) which is critical!
My next step was using Database.SqlQuery
context.Database.SqlQuery<Tuple<int, string>>("the Raw-SQLQuery above");
But I get an error that Tuple must not be abstract and must have default ctor(it doesn't).
Final step which I dislike is creating a class which has only those two properites(Code, Description), now... it works great, but I don't want to write a class for each query like that.
Ideas?
This is a no-solution answer.
I think whatever you try, you can't do that. Even if you define your own mutable generic Tuple, it will failed since the name of the property must match the name of the column:
SqlQuery(String, Object[]): Creates a raw SQL query that will
return elements of the given generic type. The type can be any type
that has properties that match the names of the columns returned from
the query, or can be a simple primitive type.
I think the best you can do is creating your own generic method for querying the database via classic Command and ExecuteReader pattern. Untested, but you get the idea:
public static IEnumerable<Tuple<T>> SqlQuery<T>(this DbContext context, string sql)
{
using(var connection = new SqlConnection(context.Database.Connection.ConnectionString))
using (var command = new SqlCommand(sql, connection))
{
var reader = command.ExecuteReader();
while (reader.NextResult())
{
yield return new Tuple<T>((T)reader[0]);
}
}
}
public static IEnumerable<Tuple<T1, T2>> SqlQuery<T1, T2>(this DbContext context, string sql)
{
using (var connection = new SqlConnection(context.Database.Connection.ConnectionString))
using (var command = new SqlCommand(sql, connection))
{
var reader = command.ExecuteReader();
while (reader.NextResult())
{
yield return new Tuple<T1, T2>((T1)reader[0], (T2)reader[1]);
}
}
}