Develop Reference

Dapper multilevel nesting

I am trying to use Dapper in my project to speed up data loading (currently using EF6)
Here is my SQL
String SQL = #"select vwArtikli_Grid_V2.ArtikalID
,vwArtikli_Grid_V2.ArtikalNaziv
,Artikli_TagLista.ArtikalTagListaID
,Artikli_TagLista.ArtikalTagID
,Artikli_Stanje.ArtikalStanjeID
,Artikli_Stanje.ObjekatID
,Artikli_Stanje.Stanje
,Artikli_Tagovi.GrupaID
,Artikli_Tagovi.ArtikalTagGrupaID
,Artikli_Tagovi.ArtikalTagNaziv
,Artikli_Tagovi.ArtikalTagPrint
,Artikli_Tagovi.ArtikalTagSlika
,Artikli_Tagovi.ArtikalTagID
,vwArtikli_Grid_V2.ArtikalID
from Artikli_Tagovi
inner join Artikli_TagLista on Artikli_Tagovi.ArtikalTagID = Artikli_TagLista.ArtikalTagID
right outer join vwArtikli_Grid_V2 on Artikli_TagLista.ArtikalID = vwArtikli_Grid_V2.ArtikalID
left outer join Artikli_Stanje on vwArtikli_Grid_V2.ArtikalID = Artikli_Stanje.ArtikalID;
I am using my Entity Framework Entities as POCOs and they are
VwArtikliGridV2, Artikli_TagLista, Artikli_Tagovi, Artikli_Stanje
VwArtikliGridV2 has two properties
public virtual ICollection<Artikli_TagLista> Artikli_TagLista { get; set; }
public virtual ICollection<Artikli_Stanje> Artikli_Stanje { get; set; }
and Artikli_TagLista has
public virtual Artikli_Tagovi Artikli_Tagovi { get; set; }
What is the easiest way to execute the query and map my data to the POCOs or Entities ?
I tried
Dapper.Mapper
var Artikli = cn.Query<VwArtikliGridV2, Artikli_TagLista, Artikli_Stanje, Artikli_Tagovi> (SQL);
but it didnot work
I also tried Slapper.AutoMapper
List<dynamic> ArtikliUM = cn.Query<dynamic>(SQL).ToList();
Slapper.AutoMapper.Configuration.AddIdentifiers(typeof(Artikli_Tagovi), new List<string> { "ArtikalTagID" });
Slapper.AutoMapper.Configuration.AddIdentifiers(typeof(Artikli_TagLista), new List<string> { "ArtikalTagListaID" });
Slapper.AutoMapper.Configuration.AddIdentifiers(typeof(Artikli_Stanje), new List<string> { "ArtikalStanjeID" });
Artikli = (Slapper.AutoMapper.MapDynamic<VwArtikliGridV2>(ArtikliUM) as IEnumerable<VwArtikliGridV2>).ToList();
But it also did not work.
I can map VwArtikliGridV2 but i cant map any of the nested objects. They are always null.
What can I try ?

Dapper maps things flat. At first this feels like a big pain because EF nests things so easily. But once you get over the initial pain you realize how simple, predictable, and performance-oriented it is. Maybe an extra 15 min. of query-crafting for untouchable performance.
I answered a similar question here return a list of data via stored proc to dapper.
You should be able to return multiple datasets (one for the parent item, one for the underlying tags, etc.) and combine them in your app tier.

As BlackjacketMack said in his answer, I think multiple data sets are the way to go.
You might want to look in the Dapper documentation at the features Multi Mapping to split single rows into multiple objects, and QueryMultiple to read multiple result sets from a single query.
Obviously, it would mean modifying your query to return multiple result sets, but could achieve what you are looking for.
Examples can be found here: https://github.com/StackExchange/Dapper/blob/master/Dapper.Tests/MultiMapTests.cs
https://github.com/StackExchange/Dapper/blob/master/Dapper.Tests/QueryMultipleTests.cs
In particular, you might want to take a look at method public void TestMultiMapThreeTypesWithGridReader() in MultiMapTests.cs. I found that very useful to help me understand a similar problem of parent with child collections, where the child contains different object types returned in a single row.

c# Linq to Sql dynamic Data Context assignment

`Hi,
Can somebody please give me a pointer on this? I have 8 servers each with 8 databases which look identical exept server/database name. We are talking thousands of tables.
I create my data contexts with sqlmetal.exe
After creating my data contexts, I import them into the application and then I run comparison scripts over the databases to compare results.
My problem is dynamically switching between data contexts.
Datacontext.DAL.DUK1 duk1sdi = new Datacontext.DAL.DUK1(connectionString);
Datacontext.DAL.DUK3 duk3sdi = new Datacontext.DAL.DUK3(connectionString);
string fromOne = runQuery(duk1sdi);
string fromThree = runQuery(duk3sdi);
public static string runQuery(DataContext duk)
{
var query =
from result in duk.TableA
select result.Total;
string returnString = query;
return returnString;
}
I have no problem with the query running when the duk is predefined, however how do I define and pass the datacontext to the function?
The error I get is:
Error 1 'System.Data.Linq.DataContext' does not contain a definition
for 'TableA' and no extension method 'TableA' accepting a first
argument of type 'System.Data.Linq.DataContext' could be found (are
you missing a using directive or an assembly reference?)

You could use the GetTable<T> method, where T is the type of the table, e.g. TableA.
public static string runQuery(DataContext duk) {
var table = duk.GetTable<TableA>();
var query = from result in table select result.Total;
...
}
However, all types of TableA will need to be the same type, strictly (I'm pretty sure).
Otherwise you would need to literally branch the logic for the handling of each context. Since you can extend your DataContext instances (in general, maybe not in your specific case) then you could have them share an interface that exposes a collection property of TableA, but you would need a higher level context wrapper to pass around then - unless you pass around the collection by altering the method signature.

You can use interfaces. Check this answer, but be sure to script the interfaces using a .tt file with the amount of tables you have.
Edit:
If you have generated contexts which you want to use interchangeably in a reusable method, you have the problem that the generated TableA classes are not reusable, since they are different types (even though the names may match, but that doesn't make them equal). Therefore you need to abstract the actual types, and one way to do this, is to use interfaces. You build your reusable method around an interface which abstracts the specific context-type and table-type. The downside is that you have to implement the interfaces on the generated contexts and tabletypes. This though is something you can solve using a .tt script.
Pseudo code:
// Define interface for table
public interface ITableA {
// ... properties
}
// Define interface for context
public interface IMyContext {
IQueryable<ITableA> TableA { get; }
}
// Extend TableA from DUK1
public partial class TableA: ITableA {
}
// Extend DUK1
public partial class Datacontext.DAL.DUK1: IMyContext {
IQueryable<ITableA> IMyContext.TableA {
get { return TableA; }
}
}
// Same for DUK3 and TableA FROM DUK3
// Finally, your code
Datacontext.DAL.DUK1 duk1sdi = new Datacontext.DAL.DUK1(connectionString);
Datacontext.DAL.DUK3 duk3sdi = new Datacontext.DAL.DUK3(connectionString);
string fromOne = runQuery(duk1sdi);
string fromThree = runQuery(duk3sdi);
public static string runQuery(IMyContext duk) {
// Note: method accepts interface, not specific context type
var query = from result in duk.TableA
select result.Total;
string returnString = query;
return returnString;
}

If your schema is identical between databases, why script the dbml for all of them? Just create one context with it's associated classes and dynamically switch out the connection string when instantiating the context.
var duk1sdi = new Datacontext.DAL.DUK1(connectionString1);
var duk3sdi = new Datacontext.DAL.DUK1(connectionString2);

Thanks, guys, I think I found the simplist solution for me based a bit of both your answers and by RTFM (Programming Microsoft Linq in Microsoft .NET Framework 4 by Paulo Pialorsi and Marco Russo)
In this way I don't have to use the large DBML files. It is a shame because I'm going to have to create hundreds of tables in this way, but I can now switch between connection strings on the fly.
First I create the table structure. (outside the program code block)
[Table(Name = "TableA")]
public class TableA
{
[Column] public int result;
}
Then I define the table for use:
Table<TableA> TableA = dc.GetTable<TableA>();
And then I can query from it:
var query =
from result in TableA
select TableA.result;

Is there a way to call DbContext.Translate on a collection of results without explicity passing in there generic types?

I have a stored procedure that I am calling through an entity framework that returns multiple results. I have been using the DbContext.Translate method to map these results back to entities like this:
List<T1> result1 = ObjectContext.Translate<T1>(reader).ToList();
reader.NextResult();
List<T2> result2 = ObjectContext.Translate<T2>(reader).ToList();
reader.NextResult();
List<T3> result3 = ObjectContext.Translate<T3>(reader).ToList();
reader.NextResult();
This works great, but now I have a procedure that is returning over 20 results. I want to be able to map the results generically without creating 20 overloads of my method (since I foresee more stored procedures returning 2-20 result set in the future).
Is there some way that I could do this mapping in a loop without having many method overloads?

Is there some way that I could do this mapping in a loop without
having many method overloads?
I would say, no.
I think what you have described already is the best way of doing this, considering you have to tell the ObjectContext what types it's dealing with.
You could create a set of generic functions (similar to Func<T> and Action<T>) with multiple overloads containing an increasing number of generic parameters.
For example, the first two implementations could look like the following:
public static Tuple<List<T1>> GetSprocResults<T1>
(DataTableReader reader, ObjectContext objectContext)
{
var t1 = objectContext.Translate<T1>(reader).ToList();
return new Tuple<List<T1>>(t1);
}
public static Tuple<List<T1>, List<T2>> GetSprocResults<T1, T2>
(DataTableReader reader, ObjectContext objectContext)
{
var t1 = objectContext.Translate<T1>(reader).ToList();
reader.NextResult();
var t2 = objectContext.Translate<T2>(reader).ToList();
return new Tuple<List<T1>, List<T2>>(t1, t2);
}
Example usage:
var result = GetSprocResults<DateTime, int>(reader, objectContext);
var datesCount = result.Item1.Count;
var intCount = result.Item2.Count;
I don't think there is a problem with having multiple overloads if it makes sense in the context of your code to re-factor this functionality to a set of methods.

Retrieve an object from entityframework without ONE field

I'm using entity framework to connect with the database. I've one little problem:
I've one table which have one varbinary(MAX) column(with filestream).
I'm using SQL request to manage the "Data" part, but EF for the rest(metadata of the file).
I've one code which has to get all files id, filename, guid, modification date, ... of a file. This doesn't need at all the "Data" field.
Is there a way to retrieve a List but without this column filled?
Something like
context.Files.Where(f=>f.xyz).Exclude(f=>f.Data).ToList();
??
I know I can create anonymous objects, but I need to transmit the result to a method, so no anonymous methods. And I don't want to put this in a list of anonymous type, and then create a list of my non-anonymous type(File).
The goal is to avoid this:
using(RsSolutionsEntities context = new RsSolutionsEntities())
{
var file = context.Files
.Where(f => f.Id == idFile)
.Select(f => new {
f.Id, f.MimeType, f.Size, f.FileName, f.DataType,
f.DateModification, f.FileId
}).FirstOrDefault();
return new File() {
DataType = file.DataType, DateModification = file.DateModification,
FileId = file.FileId, FileName = file.FileName, Id = file.Id,
MimeType = file.MimeType, Size = file.Size
};
}
(I'm using here the anonymous type because otherwise you will get a NotSupportedException: The entity or complex type 'ProjectName.File' cannot be constructed in a LINQ to Entities query.)
(e.g. this code throw the previous exception:
File file2 = context.Files.Where(f => f.Id == idFile)
.Select(f => new File() {Id = f.Id, DataType = f.DataType}).FirstOrDefault();
and "File" is the type I get with a context.Files.ToList(). This is the good class:
using File = MyProjectNamespace.Common.Data.DataModel.File;
File is a known class of my EF datacontext:
public ObjectSet<File> Files
{
get { return _files ?? (_files = CreateObjectSet<File>("Files")); }
}
private ObjectSet<File> _files;

Is there a way to retrieve a List but without this column filled?
Not without projection which you want to avoid. If the column is mapped it is natural part of your entity. Entity without this column is not complete - it is different data set = projection.
I'm using here the anonymous type because otherwise you will get a
NotSupportedException: The entity or complex type 'ProjectName.File'
cannot be constructed in a LINQ to Entities query.
As exception says you cannot project to mapped entity. I mentioned reason above - projection make different data set and EF don't like "partial entities".
Error 16 Error 3023: Problem in mapping fragments starting at line
2717:Column Files.Data in table Files must be mapped: It has no
default value and is not nullable.
It is not enough to delete property from designer. You must open EDMX as XML and delete column from SSDL as well which will make your model very fragile (each update from database will put your column back). If you don't want to map the column you should use database view without the column and map the view instead of the table but you will not be able to insert data.
As a workaround to all your problems use table splitting and separate the problematic binary column to another entity with 1 : 1 relation to your main File entity.

I'd do something like this:
var result = from thing in dbContext.Things
select new Thing {
PropertyA = thing.PropertyA,
Another = thing.Another
// and so on, skipping the VarBinary(MAX) property
};
Where Thing is your entity that EF knows how to materialize. The resulting SQL statement shouldn't include the large column in its result set, since it's not needed in the query.
EDIT: From your edits, you get the error NotSupportedException: The entity or complex type 'ProjectName.File' cannot be constructed in a LINQ to Entities query. because you haven't mapped that class as an entity. You can't include objects in LINQ to Entities queries that EF doesn't know about and expect it to generate appropriate SQL statements.
You can map another type that excludes the VarBinary(MAX) column in its definition or use the code above.

you can do this:
var files = dbContext.Database.SqlQuery<File>("select FileId, DataType, MimeType from Files");
or this:
var files = objectContext.ExecuteStoreQuery<File>("select FileId, DataType, MimeType from Files");
depending on your version of EF

I had this requirement because I have a Document entity which has a Content field with the content of the file, i.e. some 100MB in size, and I have a Search function that I wanted to return the rest of the columns.
I chose to use projection:
IQueryable<Document> results = dbContext.Documents.Include(o => o.UploadedBy).Select(o => new {
Content = (string)null,
ContentType = o.ContentType,
DocumentTypeId = o.DocumentTypeId,
FileName = o.FileName,
Id = o.Id,
// etc. even with related entities here like:
UploadedBy = o.UploadedBy
});
Then my WebApi controller passes this results object to a common Pagination function, which applies a .Skip, .Take and a .ToList.
This means that when the query gets executed, it doesn't access the Content column, so the 100MB data is not being touched, and the query is as fast as you'd want/expect it to be.
Next, I cast it back to my DTO class, which in this case is pretty much exactly the same as the entity class, so this might not be a step you need to implement, but it's follows my typical WebApi coding pattern, so:
var dtos = paginated.Select(o => new DocumentDTO
{
Content = o.Content,
ContentType = o.ContentType,
DocumentTypeId = o.DocumentTypeId,
FileName = o.FileName,
Id = o.Id,
UploadedBy = o.UploadedBy == null ? null : ModelFactory.Create(o.UploadedBy)
});
Then I return the DTO list:
return Ok(dtos);
So it uses projection, which might not fit the original poster's requirements, but if you're using DTO classes, you're converting anyway. You could just as easily do the following to return them as your actual entities:
var dtos = paginated.Select(o => new Document
{
Content = o.Content,
ContentType = o.ContentType,
DocumentTypeId = o.DocumentTypeId,
//...
Just a few extra steps but this is working nicely for me.

For EF Core 2
I implemented a solution like this:
var files = context.Files.AsNoTracking()
.IgnoreProperty(f => f.Report)
.ToList();
The base idea is to turn for example this query:
SELECT [f].[Id], [f].[Report], [f].[CreationDate]
FROM [File] AS [f]
into this:
SELECT [f].[Id], '' as [Report], [f].[CreationDate]
FROM [File] AS [f]
you can see the full source code in here:
https://github.com/aspnet/EntityFrameworkCore/issues/1387#issuecomment-495630292

I'd like to share my attempts to workaround this problem in case somebody else is in the same situation.
I started with what Jeremy Danyow suggested, which to me is the less painful option.
// You need to include all fields in the query, just make null the ones you don't want.
var results = context.Database.SqlQuery<myEntity>("SELECT Field1, Field2, Field3, HugeField4 = NULL, Field5 FROM TableName");
In my case, I needed a IQueryable<> result object so I added AsQueryable() at the end. This of course let me add calls to .Where, .Take, and the other commands we all know, and they worked fine. But there's a caveat:
The normal code (basically context.myEntity.AsQueryable()) returned a System.Data.Entity.DbSet<Data.DataModel.myEntity>, while this approach returned System.Linq.EnumerableQuery<Data.DataModel.myEntity>.
Apparently this means that my custom query gets executed "as is" as soon as needed and the filtering I added later is done afterwards and not in the database.
Therefore I tried to mimic Entity Framework's object by using the exact query EF creates, even with those [Extent1] aliases, but it didn't work. When analyzing the resulting object, its query ended like
FROM [dbo].[TableName] AS [Extent1].Where(c => ...
instead of the expected
FROM [dbo].[TableName] AS [Extent1] WHERE ([Extent1]...
Anyway, this works, and as long as the table is not huge, this method will be fast enough. Otherwise you have no option than to manually add the conditions by concatenating strings, like classic dynamic SQL. A very basic example in case you don't know what I'm talking about:
string query = "SELECT Field1, Field2, Field3, HugeField4 = NULL, Field5 FROM TableName";
if (parameterId.HasValue)
query += " WHERE Field1 = " + parameterId.Value.ToString();
var results = context.Database.SqlQuery<myEntity>(query);
In case your method sometimes needs this field, you can add a bool parameter and then do something like this:
IQueryable<myEntity> results;
if (excludeBigData)
results = context.Database.SqlQuery<myEntity>("SELECT Field1, Field2, Field3, HugeField4 = NULL, Field5 FROM TableName").AsQueryable();
else
results = context.myEntity.AsQueryable();
If anyone manages to make the Linq extensions work properly like if it was the original EF object, please comment so I can update the answer.

I'm using here the anonymous type because otherwise you will get a
NotSupportedException: The entity or complex type 'ProjectName.File'
cannot be constructed in a LINQ to Entities query.
var file = context.Files
.Where(f => f.Id == idFile)
.FirstOrDefault() // You need to exeucte the query if you want to reuse the type
.Select(f => new {
f.Id, f.MimeType, f.Size, f.FileName, f.DataType,
f.DateModification, f.FileId
}).FirstOrDefault();
And also its not a bad practice to de-normalize the table into further, i.e one with metadata and one with payload to avoid projection. Projection would work, the only issue is, need to edit any time a new column is added to the table.

I tried this:
From the edmx diagram (EF 6), I clicked the column I wanted to hide from EF and on their properties you can set their getter and setter to private. That way, for me it works.
I return some data which includes a User reference, so I wanted to hide the Password field even though it's encrypted and salted, I just didn't want it on my json, and I didn't want to do a:
Select(col => new {})
because that's a pain to create and maintain, especially for big tables with a lot of relationships.
The downside with this method is that if you ever regenerate your model, you would need to modify their getter and setter again.

Using Entity Framework Power Tools you can do the following in efpt.config.json:
"Tables": [
{
"ExcludedColumns": [
"FileData"
],
"Name": "[dbo].[Attachment]",
"ObjectType": 0
}
]

How to return query results from method that uses LINQ to SQL

Here is the code I'm working with, I'm still a bit new to LINQ, so this is a work in progress. Specifically, I'd like to get my results from this query (about 7 columns of strings, ints, and datetime), and return them to the method that called the method containing this LINQ to SQL query. A simple code example would be super helpful.
using (ormDataContext context = new ormDataContext(connStr))
{
var electionInfo = from t1 in context.elections
join t2 in context.election_status
on t1.statusID equals t2.statusID
select new { t1, t2 };
}
(In this case, my query is returning all the contents of 2 tables, election and election_status.)

Specifically, I'd like to get my
results from this query (about 7
columns of strings, ints, and
datetime), and return them
Hi, the problem you've got with your query is that you're creating an anonymous type. You cannot return an anonymous type from a method, so this is where you're going to have trouble.
What you will need to do is to create a "wrapper" type that can take an election and an election_status and then return those.
Here's a little sample of what I'm talking about; as you can see I declare a Tuple class. The method that you will wrap your query in returns an IEnumerable.
I hope this helps :-)
class Tuple
{
Election election;
Election_status election_status;
public Tuple(Election election, Election_status election_status)
{
this.election = election;
this.election_status = election_status;
}
}
public IEnumerable<Tuple> getElections()
{
IEnumerable<Tuple> result = null;
using (ormDataContext context = new ormDataContext(connStr))
{
result = from t1 in context.elections
join t2 in context.election_status
on t1.statusID equals t2.statusID
select new Tuple(t1, t2);
}
}
UPDATE
Following from NagaMensch's comments, a better way to achieve the desired result would be to use the built in LINQ to SQL associations.
If you go to your entity diagram and click on toolbox, you will see 3 options. Class, Association and Inheritance. We want to use Association.
Click on Association and click on the ElectionStatus entity, hold the mouse button down and it will allow you to draw a line to the Election entity.
Once you've drawn the line it will ask you which properties are involved in the association. You want to select the StatusId column from the Election entity, and the StatusId column from the ElectionStatus entity.
Now that you've completed your mapping you will be able to simplify your query greatly because the join will not be necessary. You can just access the election status via a brand new property that LINQ to SQL will have added to the Election entity.
Your code can now look like this:
//context has to be moved outside the function
static ExampleDataContext context = new ExampleDataContext();
//Here we can return an IEnumerable of Election now, instead of using the Tuple class
public static IEnumerable<Election> getElections()
{
return from election in context.Elections
select election;
}
static void Main(string[] args)
{
//get the elections
var elections = getElections();
//lets go through the elections
foreach (var election in elections)
{
//here we can access election status via the ElectionStatus property
Console.WriteLine("Election name: {0}; Election status: {1}", election.ElectionName, election.ElectionStatus.StatusDescription);
}
}
You can also find a "how to" on LINQ to SQL associations here.
Note: It's worth mentioning that if you have an FK relationship set up between your tables in the database; LINQ to SQL will automatically pick the relationship up and map the association for you (therefore creating the properties).

You'll need to create classes that have the same structure as the anonymous types. Then, instead of "new { t1, t2 }", you use "new MyClass(t1, t2)".
Once you have a named class, you can pass it all over the place as you were hoping.

The problem is, that you are creating a anonymous type, hence there is no way to declare a method with this return type. You have to create a new type that will hold your query result and return this type.
But I suggest not to return the result in a new type but return just a colection of election objects and access the election_status objects through the relation properties assuming you included them in your model. The data load options cause the query to include the related election status objects in the query result.
public IList<election> GetElections()
{
using (ormDataContext context = new ormDataContext(connStr))
{
DataLoadOptions dlo = new DataLoadOptions();
dlo.LoadWith<election>(e => e.election_status);
context.DeferredLoadingEnabled = false;
context.LoadOptions = dlo;
return context.elections.ToList();
}
}
Now you can do the following.
IList<election> elections = GetElections();
// Access the election status.
Console.WriteLin(elections[0].election_status);
I general LINQ to SQL could just retrieve the related entities on demand - that is called deferred loading.
ormDataContext context = new ormDataContext(connStr));
IList<election> elections = context.elections.ToList();
// This will trigger a query that loads the election
// status of the first election object.
Console.WriteLine(elections[0].election_status);
But this requires you not to close the data context until you finished using the retrieved objects, hence cannot be used with a using statement encapsulated in a method.

IEnumerable<object> getRecordsList()
{
using (var dbObj = new OrderEntryDbEntities())
{
return (from orderRec in dbObj.structSTIOrderUpdateTbls
select orderRec).ToList<object>();
}
}

maybe it can help :)
using (ormDataContext context = new ormDataContext(connStr))
{
var electionInfo = from t1 in context.elections
join t2 in context.election_status
on t1.statusID equals t2.statusID
select new {
t1.column1,
t1.column2,
t1.column3,
t1.column4,
t2.column5,
t2.column6,
t2.column7
};
}
foreach (var ei in electionInfo){
//write what do u want
}

return electionInfo.ToList();

You cannot return an anonymous type from a method. It is only available within the scope in which it is created. You'll have to create a class instead.