Develop Reference

EntityFramework6 memory usage with large amount of table due to InitializedDatabases list

In our application there are a large amount of tables (around 50k) - all of those tables are actually used and this results in a high memory consumption in entity framework.
After some memory profiling I noticed that DbCompiledModel classes were being kept in memory so after some searching tracked it down to the LazyInternalContext class that keeps a list of "InitializedDatabases".
https://github.com/dotnet/ef6/blob/master/src/EntityFramework/Internal/LazyInternalContext.cs#L670
Is there a way to prevent entity framework from doing this?, it's not a code first setup, database setup and migration are not done in this app if that is what the "InitializeDatabaseAction" implies.
Setting a "return null" or setting "InitializerDisabled" to true makes everything work but would rather not run a custom entity build plus don't know what the impact would be to just 'change' the source.
Most tables have the same definition so also tried the solution I found here:
Change table name at runtime
When trying this I'm getting an error "An open data reader exists for this command", using postgres and MARS isn't supported there (no idea why I'd need it, this just changes the sql that's run)

The solution was given in a comment bu Ivan Stoev and works.
There is no way to turn this off without using reflection, setting the "InternalContext.InitializerDisabled" property to true will make this skip the dictionary.
So:
Use a DbContext constructor that provides the DbCachedModel
Use Database.SetInitializer(null);
Set InternalContext.InitializerDisabled = true using reflection
Code from the sample I used to test this, as a test setup I had 1 main table with 30k partitions, the partitions themselves are queried because postgres (especialy 9.x) does not scale well with high number of partitions:
public class PartContext : DbContext {
private static readonly string _ConnectionString = new NpgsqlConnectionStringBuilder {
Host = "localhost",
Port = 5432,
Database = "postgres",
Username = "postgres",
Password = "password"
}.ConnectionString;
public readonly string Table;
public readonly string Partition;
public PartContext(string pMainTable, string pPartition) : base(
new NpgsqlConnection() { ConnectionString = _ConnectionString },
PartDbModelBuilder.Get(_ConnectionString, pPartition),
true
) {
Table = pMainTable;
Partition = pPartition;
Database.SetInitializer<PartContext>(null);
/**
* Disable database initialization so that the DbCachedModels are not kept internally in Entity
* This causes high memory usage when having a lot of tables
* In EF 6.4.2 there was no way to 'manage' that Dictionary externally
*/
try {
var InternalContext = typeof(PartContext).BaseType.GetProperty("InternalContext", BindingFlags.NonPublic | BindingFlags.Instance).GetValue(this, null);
InternalContext.GetType().GetProperty("InitializerDisabled").SetValue(InternalContext, true);
} catch(Exception) { }
}
public DbSet<MyPart> Parts { get; set; }
protected override void OnModelCreating(DbModelBuilder modelBuilder) {
modelBuilder.HasDefaultSchema("public");
modelBuilder.Conventions.Remove<PluralizingTableNameConvention>();
}
}
This provides the DbCachedModels:
I recommend adding some custom caching code etc, this is just from a sample
class PartDbModelBuilder {
public static DbCompiledModel Get(string pConnectionString, string pTable) {
DbModelBuilder builder = new DbModelBuilder();
builder.Entity<MyPart>().ToTable(pTable, "public");
using (var connection = new NpgsqlConnection() { ConnectionString = pConnectionString }) {
var obj = builder.Build(connection).Compile();
return obj;
}
}
}
This is the entity I used as a test:
public class MyPart {
public int id { get; set; }
public string name { get; set; }
public string value { get; set; }
}
Class I used to run the test:
class EFTest {
public void Run(int tableCount) {
int done = 0;
Parallel.For(0, tableCount, new ParallelOptions { MaxDegreeOfParallelism = 5 }, (i) => {
string id = i.ToString().PadLeft(5, '0');
using (var context = new PartContext("mypart", "mypart_" + id)) {
var objResult = context.Parts.First();
Console.WriteLine(objResult.name);
}
done++;
Console.WriteLine(done + " DONE");
});
}
}
Table definition:
CREATE TABLE IF NOT EXISTS mypart (
id SERIAL,
name text,
value text
) partition by list (name);
CREATE TABLE IF NOT EXISTS part partition of mypart_00000 for values in ('mypart00000');
CREATE TABLE IF NOT EXISTS part partition of mypart_00001 for values in ('mypart00001');
CREATE TABLE IF NOT EXISTS part partition of mypart_00002 for values in ('mypart00002');
...
Postgres 9:
CREATE TABLE IF NOT EXISTS mypart (
id SERIAL,
name text,
value text
);
CREATE TABLE IF NOT EXISTS ".$name."( CHECK ( name = 'mypart00000')) INHERITS (mypart);
CREATE TABLE IF NOT EXISTS ".$name."( CHECK ( name = 'mypart00001')) INHERITS (mypart);
CREATE TABLE IF NOT EXISTS ".$name."( CHECK ( name = 'mypart00002')) INHERITS (mypart);
...

Entity Framework: RowVersion value is null

I am using Entity Framework 6.2.0 and a local MSSQL (MDF) database.
I have several types that all descend from my main type "Entity" ("Table per Type" strategy is used). Now, I am trying to implement optimistic locking.
In my EDMX file, I added a property RowVersion to Entity (a fixed length byte array of 8 bytes, in SQL-DB : "[RowVersion] binary(8) NOT NULL") and set the Concurrency mode of that proeprty to "Fixed". I flagged the property inside the Entity class with the "Timestamp" attribute:
[System.ComponentModel.DataAnnotations.Schema.Table("EntitySet", Schema = "RightsManager")]
public partial class Entity
{
public int Id { get; set; }
public string Name { get; set; }
public System.DateTime ActiveFrom { get; set; }
public Nullable<System.DateTime> ActiveUntil { get; set; }
[System.ComponentModel.DataAnnotations.Timestamp]
public byte[] RowVersion { get; set; }
}
I also added code to OnModelCreating of my DBContext descendant to indicate RowVersion to be used:
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
Database.SetInitializer<RightsManagerContext>(null);
base.OnModelCreating(modelBuilder);
modelBuilder.Entity<Entity>().Property(p => p.RowVersion).IsRowVersion();
modelBuilder.Entity<Product>().Property(p => p.RowVersion).IsRowVersion();
}
The problem: Upon insert of a new Product, an SQL error is thrown. This is the unit test i am using:
[TestMethod]
public void TestCreateProduct()
{
using (var context = GetContext())
{
var newProduct = new Product
{
Name = "New product",
ActiveFrom = DateTime.Now
};
context.Entry(newProduct).State = System.Data.Entity.EntityState.Added;
var objectsWritten = context.SaveChanges();
Assert.AreNotEqual(0, objectsWritten);
};
}
The innermost exception thrown:
System.Data.SqlClient.SqlException: Cannot insert the value NULL into column 'RowVersion', table 'P:\VISUAL STUDIO\PROJECTS\RIGHTSMANAGER\DATABASE\RIGHTSMANAGER.MDF.RightsManager.EntitySet'; column does not allow nulls. INSERT fails.
Obviously, EF is not filling in a value automatically, it's handling the field like any other one. What am i missing here?

I think i misunderstood the IsRowVersion/Timestamp thing to be a database-agnostic one. It seems that this whole mechanism only works if using the MSSQL-specific database field type "rowversion" when creating the table. All other databases such as Oracle, DB2 etc are not in scope.
As I am trying to have DBMS neutrality in my project, I will have to manually implement such a feature with "IsConcurrencyToken".

Pass table name to dbContext and get values from table in Entity Framework

I have a DbContext class and I'm using code first apporach in my application. I have few common standard tables that contains "Id" and "Value" columns and i want to query them passing the table name and column name but there is no option in entity framework to pass.
Example:
Common tables:
Client_ClientDepartment (Id, Value)
Client_ClientDesignation (Id, Value)
Client_ClientCompany (Id, Value)
What I want to do is to pass table name and Id to get the value. I have created a common method as
public string GetClientValue(string id, string tableName)
{
DatabaseContext dbContext = new DatabaseContext();
//Query the database and get value based on table and id.
string value = dbContent. ("query here")
return value ;
}
Can I do it in entity framework? Is it possible?

using ( DatabaseContext dbContext = new DatabaseContext())
{
var blogs = dbContext.Database.SqlQuery<>("query here").ToList();
}

I believe you can run a custom query like this
using (var context = new BloggingContext())
{
var blogNames = context.Database.SqlQuery<string>(
"SELECT Name FROM dbo.Blogs").ToList();
}
Source: https://msdn.microsoft.com/en-us/library/jj592907(v=vs.113).aspx
Sorry I had to answer instead of comment, but don't got the badge yet.

Actually, you normally don't pass table and column names in EF.
You have classes and properties, which become tables and columns in the resulting database.
Your context should look something like this:
public class DatabaseContext : DbContext
{
public DatabaseContext(): base(YourWebConfigConnectionStringName){}
public DbSet<Client_ClientDepartment> ClientDepartment { get; set; }
public DbSet<Client_ClientDesignation> ClientDesignation { get; set; }
With this you are basically registering your "table" classes.
You then address them in code like this:
using (var context=new DatabaseContext())
{
var department = context.ClientDepartment.First(d => d.Id == someIdVariable);
Which is the analogy to a SQL query SELECT TOP 1 department WHERE ID=someId
You can also pass SQL statements as described in the other answers, but that too will only work if you properly registered your classes as DBSets in your DatabaseContext class.
P.S: I left out the Database initializer in the DBContext class, which is something you also need in code first.

Is there a way to get a List of all stored procedure names in C# using Entity Framework?

I know that I can run from an opened database connection:
SELECT * FROM SYS.PROCEDURES
Is there a way to do the equivalent with EF 6.1?

You can run raw sql statements directly from the Database Context object.
SomeDatabaseEntities db = new ...();
var items = db.Database.SqlQuery<Something>("SELECT * FROM SYS.PROCEDURES")
You will need to make a class called Something that will map the results. Should be as easy as having the property names match up.
From the documentation of the method:
The type can be any type that has properties that match the names of the columns returned from the query, or can be a simple primitive type.

Looks like you can just manually map that table using code-first and treat it like any other entity:
[Table("procedures", Schema = "sys")]
public class Procedure
{
[Column(Order = 0)]
public string name { get; set; }
[Key]
[Column(Order = 1)]
public int object_id { get; set; }
}
public partial class Model1 : DbContext
{
public Model1()
: base("name=Model1")
{
}
public virtual DbSet<Procedure> Procedures { get; set; }
}
using (var context = new Model1())
{
foreach (var p in context.Procedures)
{
Console.WriteLine("{0}: {1}", p.object_id, p.name);
}
}

Since you're using DB-first, you can wrap a regular view around the system one, then map an entity to it:
create view dbo.procs
as
select * from sys.procedures;
Here you've got a traditional view living in dbo, so EF shouldn't have any trouble with it.

How do I express a "has many through" relationship in Entity Framework 5?

I am attempting to use Entity Framework 5 to query an existing MySQL database. I used code-first to create a code-based model that maps to an existing database following this tutorial on MSDN.
I have two tables: users and buddies. A User has an id, a name and an email. A Buddy has a user_id and a buddy_id. A User has many Buddies (which are also Users). The buddy_id column is a foreign key back into the Users table. So each User has many Users through Buddies.
public class User
{
public int Id { get; set; }
public string Name { get; set; }
public string Email { get; set; }
public IList<User> Buddies { get; set; }
}
Here is my database access code:
using (var db = new Models.fooContext())
{
foreach (Models.user user in db.users)
{
var u = new User
{
Id = user.id,
Name = user.name,
Email = user.email,
Buddies = new List<User>()
};
// this is obviously very inefficient but I am just
// trying to get anything to work
foreach (Models.buddy buddy in db.buddies) // this line throws an exception
{
if (buddy.user_id == user.id)
{
var b = new User();
// take buddy.user_id and find this user in the user table
u.Buddies.Add(b);
}
}
// serialize User and write it to a file here
}
}
This code throws the following exception on the line indicated above:
System.Data.EntityCommandExecutionException: an error occurred while executing the command definition. See the inner exception for details."
The inner exception is a MySql.Data.MySqlClient.MySqlException with the message
There is already an open DataReader associated with this Connection which must be closed first.
My questions:
How do I create a relationship that tells EF that each User has many Buddies?
Once EF understands that a User has many Buddies, how do I use the buddy_id to find the User record for that Buddy?

You have one data type that you are calling separate names. That is a little confusing. However To get this to work with code first you just need the following fluent configuration in your Custom DbContext class:
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.Entity<user>().
HasMany(c => c.Buddies).
WithMany().
Map(
m =>
{
m.MapLeftKey("user_id");
m.MapRightKey("buddy_id");
m.ToTable("buddies");
});
}
This assuming your user class looks like this:
[Table("user")]
public class user
{
public int id { get; set; }
public string name { get; set; }
public string email { get; set; }
public virtual List<user> Buddies { get; set; }
}
If you use the above method every user object you have will have a navigation property on it called Buddies. When querying for users you will want to eager load buddy users, do:
context.users.Include("Buddies")
Further, To address your issue with multiple readers. It is because you have not enumerated the query (db.users) from your first loop. To address that you can enumerate the query like so:
var users = context.users.Include("Buddies").ToList();
foreach(var user in users)....
And if you use the configuration above, you don't need to try and match id's you simply acquire the list (null field if new) of buddies from the user using the buddies navigation property (virtual,lazy loaded), do:
user.Buddies
As you can see you don't really need a 'Model.buddy' (since it only contains an id mapping)
Entity framework will take care of the linking. However if you're not always including the Buddies in your user query you may want the table. Which would be queried with LINQ in the following way:
var userBuddies = db.buddies.Where(buddy=>buddy.user_id == user.id).ToList();
//An enumerated list of user buddies
//do stuff