Develop Reference

How to use database sharding with EF Core and C#"

I'm currently in the process of converting my 6 years old C# application to .NET Core v3 and EF Core (and also using Blazor).
Most of it is working except for the Sharding part.
Our application creates a new database for each client. We use more or less this code for it: https://learn.microsoft.com/en-us/azure/sql-database/sql-database-elastic-scale-use-entity-framework-applications-visual-studio
I'm now trying to convert it to EF Core, but get stuck at this part:
// C'tor to deploy schema and migrations to a new shard
protected internal TenantContext(string connectionString)
: base(SetInitializerForConnection(connectionString))
{
}
// Only static methods are allowed in calls into base class c'tors
private static string SetInitializerForConnection(string connnectionString)
{
// We want existence checks so that the schema can get deployed
Database.SetInitializer<TenantContext<T>>(new CreateDatabaseIfNotExists<TenantContext<T>>());
return connnectionString;
}
// C'tor for data dependent routing. This call will open a validated connection routed to the proper
// shard by the shard map manager. Note that the base class c'tor call will fail for an open connection
// if migrations need to be done and SQL credentials are used. This is the reason for the
// separation of c'tors into the DDR case (this c'tor) and the internal c'tor for new shards.
public TenantContext(ShardMap shardMap, T shardingKey, string connectionStr)
: base(CreateDDRConnection(shardMap, shardingKey, connectionStr), true /* contextOwnsConnection */)
{
}
// Only static methods are allowed in calls into base class c'tors
private static DbConnection CreateDDRConnection(ShardMap shardMap, T shardingKey, string connectionStr)
{
// No initialization
Database.SetInitializer<TenantContext<T>>(null);
// Ask shard map to broker a validated connection for the given key
var conn = shardMap.OpenConnectionForKey<T>(shardingKey, connectionStr, ConnectionOptions.Validate);
return conn;
}
The above code doesn't compile because the Database object doesn't exist in this way in EF Core.
I assume I can simplify it using TenantContext.Database.EnsureCreated(); somewhere. But I can't figure out how to modify the methods, which to remove, which to change (and how).
Of course, I've been searching for an example using sharding and EF Core but couldn't find it.
Does anybody here has done this before in EF Core and is willing the share?
I'm specifically looking for what to put in startup.cs and how to create a new sharding/database when I create a new client.

In EF.Core just resolve the shard in OnConfiguring. EG
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
var con = GetTenantConnection(this.tenantName);
optionsBuilder.UseSqlServer(con,o => o.UseRelationalNulls());
base.OnConfiguring(optionsBuilder);
}
Note that if you have a service or factory that returns open DbConnections, then you'll need to Close()/Dispose() them in the DbContext.Dispose(). If you get a connection string or a closed connection then DbContext will take care of closing the connection.
ASP.NET Core best-practices probably call for injecting an ITenantConfiguration service or somesuch in your DbContext. But the pattern is the same. Just save the injected service instance to a DbContext field and use it in OnConfiguring.

With the app that I'm working on, the desired shard is not discoverable until request time (for example, knowing what user is making the request, and then routing that user to their database). This meant that the OnConfiguring solution proposed above was not viable.
I worked around this by using IDbContextFactory<TContext>, and defining an extension on top of it, which sets the connection string based on whatever you want. I believe the database connection is created lazily in EF, and you are able to set the connection string up until the EF first needs to actually connect to the database.
In my case, it looked something like this:
var dbContext = _dbContextFactory.CreateDbContext();
var connectionString = $"DataSource={_sqlliteDirectory}/tenant_{tenant.TenantId}.db";
dbContext.Database.SetConnectionString(connectionString);
The downside is that it breaks the database abstraction (this code knows that my database is a local sqllite instance). An abstraction was not necessary in this layer of my app, but it is something very solvable if it's required.

.net Core 2, EF and Multi Tenancy - Dbcontext switch based on user

I have the (almost) worst of multi tenancy. I'm building a asp.net core website that I'm porting a bunch of pokey little intranet sites to. Each subsite will be an asp.net Area. I have an IdentityContext for the Identity stuff. I have multiple copies of vendor databases, each of those with multiple tenants. The ApplicationUserclass has an OrgCode property that I want to use to switch the db context.
I can see myself needing something that maps User.OrgCode and Area to a Connection string
There are many partial examples of this on Stack Overflow. I am very confused after an afternoons reading. The core of it seams to be:
remove DI dbcontext ref from the constructor args.
Instantiate the dbcontext in the controller constructor.
Use dbcontext as before.
Am I on the right track?
Any coherent examples?
Edit 2020/07/09
This has unfortunately become more pressing.
The Identity database is tenant agnostic. Every user in Identity has an OrgCode identifier. (Custom user property).
Each server has multi tenancy built in through the use of 'cost centers'. The server has a collection of databases named the same on every server.
core vendor database
custom database where we store our extensions
logs database for our job output
There are also small application specific databases that already use an Org Code to identify a user
Server A - 1 Org Code
Server B - 4 Org Codes
Server C - 3 Org Codes engaged in project, 50+ not yet (mostly small)
Server D - No Org Codes engaged as of now. 80+ on server. (soon)
It is not possible to consolidate all the organisations onto one server. There are legal and technical ramifications. Each server has hundreds of remote transponders reporting to them that would need updating. The data these supply is what our custom jobs work with.
The dream is to continue to use DI in each page, passing in the contexts as required. The context would then be smart enough to pick the correct underlying connection details based on the OrgCode of the username.
I hesitate to use the word proxy because it seems heavily loaded in this space.
Hell, even using a switch statement would be fine if I knew where to put it
Desired effect User from Org XYZ loads page that requires Vendor database, they get the one from the server that XYZ maps to.
Edit 2020/07/13
To tidy up referenceing, I've switched the OrgCode and Server to Enums. The context inheritance is as follows
DbContext
CustLogsContext
public virtual ServerEnum Server
{
get
{
return ServerEnum.None;
}
}
DbSet (etc)
CustLogsServerAContext
public override ServerEnum Server
{
get
{
return ServerEnum.ServerA;
}
}
CustLogsServerBContext (etc)
CustLogsServerCContext (etc)
CustLogsServerDContext (etc)
VendorContext
VendorServerAContext
VendorServerBContext (etc)
VendorServerCContext (etc)
VendorServerDContext (etc)
I've also created a static class OrgToServerMapping that contains a dictionary mapping OrgCodes to Servers. Currently hardcoded, will change eventually to load from config, and add a reload method.
Currently thinking I need a class that collects the contexts Would have a Dictionary<serverEnum, dbcontext> and be registered as a service. Pretty sure I'd need a version of the object for each inherited dbcontext, unless someone knows ome polymorphic trick I can use

I work on a similar system with thousands of databases, but with LinqToSql instead of EF (I know...). Hopefully the general ideas translate. There are connection pool fragmentation issues that you have to contend with if you end up with many databases, but for just your four databases you won't have to worry about that.
I like these two approaches - they both assume that you can set up the current ApplicationUser to be injected via DI.
Approach #1: In Startup, configure the DI that returns the data context to get the current user, then use that user to build the correct data context. Something like this:
// In Startup.ConfigureServices
services.AddScoped<ApplicationUser>((serviceProvider) =>
{
// something to return the active user however you're normally doing it.
});
services.AddTransient<CustLogsContext>((serviceProvider) =>
{
ApplicationUser currentUser = serviceProvider.GetRequiredService<ApplicationUser>();
// Use your OrgToServerMapping to create a data context
// with the correct connection
return CreateDataContextFromOrganization(currentUser.OrgCode);
});
Approach #2: Rather than injecting the CustLogsContext directly, inject a service that depends on the active user that is responsible for building the data context:
// In Startup.ConfigureServices
services.AddScoped<ApplicationUser>((serviceProvider) =>
{
// something to return the active user however you're normally doing it.
});
services.AddTransient<CustLogsContextWrapper>();
// In its own file somewhere
public class CustLogsContextWrapper
{
private ApplicationUser currentUser;
public CustLogsContextWrapper(ApplicationUser currentUser)
{
this.currentUser = currentUser;
}
public CustLogsContext GetContext()
{
// use your OrgToServerMapping to create a data context with the correct connection;
return CreateDataContextFromOrganization(user.OrgCode);
}
}
Personally I prefer the latter approach, because it avoids a call to a service locator in Startup, and I like encapsulating away the details of how the data context is created. But if I already had a bunch of code that gets the data context directly with DI, the first one would be fine.

I have created a multitenancy implementation as follows (which could scale endlessly in theorie). Create a multitenancy database (say tenantdb). Easy. But the trick is to store connectionstring details for each tenant (your target databases). Along side your user orgCode etc.
I can see myself needing something that maps User.OrgCode and Area to a Connection string
So the way to map it in code is to feed your dbcontext whith your target tenant connectionstring, which you get from your tenantdb. So you would need anohter dbcontext for you tenantdb. So first call your tenantdb get the correct tenant connectionstring by filtering with your user orgcode. And then use it to create a new target dbcontext.
The dream is to continue to use DI in each page, passing in the contexts as required. The context would then be smart enough to pick the correct underlying connection details based on the OrgCode of the username.
I have this working with DI.
I created UI elements for crud operations for this tenantdb, so I can update delete add connection string details and other needed data. The Password is encrypted on save and decrypted on the get just before passing to your target dbcontext.
So I have two connection strings in my config file. One for the tenantdb and one for a default target db. Which can be an empty/dummy one, as you probably encounter application startup errors thrown by your DI code if you don't have one, as it will most likely auto search for a connectionstring.
I also have switch code. This is where a user can switch to anohter tenant. So here the user can choose from all the tenants it has rights to (yes rights are stored in tenantdb). And this would again trigger the code steps described above.
Cheers.
Took this Razor Pages tutorial as my starting point.
This way you can have very lousily coupled target databases. The only overlap could be the User ID. (or even some token from Azure,Google,AWS etc)
Startup.
public class Startup
{
public Startup(IConfiguration configuration)
{
Configuration = configuration;
}
public IConfiguration Configuration { get; }
// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
services.AddRazorPages();
services.AddDbContext<TenantContext>(options =>
options.UseSqlServer(Configuration.GetConnectionString("TenantContext")));
//your dummy (empty) target context.
services.AddDbContext<TargetContext>(options =>
options.UseSqlServer(Configuration.GetConnectionString("TargetContext")));
}
IndexModel (Tenant pages).
public class IndexModel : PageModel
{
private readonly ContosoUniversity.Data.TenantContext _context;
private ContosoUniversity.Data.TargetContext _targetContext;
public IndexModel(ContosoUniversity.Data.TenantContext context, ContosoUniversity.Data.TargetContext targetContext)
{
_context = context;
//set as default targetcontext -> dummy/empty one.
_targetContext = targetContext;
}
public TenantContext Context => _context;
public TargetContext TargetContext { get => _targetContext; set => _targetContext = value; }
public async Task OnGetAsync()
{
//get data from default target.
var student1 = _targetContext.Students.First();
//or
//switch tenant
//lets say you login and have the users ID as guid.
//then return list of tenants for this user from tenantusers.
var ut = await _context.TenantUser.FindAsync("9245fe4a-d402-451c-b9ed-9c1a04247482");
//now get the tenant(s) for this user.
var SelectedTentant = await _context.Tenants.FindAsync(ut.TenantID);
DbContextOptionsBuilder<TargetContext> Builder = new DbContextOptionsBuilder<TargetContext>();
Builder.UseSqlServer(SelectedTentant.ConnectionString);
_targetContext = new TargetContext(Builder.Options);
//now get data from the switched to database.
var student2 = _targetContext.Students.First();
}
}
Tenant.
public class Tenant
{
public int TenantID { get; set; }
public string Name { get; set; }
//probably could slice up the connenctiing string into props.
public string ConnectionString { get; set; }
public ICollection<TenantUser> TenantUsers { get; set; }
}
TenantUser.
public class TenantUser
{
[Key]
public Guid UserID { get; set; }
public string TenantID { get; set; }
}
Default connstrings.
{ "AllowedHosts": "*",
"ConnectionStrings": {
"TenantContext": "Server=(localdb)\mssqllocaldb;Database=TenantContext;Trusted_Connection=True;MultipleActiveResultSets=true",
"TargetContext": "Server=(localdb)\mssqllocaldb;Database=TargetContext;Trusted_Connection=True;MultipleActiveResultSets=true"
}

How to DI an EF 6 Service with dynamic connection string

Our DB is setup with one admin DB and multiple client DBs. When the user logs in (via the admin DB), we determine which client DB they should use and return that connection string for future use (store in the session).
I really like the way this answer deals with working with EF6, and would like to implement something like this, but am not sure how I can use DI and update the connection string as well.
The actual update of the connection string on the DbContext is straight forward using DbConnection. It's just the ability to cleanly create and update the DbContext via some DI mechanism that I am not seeing at the moment.

DI is compile time process, however you can do some decisions at runtime using factory pattern to instantiate a class(from a set of classes that inherit from same interface) depending on some condition
as a solution for your problem you can use a static function to return the appropriate connection string, no DI is required
public static class ConnectionStringPorvider
{
public static string getConnectionString()
{
switch(dbName)
{
....
}
}
}
and in your data context class
base(global::[class name space].ConnectionStringPorvider.getConnectionString(dbName), mappingSource)

Entity framework open connections

We have a site running on ASP.NET 4.5 using a (shared) SQL Server database. Our ISP has recently told us that we have too many open connections and it is affecting the overall stability of the server. At high loads this number can reach as many as 400+ open connections with around 200 concurrent users visiting the site.
The site is made up mostly of content such as reviews and blog posts and we have implemented caching throughout the site. We are using Entity Framework exclusively to query the database. We use StructureMap to inject our connection string into our business logic classes.
StructureMap initialization:
var ecsbuilder = new EntityConnectionStringBuilder();
ecsbuilder.Provider = "System.Data.SqlClient";
ecsbuilder.ProviderConnectionString = #"data source=***;initial catalog=***;persist security info=True;User ID=***;Password=***;multipleactiveresultsets=True;App=EntityFramework";
ecsbuilder.Metadata = #"res://*/Data.***.csdl|res://*/Data.***.ssdl|res://*/Data.***.msl";
string connectionString = ecsbuilder.ToString();
For<SiteModelContainer>().Use<SiteModelContainer>().Ctor<string>("connectionString").Is(connectionString);
For<IGalleryService>().Use<GalleryService>();
...all the rest of our services
Then in our services (e.g. GalleryService) we have the following:
private readonly SiteModelContainer _context;
public GalleryService(SiteModelContainer context)
{
this._context = context;
}
In our service methods we just use _context to query the database
My questions are:
If this is the only way we are querying the database is it not closing the connections for some reason?
What methods can I use to determine where the connections are being left open?
Thanks

Use connection pooling in your connection string.
Use using block for the use of context as below.
//Create a class and static function to get the context
public class DaatabaseFramework
{
public static SiteModelContainer GetContext()
{
return new SiteModelContainer();
}
}
//Use the below to use the context
using (var context=DaatabaseFramework.GetContext())
{
}

I was see similar when I try to pass IQueryable variable from using( ... connection ... ) {} block.
When I do this, orphaned connections was stay on my site like yours.
Try to find code like this:
public IQueryable<YourClass> func(int param1) {
using(var context=DaatabaseFramework.GetContext()) {
return context.MyTable.Where(_=>_.Param1==Param1);
}
}

Entity Framework Code First without app.config

I hope somebody is able to help me, because it seems I'm totally stuck.
For upcoming projects in our company we'd like to use Entity Framework 5 with an code first approach. I played around a little while and everytime I try to use EF with our existing libraries, I fail because it seems EF heavily relies on an existing app.config.
In our company, we have an inhouse database library that allows us to connect to various data sources and database technologies taking the advantages of MEF (managed extensibility framework) for database providers. I just have to pass some database settings, such as host (or file), catalog, user credentials and a database provider name, the library looks for the appropriate plugin and returns me a custom connection string or IDbConnection.
We'd like to use this library together with EF because it allows us to be flexible about which database we use also change the database at runtime.
So. I saw that a typical DbContext object takes no parameters in the constructor. It automatically looks for the appropriate connection string in app.config. We don't like such things so I changed the default constructor to take a DbConnection object that get's passed to the DbContext base class. No deal.
Problems occur when the code first model changes. EF automatically notices this and looks for migration classes / configuration. But: A typical migration class requires a default parameterless constructor for the context! What a pity!
So we build our own migration class using the IDbContextFactory interface. But again, it seems that also this IDbContextFactory needs a parameterless constructor, otherwise I'm not able to add migrations or update the database.
Further, I made my own data migration configurator where I pass the context, also the target database. Problem is here: It doesn't find any migration classes, no matter what I try.
I'm completely stuck because it seems the only way to use EF is when connection strings are saved in app.config. And this is stupid because we need to change database connections at runtime, and app.config is read-only for default users!
How to solve this?

The answer is provided here
https://stackoverflow.com/a/15919627/941240
The trick is to slightly modify the default MigrateDatabaseToLatestVersion initializer so that:
the database is always initialized ...
... using the connection string from current context
The DbMigrator will still create a new data context but will copy the connection string from yours context according to the initializer. I was even able to shorten the code.
And here it goes:
public class MasterDetailContext : DbContext
{
public DbSet<Detail> Detail { get; set; }
public DbSet<Master> Master { get; set; }
// this one is used by DbMigrator - I am NOT going to use it in my code
public MasterDetailContext()
{
Database.Initialize( true );
}
// rather - I am going to use this, I want dynamic connection strings
public MasterDetailContext( string ConnectionString ) : base( ConnectionString )
{
Database.SetInitializer( new CustomInitializer() );
Database.Initialize( true );
}
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.Conventions.Remove<PluralizingTableNameConvention>();
}
}
public class CustomInitializer : IDatabaseInitializer<MasterDetailContext>
{
#region IDatabaseInitializer<MasterDetailContext> Members
// fix the problem with MigrateDatabaseToLatestVersion
// by copying the connection string FROM the context
public void InitializeDatabase( MasterDetailContext context )
{
Configuration cfg = new Configuration(); // migration configuration class
cfg.TargetDatabase = new DbConnectionInfo( context.Database.Connection.ConnectionString, "System.Data.SqlClient" );
DbMigrator dbMigrator = new DbMigrator( cfg );
// this will call the parameterless constructor of the datacontext
// but the connection string from above will be then set on in
dbMigrator.Update();
}
#endregion
}
Client code:
static void Main( string[] args )
{
using ( MasterDetailContext ctx = new MasterDetailContext( #"Database=ConsoleApplication801;Server=.\SQL2012;Integrated Security=true" ) )
{
}
using ( MasterDetailContext ctx = new MasterDetailContext( #"Database=ConsoleApplication802;Server=.\SQL2012;Integrated Security=true" ) )
{
}
}
Running this will cause the two databases to be created and migrated according to the migration configuration.

It needs a parameterless constructor in order to invoke it. What you could do is provide your default DbConntectionFactory in the empty constructor, something like:
public DbContext()
{
IDbContextFactory defaultFactory; //initialize your default here
DbContext(defaultFactory);
}
public DbContext(IDbContextFactory factory)
{
}