I am creating a .NET 6 API using EF core with SQL Server, and trying to implement multiple databases that would different connection strings based on an id that is passed in from an identity token. Essentially, there would be multiple databases that each contain the same tables, but storing different information. A user would be able to edit data from the front end and based on the "tenant" that they are working in, it would store the data in the appropriate database. For the time being, all of the databases will be on the same server.
When a user makes a request, I have been able to implement that correctly, using the following inside of my DbContext:
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
if(!optionsBuilder.IsConfigured)
{
var tenantId = _httpContext.GetClaim("tenantId");
optionsBuilder.UseSqlServer(GetClientConnectionString(tenantId));
}
}
where the GetClientConnectionString function would perform some logic to manipulate the default connection string and return the correct value based on that id from the user token. This works fine and when run with the appropriate token, is able to switch connection strings ok.
The part I am unsure of is how to maintain each database individually - is there a good way to run migrations for each of the databases? I know that a connection string can be passed in to the dotnet ef migrations ... command, however if the amount of databases grows to a decent number this does not seem efficient.
Is the best bet to just bite the bullet and perform the migrations manually? Or would it be possible to somehow loop through a collection of keys which would return the connection string value and apply the migration to each?
I am relatively new to anything more than a simple dataset in EF so I am hoping I am not just missing something.
Thanks in advance
I have never done this, but You can try start migration for all dbs with one DbContext(said that the bases are the same), just try to change the connection string for each db. I don't have time to test it, but I hope it will work
foreach (var connection in ConnectionList)
{
var optionBuilder = new DbContextOptionsBuilder<ApplicationDbContext>();
optionBuilder.UseSqlServer(connection);
using var dbContext = new ApplicationDbContext(optionBuilder.Options);
var pendingMigrations = await dbContext.Database.GetPendingMigrationsAsync();
if(pendingMigrations.Any())
{
await dbContext.Database.MigrateAsync();
}
await dbContext.DisposeAsync();
}
Related
I've currently built a Visual Studio C# project that saves API data into a database through Entity Framework. Everything works fine, but I've noticed each time I run the application, the data is duplicated in the table.
I have to restart it every now and then since there is new data that needs storing, but I was thinking is there an easy way or even just a way that wipes the table before you save the data to the DB each time you run the application?
Here is the code I'm running at the moment.
public static void getAllRequestData()
{
var client = new RestClient("[My API URL]");
var request = new RestRequest();
var response = client.Execute(request);
if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
string rawResponse = response.Content;
AllRequests.Rootobject result = JsonConvert.DeserializeObject<AllRequests.Rootobject>(rawResponse);
using (var db = new TransitionContext())
{
db.RequestDetails.AddRange(result.Operation.Details);
db.SaveChanges();
} //Utilising EF to save data to the DB
}
} //Method that calls and stores API data
This method is used to get API data, deserialize it, then save it to a Db.
public class TransitionContext : DbContext
{
private const string connectionString = #"[My Server]";
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseSqlServer(connectionString);
}
public DbSet<AllRequests.Detail> RequestDetails { get; set; }
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<AllRequests.Detail>().HasKey(r => r.Id);
}
}
This is my Entity Framework DbContext that scaffolds the table and maps the data to each column.
This all works, but what would I have to include in my code for it to stop duplicating data each time I run the project? Currently, I get just over 1000 rows in my table each time I run the project, eventually, it gets clogged. I want it to be the same data each time I run it except with a few additions that are added each week.
Any help would be appreciated, thank you.
Solving this issue will depend on how the API works. If the data is returned in a sequence that doesn't change between calls, then you can remember the index of the last object saved in the database and next time save only fields that have a higher index. However if order isn't maintained between calls then you will be left having to manually compare objects to check for duplicates, which is not very time efficient but will be better than nothing. The thing is that second option won't work reliably if each item returned isn't unique. In that case the only viable method would be to clear the table entirely and reinsert everything. You can empty the table by calling db.RequestDetails.RemoveRange(db.RequestDetails). This might even be faster than the comparation method anyways.
I'm currently in the process of converting my 6 years old C# application to .NET Core v3 and EF Core (and also using Blazor).
Most of it is working except for the Sharding part.
Our application creates a new database for each client. We use more or less this code for it: https://learn.microsoft.com/en-us/azure/sql-database/sql-database-elastic-scale-use-entity-framework-applications-visual-studio
I'm now trying to convert it to EF Core, but get stuck at this part:
// C'tor to deploy schema and migrations to a new shard
protected internal TenantContext(string connectionString)
: base(SetInitializerForConnection(connectionString))
{
}
// Only static methods are allowed in calls into base class c'tors
private static string SetInitializerForConnection(string connnectionString)
{
// We want existence checks so that the schema can get deployed
Database.SetInitializer<TenantContext<T>>(new CreateDatabaseIfNotExists<TenantContext<T>>());
return connnectionString;
}
// C'tor for data dependent routing. This call will open a validated connection routed to the proper
// shard by the shard map manager. Note that the base class c'tor call will fail for an open connection
// if migrations need to be done and SQL credentials are used. This is the reason for the
// separation of c'tors into the DDR case (this c'tor) and the internal c'tor for new shards.
public TenantContext(ShardMap shardMap, T shardingKey, string connectionStr)
: base(CreateDDRConnection(shardMap, shardingKey, connectionStr), true /* contextOwnsConnection */)
{
}
// Only static methods are allowed in calls into base class c'tors
private static DbConnection CreateDDRConnection(ShardMap shardMap, T shardingKey, string connectionStr)
{
// No initialization
Database.SetInitializer<TenantContext<T>>(null);
// Ask shard map to broker a validated connection for the given key
var conn = shardMap.OpenConnectionForKey<T>(shardingKey, connectionStr, ConnectionOptions.Validate);
return conn;
}
The above code doesn't compile because the Database object doesn't exist in this way in EF Core.
I assume I can simplify it using TenantContext.Database.EnsureCreated(); somewhere. But I can't figure out how to modify the methods, which to remove, which to change (and how).
Of course, I've been searching for an example using sharding and EF Core but couldn't find it.
Does anybody here has done this before in EF Core and is willing the share?
I'm specifically looking for what to put in startup.cs and how to create a new sharding/database when I create a new client.
In EF.Core just resolve the shard in OnConfiguring. EG
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
var con = GetTenantConnection(this.tenantName);
optionsBuilder.UseSqlServer(con,o => o.UseRelationalNulls());
base.OnConfiguring(optionsBuilder);
}
Note that if you have a service or factory that returns open DbConnections, then you'll need to Close()/Dispose() them in the DbContext.Dispose(). If you get a connection string or a closed connection then DbContext will take care of closing the connection.
ASP.NET Core best-practices probably call for injecting an ITenantConfiguration service or somesuch in your DbContext. But the pattern is the same. Just save the injected service instance to a DbContext field and use it in OnConfiguring.
With the app that I'm working on, the desired shard is not discoverable until request time (for example, knowing what user is making the request, and then routing that user to their database). This meant that the OnConfiguring solution proposed above was not viable.
I worked around this by using IDbContextFactory<TContext>, and defining an extension on top of it, which sets the connection string based on whatever you want. I believe the database connection is created lazily in EF, and you are able to set the connection string up until the EF first needs to actually connect to the database.
In my case, it looked something like this:
var dbContext = _dbContextFactory.CreateDbContext();
var connectionString = $"DataSource={_sqlliteDirectory}/tenant_{tenant.TenantId}.db";
dbContext.Database.SetConnectionString(connectionString);
The downside is that it breaks the database abstraction (this code knows that my database is a local sqllite instance). An abstraction was not necessary in this layer of my app, but it is something very solvable if it's required.
For integration tests I am using an EntityFrameworkCore SQLite in-memory db and creating its schema as per Microsoft docs, but when I attempt to seed data an exception is thrown that tables do not exist.
The mouse-over docs for DbContext.Database.EnsureCreated(); :
Ensure that the database for the context exists. If it exists, no
action is taken. If it does not exist then the database and all its
schema are created. If the database exists, then no action is made to
ensure it is compatible with the model for this context.
I've read that an EntityFrameworkCore in-memory db only exists as long as an open connection exists, and so I tried explicitly creating a var connection = new SqliteConnection("DataSource=:memory:"); instance and wrapping the below code in a using(connection) {} block and passing the connection instance options.UseSqlite(connection);, but DbContext.Database.EnsureCreated(); still doesn't create any db-objects
public class CustomWebApplicationFactory<TStartup> : WebApplicationFactory<Startup>
{
protected override IWebHostBuilder CreateWebHostBuilder()
{
return WebHost.CreateDefaultBuilder()
.UseStartup<Startup>();
}
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
using (var connection = new SqliteConnection("DataSource=MySharedInMemoryDb;mode=memory;cache=shared"))
{
connection.Open();
builder.ConfigureServices(services =>
{
var serviceProvider = new ServiceCollection()
.AddEntityFrameworkSqlite()
.BuildServiceProvider();
services.AddDbContext<MyDbContext>(options =>
{
options.UseSqlite(connection);
options.UseInternalServiceProvider(serviceProvider);
});
var contextServiceProvider = services.BuildServiceProvider();
// we need a scope to obtain a reference to the database contexts
using (var scope = contextServiceProvider.CreateScope())
{
var scopedProvider = scope.ServiceProvider;
var logger = scopedProvider.GetRequiredService<ILogger<CustomWebApplicationFactory<TStartup>>>();
using (var myDb = scopedProvider.GetRequiredService<MyDbContext>())
{
// DEBUG CODE
// this returns script to create db objects as expected
// proving that MyDbContext is setup correctly
var script = myDb.Database.GenerateCreateScript();
// DEBUG CODE
// this does not create the db objects ( tables etc )
// this is not as expected and contrary to ms docs
var result = myDb.Database.EnsureCreated();
try
{
SeedData.PopulateTestData(myDb);
}
catch (Exception e)
{
// exception is thrown that tables don't exist
logger.LogError(e, $"SeedData.PopulateTestData(myDb) threw exception=[{e.Message}]");
}
}
}
});
}
builder.UseContentRoot(".");
base.ConfigureWebHost(builder);
}
Please note that in this post I'm only asking the question why doesn't DbContext.Database.EnsureCreated(); create the schema as expected. I'm not presenting the above code as a general pattern for running integration-tests.
Using a non-shared SQLite in-memory database
SQLite in-memory databases are by default transient. As the documentation states:
The database ceases to exist as soon as the database connection is closed. Every :memory: database is distinct from every other.
EF Core's DbContext on the other hand, always opens and closes connections to the database automatically, unless you pass an already open connection.
Therefore, in order to use the same SQLite in-memory database across multiple calls in EF Core, you need to create a SqliteConnection object separately and then pass it to every DbContext.
For example:
var keepAliveConnection = new SqliteConnection("DataSource=:memory:");
keepAliveConnection.Open();
services.AddDbContext<MyContext>(options =>
{
options.UseSqlite(keepAliveConnection);
});
Note that SqliteConnection isn't really thread-safe, so this approach is applicable only to single-threaded scenarios. Any time you want to have a shared database that can be accessed by multiple threads (e.g. in an ASP.NET Core application, servicing multiple requests), you should consider using an on-disk database.
By the way, this is the approach currently used in the EF Core documentation on how to use SQLite in-memory databases for testing.
Using a shared SQLite in-memory database
SQLite also supports named shared in-memory databases. By using the same connection string, multiple SqliteConnection objects can connect to the same database. However:
The database is automatically deleted and memory is reclaimed when the last connection to the database closes.
So it is still necessary to maintain a separate open connection object for the database to be usable across multiple EF Core calls. For example:
var connectionString = "DataSource=myshareddb;mode=memory;cache=shared";
var keepAliveConnection = new SqliteConnection(connectionString);
keepAliveConnection.Open();
services.AddDbContext<MyContext>(options =>
{
options.UseSqlite(connectionString);
});
Note that this approach isn’t limited to a single thread, because each DbContext gets its own instance of SqliteConnection.
I am in the process of creating a C# app to replace a VB6 app that uses a MySQL database where multiple copies of the app use the same database. The new app must be able to use the current MySQL database but I would also like it to be database agnostic so future instances of the app can use whatever server the user wants. In a perfect world, I would like the app on first run to present the user with a dialog that lets them choose the database type (MySQL, SQL Server, etc...) and specify the server ip, user, password, and database name. The app would then connect to that server and either use the database if it is already there or create a new database if it isn't.
Using Code First I have gotten to the point where I understand how to use the existing database or create a new one but only by hard coding the connection string in the App.config file.
<add name="GumpIndexDatabase"
connectionString="server=localhost;userid=123;password=123;port=3306;database=gump_new_data;pooling=false;"
providerName="MySql.Data.MySqlClient"
/>
I can change the connection string and provider before launching the app and everything works as expected. I can also change the connection string after launch, but not the provider, and I have to know whether the provider is MySQL or MSSQL in order to get the connection string details correct (ex: user or userid)
class GumpIndexDatabase: DbContext
{
public GumpIndexDatabase(string connectionName)
: base(MakeConnectionString(connectionName))
{
}
private static string MakeConnectionString(string connectionName)
{
if (connectionName=MySQL) {
//return MySQL string
} else {
//return SQL Server string
}
}
Hours of searching have not turned up an example of how to do such a thing, so I'm suspecting it isn't allowed or recommended, even though it seems like such a simple thing. I have seen some articles on connection string builder but did not understand how to get a database specific string from the generic objects.
So the simple question: how to specify the database connection details at run time?
I wouldn't recommend enforcing this feature, unless you 100% positive you cannot live without it. It adds a lot of maintenance tasks, that may not be obvious just now (such as updating to newer versions, bug fixes, spending lots of time to figure out common interfaces, maintaining those interfaces, hell lot of testing, etc). So unless it is a requested business feature - forget about it.
However, given you know what you are doing, this problem is generally solved via interfaces. There may be these common interfaces (it's a task by itslef to figure out them):
IConnection
IDataProvider
IRepository<T>
At the moment you will implement interfaces using MySql database, such as class MySqlConnection : IConnection. If you need MS SQL, class MsSqlConnection : IConnection.
Effectively you must abstract all the functionality into common interfaces. You will have to provide implementations for each database/storage engine you want to support. At runtime, you will use IoC container and DI principle to set up the current implementation. All the child dependencies will use interfaces passed in as parameters to constructor (or properties or methods)
Did you try the Database property?
GumpIndexDatabase.Database.Connection.ConnectionString = "your conn string";
I just tested it very short, so no guarantee it works without problems. But I was successful using it in the contructor of one of my service layer classes:
public class MyService
{
protected DataContext DataContext { get; set; }
public MyService(DataContext dataContext)
{
DataContext = dataContext;
DataContext.Database.Connection.ConnectionString = "conn string";
}
}
Just saw that DbContext has an overload DbContext(string nameOrConnectionString). You should be able to use this too.
Using an existing connection
Or you use an existing connection. Your DbContext should have something like this:
public class DataContext : DbContext
{
public DataContext(DbConnection existingConnection)
: base(existingConnection, true) { }
}
And then initialize it whereever you need to:
public void SomeMethod()
{
var connString = "whatever"; // could also be something like Textbox1.Text
using (var connection = new SqlConnection(connString))
{
var context = new DataContext(connection);
}
}
Of course SqlConnection can be anything that inherits from DbConnection. See DbConnection Class.
We have a database that has been deployed to various clients. We are currently introducing a optional new feature that, to be used, will require the customers who want the feature to have a new table added to the existing database.
As we are rolling out a new piece of software that will have to interact with versions of the database both with and without the new table (and as we don't want 2 versions one for customers who have the new table and one for ones who don't) we were wondering if it is possible to programmatically determine (with entity framework) whether a table exists in the database (I can try to access the table and have it throw a exception but was wondering if there was a built in function to do this)
Thanks
Edit: Given that people are telling me i should be using a config file not checking with EF can anyone give me guidence on how to check the config file with, for example, a custom data annotations for a mvc controller. Something like:
[Boolean(Properties.Settings.Default.TableExists)]
public class NamedController : Controller
Which throws a page not found if false?
Edit 2: With the Suggestions given by people to use the config settings i ended up with the following solution
App settings to set whether the table exists
<appSettings>
<add key="tableExists" value="True"/>
</appSettings>
a custom data annotation to say whether to allow access to controller
[AuthoriseIfTableExistsIsTrue]
public class NamedController : Controller
the code for the custom authorise
public class AuthoriseIfTableExistsIsTrue : AuthorizeAttribute
{
private readonly bool _tableExists;
public AuthoriseIfTableExistsIsTrue()
{
_tableExists = string.Equals(bool.TrueString, ConfigurationManager.AppSettings["tableExists"], StringComparison.InvariantCultureIgnoreCase);
}
public AuthoriseIfTableExistsIsTrue(bool authorise)
{
_tableExists = authorise;
}
protected override bool AuthorizeCore(HttpContextBase httpContext)
{
if (_tableExists)
return base.AuthorizeCore(httpContext);
else
throw new HttpException(404, "HTTP/1.1 404 Not Found");
}
}
Thanks everyone for the help and telling me not to use EF for this and use config setting instead
A much better option would be to store the version differences as configuration. This could be stored in the database itself, a configuration file or even web.config.
Otherwise you'll end up with messy code like:
int result = entity.ExecuteStoreQuery<int>(#"
IF EXISTS (SELECT * FROM sys.tables WHERE name = 'TableName')
SELECT 1
ELSE
SELECT 0
").SingleOrDefault();
The only possible ways are
Query table and get exception
Use native SQL to query system views and look for that table - in EFv4 you can execute query directly from ObjectContext by calling ExecuteStoreQuery.
Your entity model will still have this table so in my opinion you should simply ship your DB with that table and in application code handle if feature is allowed or not (table will not be used but will be in DB).
If you want to make modular system then whole your feature (including application code) should not be present when client don't want to use it.