We're using DBUP to handle db migrations. Each release, we would like to run the dbup console app with a command line switch so that during dev we can re-run our scripts while we're working on them, however we don't want it to re-run all the previous releases scripts which already appear in the database. How can this be achieved?
We added a '-debug' command line switch to our DbUp console application. If this is present we switch which Journal class is used when talking to the database.
The Journal class (https://dbup.readthedocs.io/en/latest/more-info/journaling/) in DbUp is the class that interacts with the database to check and record which scripts have already been run (stored by default in the Schema Versions table). For Dev, we force this to use a read-only version of this, which can check which scripts are already present (to prevent you re-running everything each time) but prevents new records being recorded, so that next time it will attempt to re-run your new scripts again.
The read only journal looks like this;
public class ReadOnlyJournal : IJournal
{
private readonly IJournal _innerJournal;
public ReadOnlyJournal(IJournal innerJournal)
{
_innerJournal = innerJournal;
}
public void EnsureTableExistsAndIsLatestVersion(Func<IDbCommand> dbCommandFactory)
{
_innerJournal.EnsureTableExistsAndIsLatestVersion(dbCommandFactory);
}
public string[] GetExecutedScripts()
{
return _innerJournal.GetExecutedScripts().ToArray();
}
public void StoreExecutedScript(SqlScript script, Func<IDbCommand> dbCommandFactory)
{
// don't store anything
}
}
Then an extension method to allow the use of this new journal to be easier specified;
public static class DbUpHelper
{
public static UpgradeEngineBuilder WithReadOnlyJournal(this UpgradeEngineBuilder builder, string schema, string table)
{
builder.Configure(c => c.Journal = new ReadOnlyJournal(new SqlTableJournal(() => c.ConnectionManager, () => c.Log, schema, table)));
return builder;
}
}
And then finally the change to your DbUp console app;
var upgrader = debug
? DeployChanges.To
.SqlDatabase(connectionString)
.WithScriptsEmbeddedInAssembly(Assembly.GetExecutingAssembly())
.WithReadOnlyJournal("dbo", "SchemaVersions")
.LogToConsole()
.Build()
: DeployChanges.To
.SqlDatabase(connectionString)
.WithScriptsEmbeddedInAssembly(Assembly.GetExecutingAssembly())
.LogToConsole()
.Build();
var result = upgrader.PerformUpgrade();
if (!result.Successful)
....
Related
I have written a program that is running and I cannot delete the database. Now I want to add a series of new properties to the database tables. I know that I should use migration, but the point is that I have written the connection string manually with ConfigurationManager class and its functions, not in app.config. But now I have encountered a problem to add a new migration and update the database.
My question here is how to do the new migration and database update as code functions in C# instead of using package manager console and its commands i.e add-migration & update-database.
After many searches, I found a series of solutions like this link https://stackoverflow.com/questions/20216147/entity-framework-change-connection-at-runtime?answertab=modifieddesc#tab-top and using IDBConnectionIntercaptor, but in The test environment creates this database face with the error Cannot attach the file '***' as database '+++' .
my connection function is :
`
public static bool Connect(bool force = false)
{
bool result = false;
var selectedDb = new EventsDataContext();
selectedDb.ChangeDatabase
(
initialCatalog: Maincsb.InitialCatalog,
userId: "",
password: "",
dataSource: Maincsb.DataSource
);
EventsDataContext odb = null;
try
{
ConnectionInterceptor connectionInterceptor = new ConnectionInterceptor
{
connectionString = ConnectionManager.Maincsb.ConnectionString
};
DbInterception.Add(connectionInterceptor);
DbInterception.Add(new ModifyYeKeIntercpetor());
odb = new EventsDataContext();
odb.Database.Initialize(force);
Seeder(Maincsb.ConnectionString);
result = true;
}
catch (Exception e)
{
ErrorMessage.Show(e);
result = false;
}
finally
{
odb.Dispose();
odb = null;
}
return result;
}
`
And if the database has already been created, it will face the error that the database has changed and new features will not be added to the database.
I am writing some integration tests for my web API, which means that it has to be running during the execution of the tests. Is there any way to run it with an in-memory database instead of a real one based on SQL Server?
Also, I need to run a few instances at a time, so I need somehow to change the base address of each of them to be unique. For example, I could append to the base URL these instance IDs, that are mentioned in the code below.
Here is the code which I am using to run a new instance for my tests:
public static class WebApiHelper
{
private const string ExecutableFileExtension = "exe";
private static readonly Dictionary<Guid, Process> _instances = new();
public static void EnsureIsRunning(Assembly? assembly, Guid instanceId)
{
if (assembly is null)
throw new ArgumentNullException(nameof(assembly));
var executableFullName = Path.ChangeExtension(
assembly.Location, ExecutableFileExtension);
_instances.Add(instanceId, Process.Start(executableFullName));
}
public static void EnsureIsNotRunning(Guid instaceId)
=> _instances[instaceId].Kill();
}
Talking in general, is this a good way to create test instances, or maybe I am missing something? Asking this, because maybe there is another 'legal' way to achieve my goal.
Okay, so in the end, I came up with this super easy and obvious solution.
As was mentioned in the comments - using the in-memory database is not the best way to test, because relational features are not supported if using MS SQL.
So I decided to go another way.
Step 1: Overwrite the connection strings.
In my case, that was easy since I have a static IConfiguration instance and was need just to overwrite the connection string within that instance.
The method looks as follows:
private const string ConnectionStringsSectionName = "ConnectionStrings";
private const string TestConnectionStringFormat = "{0}_Test";
private static bool _connectionStringsOverwitten;
private static void OverwriteConnectionStrings()
{
if (_connectionStringsOverwitten)
return;
var connectionStrings = MyStaticConfigurationContainer.Configuration
.AsEnumerable()
.Where(entry => entry.Key.StartsWith(ConnectionStringsSectionName)
&& entry.Value is not null);
foreach (var connectionString in connectionStrings)
{
var builder = new SqlConnectionStringBuilder(connectionString.Value);
builder.InitialCatalog = string.Format(TestConnectionStringFormat,
builder.InitialCatalog);
MyStaticConfigurationContainer.Configuration[connectionString.Key] = builder.ConnectionString;
}
_connectionStringsOverwitten = true;
}
Of course, you would need to handle the database creation and deletion before and after running the tests, otherwise - your test DBs may become a mess.
Step 2: Simply run your web API instance within a separate thread.
In my case, I am using the NUnit test framework, which means I just need to implement the web API setup logic within the fixture. Basically, the process would be more or less the same for every testing framework.
The code looks as follows:
[SetUpFixture]
public class WebApiSetupFixture
{
private const string WebApiThreadName = "WebApi";
[OneTimeSetUp]
public void SetUp() => new Thread(RunWebApi)
{
Name = WebApiThreadName
}.Start();
private static void RunWebApi()
=> Program.Main(Array.Empty<string>());
// 'Program' - your main web app class with entry point.
}
Note: The code inside Program.Main(); will also look for connection strings in the MyStaticConfigurationContainer.Configuration which was changed in the previous step.
And that's it! Hope this could help somebody else :)
I am running into a problem with SQLite where I imported a database from the assets folder into forms, now I'm trying to load the database using the method given by Microsoft, but now when I am loading it shows the tablemappings don't exist.
I know this because if I set a breakpoint on the bit where it checks the tablemappings, all I'm given is the table mappings to the database file that is created in the app.
I once managed to get it to load one of the databases and it told me that it cannot write to a readonly database, so it was trying to create a new table (I honestly do not know how it happened or how to recreate it, because changed the code back and it doesnt reproduce)
The error actually thrown is a "NullRefrenceException: loading...", but I'm 90% sure I am getting this one since the initializer is an async task. Anyway I have polly to try and rerun the initializer after it should have fully initialized (A solution that works fantastically with my main database, that is created on device)
When I continue past it everything just freezes.
I used the profiler and after a bit it did save the model, it also show loads of nullrefrenceexceptions but basically it just ignores not loading the database.
Honestly I'm sitting here for hours trying different things, and can't seem to find a solution, I tried looking for a few hours as well online...
DataBase
private async Task Initialize()
{
_db = await GetDatabaseConnection<SabbathModel>();
}
static readonly Lazy<SQLiteAsyncConnection> _lazyInitializer = new Lazy<SQLiteAsyncConnection>(() =>
{
return new SQLiteAsyncConnection(Constants.InternalDatabasePath, Constants.InternalDBFlags);
});
static SQLiteAsyncConnection DB => _lazyInitializer.Value;
public static bool initialized = false;
protected static async Task<SQLiteAsyncConnection> GetDatabaseConnection<T>()
{
if (!initialized)
{
if (!DB.TableMappings.Any(x => x.MappedType == typeof(T)))
{
await DB.CreateTablesAsync(CreateFlags.None, typeof(T)).ConfigureAwait(false);
}
initialized = true;
}
return DB;
}
Constants Mentioned:
public static string InternalDatabasePath
{
get
{
var basePath = Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData);
return Path.Combine(basePath, InternalDatabaseFilename);
}
}
public const SQLite.SQLiteOpenFlags InternalDBFlags =
// open the database in read/write mode
SQLite.SQLiteOpenFlags.ReadOnly |
// enable multi-threaded database access
SQLite.SQLiteOpenFlags.PrivateCache;
public const string InternalDatabaseFilename = "InternalDatabase.db3";
and the task that loads the database on the phone
private async Task CopyInternalDataBase()
{
string dbPath = Constants.InternalDatabasePath;
if (!File.Exists(dbPath))
{
using BinaryReader br = new BinaryReader(await FileSystem.OpenAppPackageFileAsync("InternalDataBase.db"));
using BinaryWriter bw = new BinaryWriter(new FileStream(dbPath, FileMode.Create));
byte[] buffer = new byte[2048];
int len = 0;
while ((len = br.Read(buffer, 0, buffer.Length)) > 0)
{
bw.Write(buffer, 0, len);
}
}
}
I've been using a DbContext (MySql.Data.EntityFrameworkCore 6.10.6) on a WebAPI project (.NET Core 2.0) for quite a while without any problems but now I need to use the same context on a Console Application and special characters aren't being loaded correctly.
WebAPI:
public class ProductsController : Controller
{
private readonly MyDbContext _context;
public ProductsController(MyDbContext context)
{
_context = context;
}
[HttpGet]
public string Get()
{
var productName = _context.Products
.Single(x => x.Id == 1)
.Name;
return productName; = // "PÃO FRANCÊS"
}
}
Console App:
private static void Main(string[] args)
{
// setup DI
Initialize();
var context = ServiceProvider.GetService<MyDbContext>();
var productName = context.Products
.Single(x => x.Id == 1)
.Name;
Console.WriteLine(productName); // "P�O FRANC�S"
}
Why are these values different even though the same context is used?
Update 1:
Changing the Console.OutputEncoding to UTF8 or changing the console font to Lucidas Console or Console didn't change the result. The real purpose of the Console App is to call an API and the result has the � character as soon as I retrieve the value from the DB when debugging. I don't need to print the result on the console.
Update 2:
When debugging past the productName variable, the value doesn't change when calling productName or ?productName in the Immediate Window from Visual Studio.
Update 3:
I've configured a test database using MS SQL Server and the characters are loaded correctly. It looks like it's a problem with MySql.Data.EntityFrameworkCore
Update 4:
Tested it with a new Console App using .NET Framework 4.7.1 and the characters are displayed correctly.
The solution was to add this line to the console app.
Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
Reference: https://github.com/dotnet/corefx/issues/10054
Try adding this to get the special characters to load correctly.
Console.OutputEncoding = System.Text.Encoding.Unicode;
I'm using fluent migrator to manage my database migrations, but what I'd like to do is have the migrations run at app start. The closest I have managed is this:
public static void MigrateToLatest(string connectionString)
{
using (var announcer = new TextWriterAnnouncer(Console.Out)
{
ShowElapsedTime = true,
ShowSql = true
})
{
var assembly = typeof(Runner).Assembly.GetName().Name;
var migrationContext = new RunnerContext(announcer)
{
Connection = connectionString,
Database = "SqlServer2008",
Target = assembly
};
var executor = new TaskExecutor(migrationContext);
executor.Execute();
}
}
I'm sure I had this working, but I've not looked at it for sometime (hobby project) and it's now throwing null reference exceptions when it gets to the Execute line. Sadly there are no docs for this and I've been banging my head on it for ages.
Has anyone managed to get this kind of thing working with FluentMigrator?
PM> Install-Package FluentMigrator.Tools
Manually add a reference to:
packages\FluentMigrator.Tools.1.6.1\tools\AnyCPU\40\FluentMigrator.Runner.dll
Note that the folder name will vary on version number, this illustration uses the current 1.6.1 release. If you need the .NET 3.5 runner use the \35\ directory.
public static class Runner
{
public class MigrationOptions : IMigrationProcessorOptions
{
public bool PreviewOnly { get; set; }
public string ProviderSwitches { get; set; }
public int Timeout { get; set; }
}
public static void MigrateToLatest(string connectionString)
{
// var announcer = new NullAnnouncer();
var announcer = new TextWriterAnnouncer(s => System.Diagnostics.Debug.WriteLine(s));
var assembly = Assembly.GetExecutingAssembly();
var migrationContext = new RunnerContext(announcer)
{
Namespace = "MyApp.Sql.Migrations"
};
var options = new MigrationOptions { PreviewOnly=false, Timeout=60 };
var factory =
new FluentMigrator.Runner.Processors.SqlServer.SqlServer2008ProcessorFactory();
using (var processor = factory.Create(connectionString, announcer, options))
{
var runner = new MigrationRunner(assembly, migrationContext, processor);
runner.MigrateUp(true);
}
}
}
Note the SqlServer2008ProcessorFactory this is configurable dependent upon your database, there is support for: 2000, 2005, 2008, 2012, and 2014.
I have actually accomplished running migrations in the application_start however it is hard to tell from that code what could be wrong... Since it is open source I would just grab the code and pull it into your solution and build it to find out what the Execute method is complaining about. I found that the source code for Fluent Migrator is organized pretty well.
One thing that you might have to be concerned about if this is a web app is making sure that no one uses the database while you are migrating. I used a strategy of establishing a connection, setting the database to single user mode, running the migrations, setting the database to multi user mode, then closing the connection. This also handles the scenario of a load balanced web application on multiple servers so 2 servers don't try to run migrations against the same database.