I am having an issue getting logs that are written to the logger during an test runs to actually be written out to the console unless the corresponding WebApplicationFactory is created and disposed between tests runs...which isn't really that optimal :/
We have an a bunch of api integration tests, the tests conceptual all have the following pattern (not structure).
[TestFixture]
public class MyControllerTest
{
CustomWebApplicationFactory _factory;
HttpClient _client;
public OneTimeSetup()
{
_factory = new CustomWebApplicationFactory();
}
public SetUp()
{
_client = _factory.CreateClient();
}
public async Task TestA()
{
/* Some Test Using the _client */
}
public async Task TestA()
{
/* Some Test Using the _client */
}
public TearDown()
{
_client.Dispose();
}
public OneTimeTearDown()
{
_factory.Dispose();
}
}
Nothing inherently peculiar.
The CustomWebApplicationFactory above looks something like:
public class CustomWebApplicationFactory : WebApplicationFactory<Startup>
{
/* Bunch of stuff removed for brevity */
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
base.ConfigureWebHost(builder);
builder.ConfigureTestServices(s => { /* Some general mocked services */ });
builder
.ConfigureLogging(loggingBuilder =>
{
loggingBuilder
.ClearProviders()
.AddConsole()
.AddDebug()
.AddFilter("Microsoft.EntityFrameworkCore.Database.Command", LogLevel.Information)
.SetMinimumLevel(LogLevel.Information);
});
}
}
The issue I am having is particularly to do with the ef core DbContext query logging. These logs are not written to the console if the factory is not disposed between test runs. So in short I was able to get the behaviour I wanted by moving the moving the _factory initialization and disposing too the SetUp and TearDown functions respectively. But this seems to introduce a bit of fun issue on the testing servers...and also I don't really want to create and dispose the factory for every Test regardless.
Has anyone had a similar scenario? Is there away to flush the logs? instead of disposing the factory everyime?
Related
I have an Asp.net core 6 Web api.
I make Integration tests with xUnit following the pattern here. WebApplicationFactory creates a client which the tests can use to make calls to the application instance.
Now, xUnit runs the class constructor for every test. To share a member between all tests in a test class, you need to use a xUnit class fixture. This is the example from MSDN:
public class IndexPageTests :
IClassFixture<CustomWebApplicationFactory<RazorPagesProject.Startup>>
{
private readonly HttpClient _client;
private readonly CustomWebApplicationFactory<RazorPagesProject.Startup>
_factory;
public IndexPageTests(
CustomWebApplicationFactory<RazorPagesProject.Startup> factory)
{
_factory = factory;
_client = factory.CreateClient(new WebApplicationFactoryClientOptions
{
AllowAutoRedirect = false
});
}
I don't understand - why they put the CreateClient method in the constructor? Obviously, it will create a new client for every test. Why don't we create a separate Fixture class, create the client there and just inject it in the Test class?
For example:
public class TestFixture
{
public TestFixture()
{
var factory = new CustomWebApplicationFactory<RazorPagesProject.Startup>();
Client = factory.CreateClient(new WebApplicationFactoryClientOptions
{
AllowAutoRedirect = false
});
}
public HttpClient Client { get; }
}
public class TestClass : IClassFixture<TestFixture>
{
private readonly TestFixture _fixture;
public TestClass(TestFixture fixture)
{
_fixture = fixture;
}
[Fact]
public async Task FirstTest()
{
// use the fixture client directly
await _fixture.Client.GetAsync("..../url");
}
}
Does anyone see a problem with this approach? Memory leaks or other issues?
Is the fact that we instantiate CustomWebApplicationFactory<RazorPagesProject.Startup>() in the fixture constructor and not disposing it afterwards - causing memory problems, etc?
I have a .NET 6 web API project with existing integration tests for some of the API endpoints. The project uses Serilog for logging and everything was fine so far.
I migrated the code to the new minimal hosting model removing the Startup class in the process. I fixed the integration tests to work with the new model and everything is running so far. The only problem I have is, that the integration tests now spams log statements.
For Serilog I have the two staged setup, this is how Program.cs is looking like:
public partial class Program
{
public static string ApplicationVersion => typeof(Program).Assembly
.GetCustomAttribute<AssemblyInformationalVersionAttribute>()
.InformationalVersion;
/// <summary>
/// Hack to prevent duplicate logger initialization when integration tests run in parallel.
/// </summary>
public static bool IsIntegrationTestRun = false;
public static int Main(string[] args)
{
var env = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "Production";
if (!IsIntegrationTestRun)
{
// extra logger only for app startup
Log.Logger = new LoggerConfiguration()
.Enrich.FromLogContext()
.WriteTo.Console()
.CreateBootstrapLogger();
}
try
{
Log.Information("Starting <my application> v{version} in env {env}.", ApplicationVersion, env);
var builder = WebApplication.CreateBuilder(args);
builder.Configuration.AddJsonFile("appsettings.Local.json", true, true);
// Actual logger for dependency injection
builder.Host.UseSerilog((ctx, lc) =>
{
lc.ReadFrom.Configuration(ctx.Configuration);
});
// ...
var app = builder.Build();
// ...
using (IServiceScope scope = app.Services.CreateScope())
{
var dataContext = scope.ServiceProvider.GetRequiredService<DataContext>();
dataContext.Database.Migrate();
}
app.UseSerilogRequestLogging(c =>
{
c.EnrichDiagnosticContext = (diagnosticContext, httpContext) =>
{
diagnosticContext.Set("Host", httpContext.Request.Host.ToString());
diagnosticContext.Set("UserAgent", httpContext.Request.Headers["User-Agent"]);
};
c.GetLevel = LogLevelHelper.GetRequestLevel;
});
// ...
app.Run();
return 0;
}
catch (Exception ex)
{
Log.Fatal(ex, "Host terminated unexpectedly.");
return 1;
}
finally
{
Log.CloseAndFlush();
}
}
}
This is my WebApplicationFactory:
[CollectionDefinition("WebApplicationFactory")]
public class CustomWebApplicationFactory<TStartup> : WebApplicationFactory<TStartup>
where TStartup : class
{
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
// Somewhat hacky but prevents duplicate logger initialization when integration tests run in parallel.
Program.IsIntegrationTestRun = true;
builder.ConfigureAppConfiguration((context, builder) =>
{
// Load custom appsettings for Test
builder.AddJsonFile(Path.Combine(Directory.GetCurrentDirectory(), "appsettings.Test.json"));
// optional load personal settings included in gitignore
builder.AddJsonFile(Path.Combine(Directory.GetCurrentDirectory(), "appsettings.LocalTest.json"), true);
builder.AddEnvironmentVariables();
});
// builder.ConfigureLogging(lb => lb.ClearProviders());
Log.Logger = new LoggerConfiguration().MinimumLevel.Fatal().CreateLogger();
// ...
}
}
which is used like this:
[Collection("WebApplicationFactory")]
public class SomeTests : IClassFixture<CustomWebApplicationFactory<Program>>
{
private readonly CustomWebApplicationFactory<Program> _factory;
public SomeTests(CustomWebApplicationFactory<Program> factory)
{
_factory = factory;
}
[Fact]
public async Task Get_Some_ReturnsSomething()
{
// setup ...
HttpClient client = _factory.CreateClient();
client.DefaultRequestHeaders.Add("Authorization", RequestHelper.GetBearerAuthenticationHeaderValue(user));
RequestHelper.AddStrangeHeader(client, user.StrangeKey);
HttpResponseMessage response = await client.GetAsync("/api/some");
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
var res = await RequestHelper.DeserializeResponse<List<SomeModel>>(response);
Assert.Equal(SomeCount, res.Count);
}
}
As you can see I have extended the appsettings.json pattern to use a local gitignored file for local development (to keep secrets out of the repostiory) and an extra appsettings.Test.json (and another git ignored appsettings.LocalTest.json with extra settings for tests like a different db connection).
When I run the integration tests console is spammed with log statements. Strangely it seems not everything is logged, for example I can't see any request logs. But I can see logs for database migration multiple times like the following:
[09:57:38 INF Microsoft.EntityFrameworkCore.Migrations] Applying migration '20210224073743_InitialSchema'
or this one
[09:57:40 DBG lJty8ESu24x-MY6n4EYr Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler] Successfully validated the token..
I have tried many things like settings the minimum log level to Fatal or directly replace Log.Logger with a new logger.
The application itself is using the injected ILogger instead of the static Log.Logger. Can anyone guide me how to solve this or what I could try next?
The logging seems to respect the settings from my appsettings.Test.json file, when I reduce the minimum level to debug I can see more logs getting printed on the test run. But why is the migration message logged even when I set the minimum level to Fatal?
I think I've managed to do this. In your CustomWebApplicationFactory, put this in your ConfigureWebHost:
[CollectionDefinition("WebApplicationFactory")]
public class CustomWebApplicationFactory<TStartup> : WebApplicationFactory<TStartup>
where TStartup : class
{
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
#pragma warning disable CS0618
builder.UseSerilog((_, _) => { });
#pragma warning restore CS0618
// ... other customizations
base.ConfigureWebHost(builder);
}
}
It will complain about the method being obsolete, but this worked for me, it stops calling my original Serilog configuration, and simply will stop logging anything.
I believe you can also use this to change the configuration if you wish.
Update 2022-12-19: I just found a more clean way that works even without Serilog:
[CollectionDefinition("WebApplicationFactory")]
public class CustomWebApplicationFactory<TStartup> : WebApplicationFactory<TStartup>
where TStartup : class
{
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
builder.ConfigureTestServices(services =>
{
services.AddSingleton<ILoggerFactory, NullLoggerFactory>();
});
// ... other customizations
base.ConfigureWebHost(builder);
}
}
This silenced all the logs for me.
I have created some basic Integration tests to call my Api and see if the permissions work properly. Now I have encountered a problem where running more all of the tests one of them fails - if run seperately though, it doesnt.
The reason is, that I am using IMemoryCache to store certain permissions once a user is logged in. But for my integration tests, the permissions are stored in the cache and when I try to change them for a test they are not refreshed.
In general, is there a way to invalidate the MemoryCache for every Integration test?
One of my integrationtest class basically does this:
public IntegrationTest(CustomWebApplicationFactory<Namespace.Startup> factory)
{
_factory = factory;
_client = _factory.CreateClient();
// init the DB here etc...
var response = await _client.GetAsync("api/Some/Path");
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
is there a way to tell the factory not to use a cache or use a mock cache oder something like that?
Edit:
The cache is setup in my startup.cs like this:
public class Startup
{
public void ConfigureServices(IServiceCollection services)
{
services.AddMemoryCache();
[...]
}
}
And that is injected via DependenyInjection into my controllers, like this:
private IMemoryCache _cache;
private MemoryCacheEntryOptions _cacheOptions;
const int CACHE_LIFETIME_IN_DAYS = 7;
public SomeController(IMemoryCache cache) {
_cache = cache;
_cacheOptions = new MemoryCacheEntryOptions().SetSlidingExpiration(TimeSpan.FromDays(CACHE_LIFETIME_IN_DAYS));
}
and I'm using it in my controllers with _cache.TryGetValue and _cache.Set
As a quick fix you can try to do something like this:
var memoryCache = _factory.Services.GetService<IMemoryCache>() as MemoryCache;
memoryCache.Compact(1.0);
When you need to reset cache.
But I would recommend either to look into not sharing _factory between tests (though it can have some performance implications) or overwriting (like it is done in the docs with context) IMemoryCache to something that you can control outside as you need.
UPD
Since tests by default are not run in parrallel you can just manually register instance of MemoryCache. Something like this:
public class CustomWebApplicationFactory<TStartup>
: WebApplicationFactory<TStartup> where TStartup : class
{
internal readonly MemoryCache MemoryCache;
public CustomWebApplicationFactory()
{
MemoryCache = new MemoryCache(new MemoryCacheOptions());
}
public void ClearCache() => MemoryCache.Compact(1.0);
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
builder.ConfigureServices(services =>
{
var descriptor = services.SingleOrDefault(
d => d.ServiceType ==
typeof(IMemoryCache));
services.Remove(descriptor);
services.AddSingleton<IMemoryCache>(MemoryCache);
});
}
}
And in test call factory.ClearCache():
public void Test1()
{
var factory = new CustomWebApplicationFactory<Startup>();
var memoryCache = factory.Services.GetService<IMemoryCache>() as MemoryCache;
memoryCache.Set("test", "test");
factory.ClearCache();
Assert.IsFalse(memoryCache.TryGetValue("test", out var val));
}
If you will need to run tests with the same factory in parallel (though I would say better just create different factories) then you can create IMemoryCache implementation which will determine somehow (for example passing some specific header in client request) different test runs and return different instances of MemoryCache for them.
We have our integration tests set up using xUnit and Microsoft.AspNetCore.TestHost.TestServer to run tests against Web API running on ASP.NET Core 2.2.
Our Web API is a single code base that would be deployed separately multiple times based on some configuration or application setting differences like country, currency, etc.
Below diagram tries to explain our deployment set up:
We want to ensure that our integration tests run against all the deployments.
For both deployments, X and X` the API endpoint, request, and response are absolutely same. Hence, We would like to avoid repeating ourselves when it comes to integration tests for each deployment.
Here is the sample code explaining our current test set up:
TestStartup.cs
public class TestStartup : IStartup
{
public IServiceProvider ConfigureServices(IServiceCollection services)
{
var configuration = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", false)
.AddEnvironmentVariables()
.Build();
services.AddMvc()
.SetCompatibilityVersion(version: CompatibilityVersion.Version_2_2);
// Code to add required services based on configuration
return services.BuildServiceProvider();
}
public void Configure(IApplicationBuilder app)
{
app.UseMvc();
// Code to configure test Startup
}
}
TestServerFixture.cs
public class TestServerFixture
{
public TestServerFixture()
{
var builder = new WebHostBuilder().ConfigureServices(services =>
{
services.AddSingleton<IStartup>(new TestStartup());
});
var server = new TestServer(builder);
Client = server.CreateClient();
}
public HttpClient Client { get; private set; }
}
MyTest.cs
public class MyTest : IClassFixture<TestServerFixture>
{
private readonly TestServerFixture _fixture;
public MyTest(TestServerFixture fixture)
{
_fixture = fixture;
}
[Fact]
public void ItShouldExecuteTwice_AgainstTwoSeparateConfigurations()
{
//...
}
}
Now, I'm looking to run ItShouldExecuteTwice_AgainstTwoSeparateConfigurations test in class MyTest more than once against two different configurations/ app settings or in other words against two different test deployments within Visual Studio.
I know, I should be able to achieve this using a combination of build configurations (like DEBUG_SETTING1, DEBUG_SETTING2) and preprocessor directive (#if DEBUG_SETTING1).
The other option could be to have a base test helper project with common methods and a separate integration project for each deployment.
Is there a better and more elegant way to achieve this?
Refactor the test startup to allow for it to be modified as needed for its test
For example
public class TestStartup : IStartup {
private readonly string settings;
public TestStartup(string settings) {
this.settings = settings;
}
public void ConfigureServices(IServiceCollection services) {
var configuration = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile(settings, false) //<--just an example
.AddEnvironmentVariables()
.Build();
services.AddMvc()
.SetCompatibilityVersion(version: CompatibilityVersion.Version_2_2);
//...Code to add required services based on configuration
}
public void Configure(IApplicationBuilder app) {
app.UseMvc();
//...Code to configure test Startup
}
}
And have that pattern filter up through the fixture
public class TestServerFixture {
static readonly Dictionary<string, TestServer> cache =
new Dictionary<string, TestServer>();
public TestServerFixture() {
//...
}
public HttpClient GetClient(string settings) {
TestServer server = null;
if(!cache.TryGetValue(settings, out server)) {
var startup = new TestStartup(settings); //<---
var builder = new WebHostBuilder()
.ConfigureServices(services => {
services.AddSingleton<IStartup>(startup);
});
server = new TestServer(builder);
cache.Add(settings, server);
}
return server.CreateClient();
}
}
And eventually the test itself
public class MyTest : IClassFixture<TestServerFixture> {
private readonly TestServerFixture fixture;
public MyTest(TestServerFixture fixture) {
this.fixture = fixture;
}
[Theory]
[InlineData("settings1.json")]
[InlineData("settings2.json")]
public async Task Should_Execute_Using_Configurations(string settings) {
var client = fixture.CreateClient(settings);
//...use client
}
}
#Nkosi's post fits very well with our scenario and my asked question. It's a simple, clean and easy to understand approach with maximum reusability. Full marks to the answer.
However, there were a few reasons why I could not go forward with the approach:
In the suggested approach we couldn't run tests for only one particular setting. The reason it was important for us as in the future, there could two different teams maintaining their specific implementation and deployment. With Theory, it becomes slightly difficult to run only one setting for all the tests.
There is a high probability that we may need two separate build and deployment pipelines for each setting/ deployment.
While the API endpoints, Request, and Response are absolutely the same today, we do not know if it will continue to be the case as our development proceed.
Due to the above reasons we also considered the following two approaches:
Approach 1
Have a common class library which has common Fixture and Tests as abstract class
Project Common.IntegrationTests
TestStartup.cs
public abstract class TestStartup : IStartup
{
public abstract IServiceProvider ConfigureServices(IServiceCollection services);
public void Configure(IApplicationBuilder app)
{
app.UseMvc();
// Code to configure test Startup
}
}
TestServerFixture.cs
public abstract class TestServerFixture
{
protected TestServerFixture(IStartup startup)
{
var builder = new WebHostBuilder().ConfigureServices(services =>
{
services.AddSingleton<IStartup>(startup);
});
var server = new TestServer(builder);
Client = server.CreateClient();
}
public HttpClient Client { get; private set; }
}
MyTest.cs
public abstract class MyTest
{
private readonly TestServerFixture _fixture;
protected MyTest(TestServerFixture fixture)
{
_fixture = fixture;
}
[Fact]
public void ItShouldExecuteTwice_AgainstTwoSeparateConfigurations()
{
//...
}
}
Project Setting1.IntegrationTests (References Common.IntegrationTests)
TestStartup.cs
public class TestStartup : Common.IntegrationTests.TestStartup
{
public override IServiceProvider ConfigureServices(IServiceCollection services)
{
var configuration = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", false) // appsettings for Setting1
.AddEnvironmentVariables()
.Build();
services.AddMvc()
.SetCompatibilityVersion(version: CompatibilityVersion.Version_2_2);
// Code to add required services based on configuration
return services.BuildServiceProvider();
}
}
TestServerFixture.cs
public class TestServerFixture : Fixtures.TestServerFixture
{
public TestServerFixture() : base(new TestStartup())
{
}
}
MyTests.cs
public class MyTests : Common.IntegrationTests.MyTests, IClassFixture<TestServerFixture>
{
public MyTests(TestServerFixture fixture) : base(fixture)
{
}
}
Project Setting2.IntegrationTests (References Common.IntegrationTests)
A similar structure as Setting1.IntegrationTests
This approach provided a good balance of reusability and flexibility to run/ modify the tests independently. However, I was still not 100% convinced with this approach as it meant for each common Test class we would need to have an implementation where we are not doing anything other than calling the base constructor.
Approach 2
In the second approach, we took the Approach 1 further and try to fix the issue we had with Approach 1 with Shared Project. From the documentation:
Shared Projects let you write common code that is referenced by a
number of different application projects. The code is compiled as part
of each referencing project and can include compiler directives to
help incorporate platform-specific functionality into the shared code
base.
Shared Project gave us the best of both worlds without the ugliness of link files and unnecessary class inheritance or abstraction. Our new set up is as follows:
Edit:
I wrote a blog post on this where I have talked about our use-case and the solution in detail. Here is the link:
https://ankitvijay.net/2020/01/04/running-an-asp-net-core-application-against-multiple-db-providers-part-2/
I write integration tests for my application, and use my container for this. I want to be able to register all the components as I do in real running, and then override some of the components and switch them to use stubs implementations.
I wouldn't want to seperate the DI and have a container for tests only because I want to test the real thing.
Doing this also seems ugly:
public class MyRegistrations
{
public static RegisterAll(bool isInTest= false)
{
if (isTest)
{
// Register test fakes
}
else
// Register real components
}
}
So I thought of overriding registrations in my test enviorment. How should it be done?
Any other better ways for achieving my goal?
Thanks
Autofac will use the last registered component as the default provider
of that service
From the AutoFac documation.
In your arrange/setup/testInit phase register the mocks, then resolve the SUT:
[SetUp]
public void TestInit()
{
Mock<IFoo> mock = new Mock<IFoo>();
builder.RegisterInstance(mock.object).As<IFoo>();
...
...
_target = builder.Resolve<The component>();
}
Note:
Singletons, static members and SingletonLifestyle(registration) may cause some troubles....
Well, for example you can create a static action method inside your composition root to alter the current configuration and call it during testing. For example:
public class CompositionRoot
{
public static Action<IContainer> OverrideContainer = c => { };
internal static IContainer CreateContainer()
{
ContainerBuilder builder = new ContainerBuilder();
/// etc. etc.
var container = builder.Build();
OverrideContainer(container);
return container;
}
}
After that you can create a mock of you server, for example, like this:
[TestFixture]
public class ConfigurationControllerFixture : BaseServer
{
[Test]
public async Task verify_should_get_data()
{
var response = await GetAsync(Uri);
Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
}
protected override string Uri
{
get { return "api/configuration"; }
}
}
public abstract class BaseServer
{
protected TestServer Server;
protected abstract string Uri { get; }
protected virtual void OverrideConfiguration()
{
CompositionRoot.OverrideContainer = c =>
{
// new autofac configuration
cb.Update(c);
};
AppStartup.OverrideConfiguration = c =>
{
// same as explained, but for HttpConfiguration
};
}
}
[SetUp]
public void Setup()
{
OverrideConfiguration();
Server = Microsoft.Owin.Testing.TestServer.Create(app =>
{
var startup = new AppStartup();
startup.Configuration(app);
});
PostSetup(Server);
}
Hope it helps :)
If you want to write integration test from API to database you can use XUnit. XUnit use TestHost and WebApplicationFactory to create a System under test. With XUnit, it's very easy to mock a test service by add test service to service collection.
I made a open source project use XUnit to test my API work with mySQL database. Please visit here for example https://gitlab.com/quorion-group/quorion-backend-crm