Develop Reference

Do not collect child opentelemetry trace when parent trace is not recorded

I'm in the middle of introducing distributed tracing in a microservice app. All my services have the tracing enabled and everything works fine. But...
My app runs on a K8S cluster, so that cluster makes a lot of call to health endpoint. A lot is a lot. for the moment when nothing happens on a setup, I get 50Gb of trace recorder per day.
trace lock like this: one trace for the /health call and one children trace for a database call (call to check if db is available)
So, i decided to not record Health trace this is easily done by
// in startup
tracerProviderBuilder.AddAspNetCoreInstrumentation(x => x.Filter = AspnetCoreOtelFilter.Filter)
//the filter
public static class AspnetCoreOtelFilter
{
public static bool Filter(HttpContext httpContext)
{
if (httpContext.Request.Path == "/health/liveness")
return false;
if (httpContext.Request.Path == "/health/readiness")
return false;
return true;
}
}
as a result, I don't see all the health trace in my APM, anymore, but I keep receiving the trace regarding the log in DB
I was expecting that ParentBasedSampler (https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry/Trace/ParentBasedSampler.cs) would do the magic, but it's not the case.
Do you have any idea.
Thx for your help
Here is my test class:
[TestMethod, Ignore("Didn't succeed to avoid child message for now")]
public async Task HealthWithSubTrace_NoTracing()
{
var exportedItems = new List<Activity>();
var factory = new WebApplicationFactory<Program>();
using (var client = factory.WithWebHostBuilder(builder =>
builder.ConfigureTestServices(serviceCollection => ConfigureInMemoryExporterForOpenTelemetry()).CreateClient())
{
using var response = await client.GetAsync((string?)"/health/details").ConfigureAwait(false);
response.EnsureSuccessStatusCode();
WaitForActivityExport(exportedItems, 1);
}
Assert.AreEqual(0, exportedItems.Count);
}
//OTEL configuration is this
tracerProviderBuilder
.SetResourceBuilder(ResourceBuilder.CreateDefault()
.AddService(serviceName))
.SetSampler(sp => new ParentBasedSampler(new AlwaysOnSampler())
.AddAspNetCoreInstrumentation(x => x.Filter = AspnetCoreOtelFilter.Filter)
.AddHttpClientInstrumentation()
.AddProcessor<K8STagsProcessor>()
.AddSource(OpenTelemetrySources.Npgsql)
.AddSource(Program.source.Name);
//with program defined as this
public class Program
{
public static readonly ActivitySource source = new("OpenTelemtry.Test.App");
public static void Main(string[] args)
{
var builder = WebApplication.CreateBuilder(args);
builder.Services.ConfigurePhoenixHealth()
.AddHealthChecks().AddCheck(HealthConstants.Self, () => HealthCheckResult.Healthy());
var app = builder.Build();
app.UsePhoenixHealth();
app.MapGet("/health/details", () =>
{
using var activity = source.StartActivity("health.detail");
return "all fine";
});
app.Run();
}
}

Unknown command: --environment=Development during testing

I created an ASP.NET project and wrote some integration tests for it. But when I tried to run dotnet test this shows up:
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
The active test run was aborted. Reason: Test host process crashed : Unknown command: --environment=Development
Test Run Aborted with error System.Exception: One or more errors occurred.
---> System.Exception: Unable to read beyond the end of the stream.
at System.IO.BinaryReader.Read7BitEncodedInt()
at System.IO.BinaryReader.ReadString()
at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.LengthPrefixCommunicationChannel.NotifyDataAvailable()
at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TcpClientExtensions.MessageLoopAsync(TcpClient client, ICommunicationChannel channel, Action`1 errorHandler, CancellationToken cancellationToken)
--- End of inner exception stack trace ---.
As I understand something tries to run dotnet executable with --environment=Development but this argument is invalid even though it is used in Microsoft docs.
I tried creating new ASP.NET project (no controllers, services, database etc. just API that does nothing and an empty test) but I couldn't reproduce the error again.
Initially I created my project and solution in the same folder by accident and had to manually move project to subfolder. Everything worked fine after I did that so I assumed it's fine. Maybe that is the reason.
Here's how I access application during testing:
// TestingApplication.cs
public class TestingApplication : WebApplicationFactory<Program>
{
private readonly Guid _appId = Guid.NewGuid();
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
// Add mock/test services to the builder here
builder.ConfigureServices(services =>
{
services.AddMvcCore().AddApplicationPart(typeof(Program).Assembly);
services.AddScoped(sp => new DbContextOptionsBuilder<EfDbContext>()
.UseSqlServer(
$"DATABASE CONNECTION STRING")
.UseApplicationServiceProvider(sp)
.Options);
});
}
protected override IHost CreateHost(IHostBuilder builder)
{
var host = base.CreateHost(builder);
using (var serviceScope = host.Services.GetRequiredService<IServiceScopeFactory>().CreateScope())
{
var context = serviceScope.ServiceProvider.GetRequiredService<EfDbContext>();
context.Database.EnsureCreated();
}
return host;
}
protected override void Dispose(bool disposing)
{
base.Dispose(disposing);
using (var serviceScope = Server.Services.GetRequiredService<IServiceScopeFactory>().CreateScope())
{
var context = serviceScope.ServiceProvider.GetRequiredService<EfDbContext>();
context.Database.EnsureDeleted();
}
}
}
// BaseTest.cs
public class BaseTest : IDisposable, IClassFixture<TestingApplication>
{
protected readonly TestingApplication Application;
private HttpClient? _client;
protected HttpClient Client => _client ??= Application.CreateClient();
public BaseTest(TestingApplication testingApplication)
{
Application = testingApplication;
}
public void Dispose()
{
Application.Dispose();
}
}
Some more info:
Unit tests work just fine
Initially I forgot to add <InternalsVisibleTo Include="NameOfTestsProject" /> to the main project file, but it doesn't work either way.
.NET 6, OS - Linux, IDE - Jetbrains Rider
Rebuilding solution does not work
Creating new project for unit tests doesn't help either
Does anyone know what the problem is?
UPD I figured it out

Okay, So this is just another example of copy-pasting someone else's code without checking. I had copied something along the lines of:
if (args[0] == "something") {
...
} else if (args[0] == "something else") {
...
} else {
// exit with code 1 here and print error
}
in my Program.cs. It worked fine by itself but when testing it caused this problem.

Can Entity Framework Core bet setup to use a backup connection string?

My company is using 2 Windows servers. 1 Server is running as a backup server and other than SQL replication, the backup server requires manual intervention to get it running as the primary. I have no control over this, but I do have control of the apps/services running on the servers.
What I have done is I got all the services to be running on both and added Rabbit MQ as a clustered message broker to kind of distribute the work between the servers. This is all working great and when I take a server down, nothing is affected.
Anyway, to the point of the question, the only issue I see is that the services are using the same SQL server and I have nothing in place to automatically switch server if the primary goes down.
So my question is, is there a way to get Entity Framework to use an alternative connection string should one fail?
I am using the module approach with autofac as dependency injection for my services. This is the database registration.
public class AppsDbModule : Module
{
protected override void Load(ContainerBuilder builder)
{
RegisterContext<AppsDbContext>(builder);
}
private void RegisterContext<TContext>(ContainerBuilder builder) where TContext : DbContext
{
builder.Register(componentContext =>
{
var serviceProvider = componentContext.Resolve<IServiceProvider>();
var configuration = componentContext.Resolve<IConfiguration>();
var dbContextOptions = new DbContextOptions<TContext>(new Dictionary<Type, IDbContextOptionsExtension>());
var optionsBuilder = new DbContextOptionsBuilder<TContext>(dbContextOptions)
.UseApplicationServiceProvider(serviceProvider)
.UseSqlServer(configuration.GetConnectionString("AppsConnection"),
serverOptions => serverOptions.EnableRetryOnFailure(5, TimeSpan.FromSeconds(30), null));
return optionsBuilder.Options;
}).As<DbContextOptions<TContext>>()
.InstancePerLifetimeScope();
builder.Register(context => context.Resolve<DbContextOptions<TContext>>())
.As<DbContextOptions>()
.InstancePerLifetimeScope();
builder.RegisterType<TContext>()
.AsSelf()
.InstancePerLifetimeScope();
}
}
and my appsettings.json as this
"ConnectionStrings": {
"AppsConnection": "Data Source=primary;Initial Catalog=Apps;User Id=me;Password=topsecret"
}
Couldn't really find anything on the web, other than posts where you was in full control of creating the db connection, but I am being supplied the connection via DI.
Using .Net 5 and the applications are worker services.

You can define on custom retry strategy on implementing the interface IExecutionStrategy.
If you want reuse the default SQL Server retry strategy, you can derive from SqlServerRetryingExecutionStrategy on override the method ShouldRetryOn :
public class SqlServerSwitchRetryingExecutionStrategy : SqlServerRetryingExecutionStrategy
{
public string _switchConnectionString;
public SqlServerSwitchRetryingExecutionStrategy(ExecutionStrategyDependencies dependencies, string switchConnectionString)
: base(dependencies, 3)
{
_switchConnectionString = switchConnectionString;
}
protected override bool ShouldRetryOn(Exception exception)
{
if (exception is SqlException sqlException)
{
foreach (SqlError err in sqlException.Errors)
{
switch (err.Number)
{
// For this type of error, switch the connection string and retry
case 1418: // The server can't be reached or does not exist
case 4060: // Cannot open database
case 4064: // Cannot open user default database database
var db = Dependencies.CurrentContext.Context.Database;
var current = db.GetConnectionString();
if(current != _switchConnectionString)
db.SetConnectionString(_switchConnectionString);
return true;
}
}
}
return base.ShouldRetryOn(exception);
}
}
I am not sure which errors to catch for your scenario.
You should test and find the errors to handle.
The full list is available Database engine errors.
To inject the strategy :
new DbContextOptionsBuilder<TContext>(dbContextOptions)
.UseSqlServer(
configuration.GetConnectionString("AppsConnection"),
serverOptions => serverOptions.ExecutionStrategy(dependencies =>
new SqlServerSwitchRetryingExecutionStrategy(
dependencies,
configuration.GetConnectionString("AppsConnectionBackup"))
)
);
If you want a full custom strategy, you can get inspiration from SqlServerRetryingExecutionStrategy.

Environment specific error: Cannot consume scoped service from singleton 'Microsoft.AspNetCore.Hosting.Internal.HostedServiceExecutor'

I have developed a .Net Core background service which was working fine both when running as a service and when debugging in Visual Studio.
Then the following error started happening but only when debugging in Visual Studio
Cannot consume scoped service myDbContext from singleton 'Microsoft.AspNetCore.Hosting.Internal.HostedServiceExecutor'"
On deployment the service does not experience the same problem.
I've read several posts saying to get around this by creating a scoped service, however if I set the project up on a different machine by getting the code from source control, the error doesn't happen.
I have deleted the project from the problem environment and pulled the code from source control again but the issue still arises.
To me this suggests it's an environment issue, rather than a coding one.
The basic code is below.
The issue arises at the line host.Run();.
Does anyone have any pointers as to how to isolate the cause and therefore potentially find a fix?
Thanks
public class Program
{
public static void Main(string[] args)
{
var method = MethodBase.GetCurrentMethod();
var methodName = method.Name;
var type = method.DeclaringType;
Log.Information("{0}.{1}: Started.", type, methodName);
try
{
var isService = !(Debugger.IsAttached || args.Contains("--console"));
if (isService)
{
var pathToExe = Process.GetCurrentProcess().MainModule.FileName;
var pathToContentRoot = Path.GetDirectoryName(pathToExe);
Directory.SetCurrentDirectory(pathToContentRoot);
}
var builder = CreateWebHostBuilder(args.Where(arg => arg != "--console").ToArray());
var host = builder.Build();
if (isService)
{
host.RunAsWebCustomService();
}
else
{
host.Run();
}
return;
}
catch (Exception ex)
{
Log.Fatal(ex, "Error in {0{.{1}: {2}", type, methodName, ex.Message);
throw;
}
finally
{
Log.CloseAndFlush();
}
}
public static IWebHostBuilder CreateWebHostBuilder(string[] args)
{
var method = MethodBase.GetCurrentMethod();
var methodName = method.Name;
var type = method.DeclaringType;
Log.Information("{0}.{1}:called", type, methodName);
var environment = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");
var builder = new ConfigurationBuilder()
.SetBasePath(Path.Combine(AppContext.BaseDirectory))
.AddJsonFile("appsettings.json", optional: true);
var configuration = builder.Build();
var url = configuration["WebHostBuilder:Url"];
return WebHost.CreateDefaultBuilder(args)
.UseUrls(url)
.ConfigureServices(
(hostContext, services) =>{services.AddHostedService<Worker>();})
.UseStartup<Startup>()
.UseSerilog((hostingContext, loggerConfiguration) => loggerConfiguration.ReadFrom.Configuration(hostingContext.Configuration));
}
}

I have been working in dotnet core since 2 years and things that i found out is:-
Do not resolve a scoped service from a singleton. It may cause the service to have incorrect state when processing subsequent requests. It's fine to:
Resolve a singleton service from a scoped or transient service.
Resolve a scoped service from another scoped or transient service.
By default, in the development environment, resolving a service from another service with a longer lifetime throws an exception. For more information, Scope Validation

Merge two Asp.Net Core APIs in one application

I have two Restful APIs projects that am trying to merge in one application project ( new .net core one) I modified the code in Running multiple independent ASP.NET Core pipelines side by side in the same application to accept WebSockets as following the extension method looks like :
public static IApplicationBuilder UseBranchWithServices(
this IApplicationBuilder app,
PathString path,
Type requiredStartup) {
var webHost = WebHost.CreateDefaultBuilder()
.UseStartup(requiredStartup).Build();
var serviceProvider = webHost.Services;
var serverFeatures = webHost.ServerFeatures;
var appBuilderFactory =
serviceProvider.GetRequiredService<IApplicationBuilderFactory>();
var branchBuilder = appBuilderFactory.CreateBuilder(serverFeatures);
var factory = serviceProvider.GetRequiredService<IServiceScopeFactory>();
if (path.Value.Contains("/project2")) {
branchBuilder.Map(
"/project2/ws",
x =>
x.UseMiddleware<project2MicroService.WebSockets.WebSocketMiddleWare>(
serviceProvider.GetService<SceneWebSocketHandler>()));
} else if (path.Value.Contains("/project1")) {
branchBuilder.Map(
"/project1/ws",
x => x.UseMiddleware<project1Service.WebSockets.WebSocketMiddleWare>(
serviceProvider.GetService<project1WebSocketHandler>()));
}
var branchDelegate = branchBuilder.Build();
return app.Map(
path,
builder => {
builder.Use(
async (context, next) => {
if (!context.WebSockets.IsWebSocketRequest) {
await branchDelegate(context).ConfigureAwait(false);
} else {
await context.WebSockets.AcceptWebSocketAsync().ConfigureAwait(false);
await branchDelegate(context).ConfigureAwait(false);
}
});
});
}
and I call it in my new application for example like
app.UseBranchWithServices("/project2", typeof(project2MicroService.Startup));
while running unit tests the WebSocket connection is accepted but the middleware never been hit
any idea how to fix this, please , my unit test
[ClassInitialize]
public static void TestOneTimeSetUp(TestContext context) {
var webHostBuilder = WebHost.CreateDefaultBuilder();
webHostBuilder.UseContentRoot(Directory.GetCurrentDirectory());
webHostBuilder.UseStartup<Startup>();
server = new TestServer(webHostBuilder);
client = server.CreateWebSocketClient();
}
/// <summary>
/// OneTimeTearDown
/// </summary>
[ClassCleanup]
public static void TestOneTimeTeardown() {
server.Dispose();
}
/// <summary>
/// TestWebsocketCanBeCreated
/// </summary>
[TestMethod]
public void TestWebsocketCanBeCreated() {
var TEST1wsUri = new UriBuilder(server.BaseAddress + "project1/ws") { Scheme = "ws" }.Uri;
var TEST1websocket = client.ConnectAsync(TEST1wsUri, CancellationToken.None).Result;
var TEST2wsUri = new UriBuilder(server.BaseAddress + "project2/ws") { Scheme = "ws" }.Uri;
var TEST2websocket = client.ConnectAsync(TEST2wsUri, CancellationToken.None).Result;
Assert.AreEqual(WebSocketState.Open, TEST2websocket.State);
Assert.AreEqual(WebSocketState.Open, TEST1websocket.State);
Task.WaitAll(
TEST1websocket.CloseAsync(
WebSocketCloseStatus.NormalClosure,
"",
CancellationToken.None));
Task.WaitAll(
TEST2websocket.CloseAsync(
WebSocketCloseStatus.NormalClosure,
"",
CancellationToken.None));
Assert.AreEqual(WebSocketState.Closed, TEST2websocket.State);
Assert.AreEqual(WebSocketState.Closed, TEST1websocket.State);
}

You're doing a couple things wrong:
1) you're trying to define route behavior with if/else logic. don't do that.
2) you're not actually declaring what you're trying to hit as part of your pipeline. consider the following:
// https://stackoverflow.com/questions/48216929/how-to-configure-asp-net-core-server-routing-for-multiple-spas-hosted-with-spase
app.Map("/rx", rx => {
rx.UseSpa(rxApp => {
rxApp.Options.SourcePath = "../RX";
if (envIsDevelopment) rxApp.UseProxyToSpaDevelopmentServer("http://localhost:3000");
});
});
app.Map("/V2", ng => {
// https://learn.microsoft.com/en-us/aspnet/core/client-side/spa/angular?view=aspnetcore-2.2
app.UseSpa(angularApp =>
{
angularApp.Options.SourcePath = "../UI";
if (envIsDevelopment) angularApp.UseProxyToSpaDevelopmentServer("http://localhost:4200");
});
});
source
Note that link there: Filip W.'s blog
This is a different use case but it's an example of how you can map two different routes to different destinations. You're trying to switch on the URL and that's not how the pipeline works. It's a declarative pipeline; you have to define the routes according to .NET Core's built-in plugins (or add dependencies that contain other middleware plugins).
Take a look here:
https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.builder.mapextensions.map?view=aspnetcore-3.1
...and don't reinvent the wheel.

I had looked at the solution you got but it didn't work for me. So, we created a solution for that it does exactly the job that you wanted and works seamlessly for a long time.
https://github.com/damianh/lab/tree/master/dotnet/AspNetCoreNestedApps/AspNetCoreNestedApps
If I summarize with code;
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
app.IsolatedMap<NestedStartup>("/nested");
app.IsolatedMap<XApp>("/xroute");
app.Run(async context => await context.Response.WriteAsync("Hello World!"));
}
You can separate your applications based on Startup and routing easily. But, keep that in mind somethings might not work for pipeline since you're branching after the main container built. We covered hosted services for branches.