I recently ran into a problem where I was developing an API which talked to two data sources in some methods. The POST for a couple methods modified SQL data through the use of entity framework as well a data source using as an old SDK that was STA COM based. To get the STA COM SDK code to work correctly from within the API methods, I had to create method attributes that identified the methods as needing to be single threaded. I forced single threading by overriding the InvokeActionAsync() method from ApiControllerActionInvoker. If a method was not given an attribute to be single threaded, the overridden invoker simply used the normal base class InvokeActionAsync().
public class SmartHttpActionInvoker: ApiControllerActionInvoker
{
public override Task<HttpResponseMessage> InvokeActionAsync(HttpActionContext context, CancellationToken cancellationToken)
{
// Determine whether action has attribute UseStaThread
bool useStaThread = context.ActionDescriptor.GetCustomAttributes<UseStaThreadAttribute>().Any();
// If it doesn't, simply return the result of the base method
if (!useStaThread)
{
return base.InvokeActionAsync(context, cancellationToken);
}
// Otherwise, create an single thread and then call the base method
Task<HttpResponseMessage> responseTask = Task.Factory.StartNewSta(() => base.InvokeActionAsync(context, cancellationToken).Result);
return responseTask;
}
}
public static class TaskFactoryExtensions
{
private static readonly TaskScheduler _staScheduler = new StaTaskScheduler(numberOfThreads: 1);
public static Task<TResult> StartNewSta<TResult>(this TaskFactory factory, Func<TResult> action)
{
return factory.StartNew(action, CancellationToken.None, TaskCreationOptions.None, _staScheduler);
}
}
public static void Register(HttpConfiguration config)
{
....
config.Services.Replace(typeof(IHttpActionInvoker), new SmartHttpActionInvoker());
...
}
This worked well until I noticed something odd. My Logging database was logging duplicate records when a method NOT marked as single threaded was throwing a HttpResponseException back to the client. This behavior did not exist when the same method returned OK().
Debugging, I noticed the code execute in the API method, then reach the throw statement. The next line after the exception was thrown to be shown in debugger was the InvokeActionAsync() code I wrote. Following this the method was run again, in full, hitting the thrown exception, the action invoker, and then returning the result to the client. Effectively, it appears my use of overriding the InvokeActionAsync causes the Action invoker to be called twice somehow... but I am not sure how.
EDIT: Confirmed that the System.Threading.Thread.CurrentThread.ManagedThreadId for the current thread when it is thrown and logged is different for each execution of the API method. So, this reinforces my belief two threads are being created instead of one. Still not sure why.
Anyone have any experience with overriding the InvokeActionAsync behavior that might be able to explain this behavior? Thanks!
Related
The Durable Functions documentation specifies the following pattern to set up automatic handling of retries when an exception is raised within an activity function:
public static async Task Run(DurableOrchestrationContext context)
{
var retryOptions = new RetryOptions(
firstRetryInterval: TimeSpan.FromSeconds(5),
maxNumberOfAttempts: 3);
await ctx.CallActivityWithRetryAsync("FlakyFunction", retryOptions, "ABC");
// ...
}
However I can't see a way to check which retry you're up to within the activity function:
[FunctionName("FlakyFunction")]
public static string[] MyFlakyFunction(
[ActivityTrigger] string id,
ILogger log)
{
// Is there a built-in way to tell what retry attempt I'm up to here?
var retry = ??
DoFlakyStuffThatMayCauseException();
}
EDIT: I know it can probably be handled by mangling some sort of count into the RetryOptions.Handle delegate, but that's a horrible solution. It can be handled manually by maintaining an external state each time it's executed, but given that there's an internal count of retries I'm just wondering if there's any way to access that. Primary intended use is debugging and logging, but I can think of many other uses.
There does not seem to be a way to identify the retry. Activity functions are unaware of state and retries. When the CallActivityWithRetryAsync call is made the DurableOrchestrationContext calls the ScheduleWithRetry method of the OrchestrationContext class inside the DurableTask framework:
public virtual Task<T> ScheduleWithRetry<T>(string name, string version, RetryOptions retryOptions, params object[] parameters)
{
Task<T> RetryCall() => ScheduleTask<T>(name, version, parameters);
var retryInterceptor = new RetryInterceptor<T>(this, retryOptions, RetryCall);
return retryInterceptor.Invoke();
}
There the Invoke method on the RetryInterceptor class is called and that does a foreach loop over the maximum number of retries. This class does not expose properties or methods to obtain the number of retries.
Another workaround to help with debugging could be logging statements inside the activity function. And when you're running it locally you can put in a breakpoint there to see how often it stops there. Note that there is already a feature request to handle better logging for retries. You could add your feedback there or raise a new issue if you feel that's more appropriate.
To be honest, I think it's good that an activity is unaware of state and retries. That should be the responsibility of the orchestrator. It would be useful however if you could get some statistics on retries to see if there is a trend in performance degradation.
I have a Web App that does some processing in the background via QueueBackgroundWorkItem.
I'm wiring up unit tests for functionality in the app and when it attempts to invoke this I get the following error:
System.InvalidOperationException occurred
HResult=-2146233079
Message=Operation is not valid due to the current state of the object.
Source=System.Web
StackTrace:
at System.Web.Hosting.HostingEnvironment.QueueBackgroundWorkItem(Func`2 workItem)
Comparing the environments between when this gets invoked from a unit test vs when it gets invoked as part of a running web server, I see that the AppDomain / HostingEnvironment are different.
Without doing a full web app deployment for testing, is there a way to structure the unit test so that the background work item can run in the proper context?
Preferably without changing the existing target code, just by changing the test code - and if that isn't possible, maybe use IOC to run the background work item in an appropriate background thread depending on its context.
update
This works, although probably there is a more elegant way to do it. Basically only runs it background if invoked from a hosted environment:
private void GetSomeData(SomeCriteria criteria, Dictionary<int, List<Tuple<int, string>>> someParam)
{
if (System.Web.Hosting.HostingEnvironment.IsHosted)
{
System.Web.Hosting.HostingEnvironment.QueueBackgroundWorkItem((token) =>
{
GenerateSomeData(token, criteria, someParam);
});
}
else
{
CancellationToken token = new CancellationToken();
GenerateSomeData(token, criteria, someParam);
}
}
I know it's kinda ugly, but I ended up creating my own helper class
public static class BackgroundWorkItemX
{
public static void QueueBackgroundWorkItem(Action<CancellationToken> workItem)
{
try
{
HostingEnvironment.QueueBackgroundWorkItem(workItem);
}
catch (InvalidOperationException)
{
workItem.Invoke(new CancellationToken());
}
}
}
And changed all references to QueueBackgroundWorkItem to this class
I am trying to load a document out of RavenDb via a WebAPI call. When I open an async IDocumentSession and call LoadAsync, I get no exception or result, and the thread exits instantly with no error code.
I was able to bypass all the structure of my API and reproduce the error.
Here is the code that will not work:
public IHttpActionResult GetMyObject(long id)
{
try
{
var session = RavenDbStoreHolderSingleton.Store.OpenAsyncSession();
var myObject= session.LoadAsync<MyObject>("MyObject/1").Result;
return Ok(myObject);
}
catch (Exception e)
{
return InternalServerError(e);
}
}
I simply hard coded the object's Id to 1 for testing, but calling the function for an object that doesn't exist (such as "MyObject/1") has the same result.
However, this code works:
public async Task<IHttpActionResult> GetMyObject(long id)
{
try
{
var session = RavenDbStoreHolderSingleton.Store.OpenAsyncSession();
var myObject= await session.LoadAsync<MyObject>("MyObject/1");
return Ok(myObject);
}
catch (Exception e)
{
return InternalServerError(e);
}
}
Things I tried/fiddled with:
Changing the exceptions that are caught in debugging
Carefully monitoring Raven Studio to see if I could find any problems (I didn't, but I'm not sure I was looking in the right places)
Running the API without the debugger attached to see if the error occurred or if something showed up in Raven Studio (no changes)
So I guess I have stumbled on a "fix", but can someone explain why one of these would fail in such an odd way while the other one would work perfectly fine?
In the real application, the API call did not have the async/await pair, but the code that was making the call was actually using async/await.
Here is the repository class that was failing which caused me to look into this issue:
public async Task<MyObject> Load(string id)
{
return await _session.LoadAsync<MyObject>(id);
}
The first part that is failing is as per design, for ASP.Net async call, you are blocking the Synchronization context, when you call the Result on a Task returned and same Synchronization context is required for call to return the data. Check out the following link by Stephen Cleary, where the same mechanism is explained in detail.
Second part works since that is correct way of using it and it's not getting into the deadlock anymore. First part can only work if you are using the Console application, which doesn't have a synchronization context to block, even other UI like winforms will have a similar issue and need to use the use the Second part of the code
I am using ASP.NET MVC 5 and Identity Framework. When I call UserManager.UpdateAsync(...) my eventhandlers on ApplicationDbContext() SaveChanges will run. Here I am using HttpContext.Current for different purposes (logging and auditing) so I must get say current user. However the whole method runs in a worker thread, and here HttpContext.Current is null.
The biggest problem that the UserManager's "sync" methods are only wrappers around the async version, so the calls are serialized, but the methods (and eventhandlers) still run in a different worker thread.
Please note this issue has nothing to do with the async/await context. In the controller after the await (or calling the 'sync' version) I have back the correct HttpContext, even the controller's method is continuing in an other thread. That's fine.
So the problem is inside the async worker which will run in both the "sync" and async versions. I think I am understanding the phenomena (but I am not happy with the fake 'sync' method versions, real sync methods would not exhibit this issue.) I just does not know how to deal/workaround it.
[btw: Would not it be more natural to implement UserManager's operarations as simple pure sync versions, then wrap them by async multithreaded wrappers?. IF we continue this async fashion without thinking we will soon invent the async assignment operator. It costs me dozens of hours (just this issue), and costs worldwide zillion dollars, I am sure in many cases less return than its price.]
Bonus: We are talking about UserManager which's impact pretty marginal, but the same principles and issues can apply any out of the box library (black box for you) which authors do not implement sync versions and or do not care about the controller thread's context. What about EF, it is not so marginal... and what about DI containers instantiation infrastructure like "request scope" or "session scope". Surely they misbehave if resolving occurs in a thread with no HttpContext.Current. Recently I refreshed SendGrid NuGet, and (as a breaking change) Deliver() method gone, and now only DeliverAsync() is existing...
I would like to have a safe reliable way, how can I access the HttpContext inside this worker for logging and audit purposes.
Sample code, the controller 'sync' version:
[AcceptVerbs(HttpVerbs.Post)]
public virtual ActionResult Edit(ApplicationUser user)
{
// validation etc
// Update() seems to be only a poor wrapper around the async version, still uses a worker thread.
var result = UserManager.Update(user);
// Note: HttpContext is correct here so it is not an async/await problem
// error handling, creating ActionResult etc.
}
Sample code, the controller async version:
[AcceptVerbs(HttpVerbs.Post)]
public virtual async Task<ActionResult> Edit(ApplicationUser user)
{
// validation etc
var result = await UserManager.UpdateAsync(user);
// Note: HttpContext is correct here so it is not an async/await problem
// error handling, creating ActionResult etc.
}
and the event handler where HttpContext is null:
public ApplicationDbContext() : base("DefaultConnection", false)
{
InitializeAudit();
}
private void InitializeAudit()
{
var octx = ((IObjectContextAdapter) this).ObjectContext;
octx.SavingChanges +=
(sender, args) =>
{
// HttpContext.Current is null here
};
}
Any ideas?
As you said, this occurs because of threading. The delegate runs in a different thread, making the HttpContext inaccessible.
You can move the variable outside of the delegate, making it a closure.
private void InitializeAudit()
{
var octx = ((IObjectContextAdapter) this).ObjectContext;
HttpContext context = HttpContext.Current;
octx.SavingChanges +=
(sender, args) =>
{
// context is not null
};
}
You are using asp.net identity through owin,
so one instance of the dbcontext is created per request,
and you can get this reference from anywhere in the request pipeline.
nb. this is handy but i think the dbcontext shouldn't be accessed outside the manager.
In asp.net identity design, only the manager should be aware of the store.
I believe the dbcontext is exposed because several asp.net identity middleware have a dependance on it.
But, it could help resolve you problem:
Allow your custom dbcontext handler to be set outside the class:
public EventHandler SavingChangesEventHandler
{
set
{
(((System.Data.Entity.Infrastructure.IObjectContextAdapter)this).ObjectContext).SavingChanges += value;
}
}
Declare a custom ActionFilter class and register it, then override OnActionExecuting:
Filtering in ASP.NET MVC
https://msdn.microsoft.com/en-us/library/gg416513(VS.98).aspx
public class CustomizeAppDbcontextFilter : ActionFilterAttribute
{
public override void OnActionExecuting(HttpActionContext actionContext)
{
var dbcontext = HttpContext.GetOwinContext().Get<ApplicationDbContext>();
var currentuser = HttpContext.Current.User;
dbcontext.SavingChangesEventHandler = (sender, args) =>
{
// use currentuser
};
}
}
you may need these using statements to be able to call the identity.owin extension methods:
using Microsoft.AspNet.Identity;
using Microsoft.AspNet.Identity.Owin;
You should be in the controller thread because OnActionExecuting is wrapping the controller action.
I did not test it, so it may need some polishing but the concept should work.
We are using Ninject in an ASP.NET Web Api application, and we bind our DbContext with InRequestScope. This works well with most of our requests, because they do all their work synchronously, so the context can be safely disposed after the request is completed.
However, we have on request in which we do an asynchronous web service call, that has a continuation method passed as a callback, and that callback method needs to use the database context. However our request shouldn't wait for the asynchronous service call to finish, but return immediately (this is an explicit requirement).
Here is a simplified example of the situation.
public class MyController : ApiController
{
private readonly MyDbContext dbContext;
private readonly SomeWebService service;
public MyController(MyDbContext dbContext, SomeWebService service)
{
this.dbContext = dbContext;
this.service = service;
}
public IHttpActionResult MyActionWithAsyncCall()
{
// Doing stuff.
// Calling webservice method, passing the Callback as the continuation.
service.MethodWithCallback(param1, param2, this.Callback);
// Returning without waiting for the service call to be completed.
return Ok();
}
private void Callback()
{
// Trying to use the DbContext:
var person = dbContext.People.First();
// The above line sometimes throws exception, because the context has been disposed.
}
}
How should this situation be handled with Ninject? Is there a way to somehow "prolong" the lifetime of a bound DbContext instance explicitly? Or should the Callback method create completely new DbContext? If it should, what scope should it use?
There's is no way to explicitly prolong the lifetime of an object with .InRequestScope() to extend to after the request end.
If there's not a business requirement that the work during the request and # callback must happen in a single transaction i would go for using two DbContext instances. One during the request and one during the callback. Note: As far as i know this also means you can't take an entity from the first context and update/save it in the second context. This means you must only pass identifier (and other data relevant to the operation) from request to callback. The callback has to "create" a new DbContext and retrieve the according entitites from the context.
Conditional Binding Alternative
As an alternative you might declare a special binding for this special case. Ninject supports so called contextual bindings. This means you would have two bindings, the standard binding and a contextual, special case binding:
Bind<DbContext>().ToSelf().InRequestScope();
Bind<DbContext>().ToSelf()
.WhenInjectedInto<SomeController>();
Notice that the second binding does not specify a scope - that means SomeController is responsible to call .Dispose(). In your case that would mean the callback would have to dispose the context. You'd also need to dispose of the context in all errors cases (errors in the callback code, errors occurring before callback is triggered,....).
Also, in reality your application is probably a bite more complex and .WhenInjectedInto<SomeController> is not going to be enough/correct, because you might want to inject the same instance into the controller plus a repository plus a query object.. what not.
That means you will need scoping, but a scope different from .InRequestScope(). You might use .InCallScope() or named scope - both are included in the named scope extension.
Furthermore you would need to adapt the When condition. You could adapt it so to traverse the requests and see if there is FooController anywhere in the request chain. But that's not very performant, instead i would recommend using a ninject IParameter to specify that you want special case treatment. The parameter would be:
public class NonRequestScopedParameter : Ninject.Parameters.IParameter
{
public bool Equals(IParameter other)
{
if (other == null)
{
return false;
}
return other is NonRequestScopedParameter;
}
public object GetValue(IContext context, ITarget target)
{
throw new NotSupportedException("this parameter does not provide a value");
}
public string Name
{
get { return typeof(NonRequestScopedParameter).Name; }
}
// this is very important
public bool ShouldInherit
{
get { return true; }
}
}
which would be applied at the bindings like:
kernel.Bind<SomeController>().ToSelf()
.WithParameter(new NonRequestScopedParameter());
kernel.Bind<DbContext>().ToSelf()
.When(x => x.Parameters.OfType<NonRequestScopedParameter>().Any())
.InCallScope(); // or whatever scope you're using