I have this assert in my test code
Assert.That(() => eventData.Count == 0,
Is.True.After(notificationPollingDelay),
"Received unexpected event with last event data" + eventData.Last().Description());
that asserts some condition after a period of time and on failure produces a message. it fails to run because the message string is constructed when the assert starts and not when the assert ends. therefore the eventData collection is still empty (as it is initially) and the attempt to get the Description of the last item in the collection fails. is there a workaround or decent alternative to this in NUnit or do I have to revert to using Thread.Sleep in my tests?
PS: I'm using NUnit 2.5.10.
You may use this scheme:
var constrain = Is.True.After(notificationPollingDelay);
var condition = constrain.Matches(() => eventData.Count == 0);
Assert.IsTrue(condition,
"Received unexpected event with last event data" +
eventData.Last().Description());
This method is similar to the use Thread.Sleep
In NUnit version 3.50 I had to use a different syntax.
Here is the example:
var delayedConstraint = Is.True.After( delayInMilliseconds: 100000, pollingInterval: 100);
Assert.That( () => yourCondition, delayedConstraint );
This will test whether `yourCondition` is true waiting a certain maximum time using the `DelayedConstraint` created by the `Is.True.After` method.
In this example the DelayedConstraint is configured to use maximum time of 100 seconds polling every 0.1 seconds.
See aslo the legacy NUnit 2.5 documentation for DelayedConstraint.
The simplest answer is "don't include that text in your failure message". I personally almost never include a failure message; if your test is atomic enough you don't need to do it. Usually if I need to figure out a cryptic failure, only a debugger helps anyway.
If you really want to do it, this code should work without managing the threads yourself.
try
{
Assert.That(() => eventData.Count == 0, Is.True.After(notificationPollingDelay));
}
catch(AssertionException)
{
throw new Exception("Received unexpected event with last event data" + eventData.Last().Description());
}
Related
I am using Azure Service Bus and I have the code below (c# .NetCore 3.1). I am constantly getting the error "The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue, or was received by a different receiver instance." when I call "CompleteAsync"
As you can see in the code I have the "ReceiveMode.PeekLock", "AutoComplete = false" and MaxAutoRenewDuration to 5 min. The code that handles the message completes in less than 1 second and I still get that error every single time.
What drove me crazy is that after hours reading posts, rewriting my code and a lot of "try and error" I decided to increase the MaxConcurrentCalls from 1 to 2 and magically the error disappeared.
Does anybody knows what is going on here?
public void OpenQueue(string queueName)
{
var messageHandlerOptions = new MessageHandlerOptions(exceptionReceivedEventArgs =>
{
Log.Error($"Message handler encountered an exception {exceptionReceivedEventArgs.Exception}.");
return Task.CompletedTask;
});
messageHandlerOptions.MaxConcurrentCalls = 1;
messageHandlerOptions.AutoComplete = false;
messageHandlerOptions.MaxAutoRenewDuration = TimeSpan.FromSeconds(300);
messageReceiver = queueManagers.OpenReceiver(queueName, ReceiveMode.PeekLock);
messageReceiver.RegisterMessageHandler(async (message, token) =>
{
if (await ProcessMessage(message)) //really quick operation less than 1 second
{
await messageReceiver.CompleteAsync(message.SystemProperties.LockToken);
}
else
{
await messageReceiver.AbandonAsync(message.SystemProperties.LockToken);
}
}, messageHandlerOptions);
}
I decided to increase the MaxConcurrentCalls from 1 to 2 and magically the error disappeared.
Concurrency and lock duration is not the only variables in the equation. This sounds like a prefetch issue. If enabled, more messages are prefetched than processed to save on the latency and the roundtrips. If the prefetch is too aggressive, messages that are pre-fetched and waiting are still going to be processed, and while the processing would normally be short enough, the combined time of waiting for processing and the actual processing would exceed the lock duration.
I would suggest to:
Increase MaxLockDuration on the queue
Validate the prefetch count
Regarding MaxLockDuration vs MaxAutoRenewDuration these two are tricky. While the first is guaranteed, the second is not and is a best-effort by the client.
I'm writing the solution for my problem as it may help others.
Turns out the root cause of the problem was a quite basic mistake, but the error got me really confused.
The method OpenQueue was called more than once on the same class instance (multiple queues scenario) what was a mistake. The behavior was quite weird. Looks like queueManagers registered all queues as expected but the token got overwritten causing it to always be invalid.
When I wrote:
I decided to increase the MaxConcurrentCalls from 1 to 2 and magically the error disappeared.
Later that statement proved to be incorrect. When I enabled multiple queues that failed miserably.
The block of code I posted here is actually working. What was around it was broken. I was trying to gain some time and ended up writing bad code. I fixed my design to manage things properly and everything is now running smooth.
So, I'm writing some retry logic for acquiring a lock using Polly. The overall timeout value will be provided by the API caller. I know I can wrap a policy in an overall timeout. However, if the supplied timeout value is too low is there a way I can ensure that the policy is executed at least once?
Obviously I could call the delegate separately before the policy is executed but I was just wondering if there was a way to express this requriment in Polly.
var result = Policy.Timeout(timeoutFromApiCaller)
.Wrap(Policy.HandleResult(false)
.WaitAndRetryForever(_ => TimeSpan.FromMilliseconds(500))
.Execute(() => this.TryEnterLock());
If timeoutFromApiCaller is say 1 tick and there's a good chance it takes longer than that to reach the timeout policy then the delegate wouldn't get called (the policy would timeout and throw TimeoutRejectedException).
What I'd like to happen can be expressed as:
var result = this.TryEnterLock();
if (!result)
{
result = Policy.Timeout(timeoutFromApiCaller)
.Wrap(Policy.HandleResult(false)
.WaitAndRetryForever(_ => TimeSpan.FromMilliseconds(500))
.Execute(() => this.TryEnterLock());
}
But it'd be really nice if it could be expressed in pure-Polly...
To be honest I don't understand what does it mean 1 tick, in your case? Is it a nanosecond or greater than that? Your global timeout should be greater than your local timeout.
But as I can see you have not specified a local one. TryEnterLock should receive a TimeSpan in order to do not block the caller for infinite time. If you look at the built in sync primitives most of them provide such a capabilities: Monitor.TryEnter, SpinLock.TryEnter, WaitHandle.WaitOne, etc.
So, just to wrap it up:
var timeoutPolicy = Policy.Timeout(TimeSpan.FromMilliseconds(1000));
var retryPolicy = Policy.HandleResult(false)
.WaitAndRetryForever(_ => TimeSpan.FromMilliseconds(500));
var resilientStrategy = Policy.Wrap(timeoutPolicy, retryPolicy);
var result = resilientStrategy.Execute(() => this.TryEnterLock(TimeSpan.FromMilliseconds(100)));
The timeout and delay values should be adjusted to your business needs. I highly encourage you to log when the global Timeout (onTimeout / onTimeoutAsync) fires and when the retries (onRetry / onRetryAsync) to be able to fine tune / calibrate these values.
EDIT: Based on the comments of this post
As it turned out there is no control over the timeoutFromApiCaller so it can be arbitrary small. (In the given example it is just a few nano-seconds, with the intent to emphasize the problem.) So, in order to have at least one call guarantee we have to make use the Fallback policy.
Instead of calling manually upfront the TryEnterLock outside the policies, we should call it as the last action to satisfy the requirement. Because policies uses escalation, that's why whenever the inner fails then it delegates the problem to the next outer policy.
So, if the provided timeout is so tiny that action can not finish until that period then it will throw a TimeoutRejectedException. With the Fallback we can handle that and the action can be performed again but now without any timeout constraint. This will provide us the desired at least one guarantee.
var atLeastOnce = Policy.Handle<TimeoutRejectedException>
.Fallback((ct) => this.TryEnterLock());
var globalTimeout = Policy.Timeout(TimeSpan.FromMilliseconds(1000));
var foreverRetry = Policy.HandleResult(false)
.WaitAndRetryForever(_ => TimeSpan.FromMilliseconds(500));
var resilientStrategy = Policy.Wrap(atLeastOnce, globalTimeout, foreverRetry);
var result = resilientStrategy.Execute(() => this.TryEnterLock());
I'm trying to write file watcher on certain folder using the reactive extensions library
The idea is to monitor hard drive folder for new files, wait until file is written completely and push event to the subscriber. I do not want to use FileSystemWatcher since it raises Changed event twice for the same file.
So I've wrote it in the "reactive way" (I hope) like below:
var provider = new MessageProviderFake();
var source = Observable.Interval(TimeSpan.FromSeconds(2), NewThreadScheduler.Default).SelectMany(_ => provider.GetFiles());
using (source.Subscribe(_ => Console.WriteLine(_.Name), () => Console.WriteLine("completed to Console")))
{
Console.WriteLine("press Enter to stop");
Console.ReadLine();
}
However I can't find "reactive way" to handle errors. For example, the file directory can be located on the external drive and became unavailable because of connection problem.
So I've added GetFilesSafe that will handle exception errors from the Reactive Extensions:
static IEnumerable<MessageArg> GetFilesSafe(IMessageProvider provider)
{
try
{
return provider.GetFiles();
}
catch (Exception e)
{
Console.WriteLine(e.Message);
return new MessageArg[0];
}
}
and used it like
var source = Observable.Interval(TimeSpan.FromSeconds(2), NewThreadScheduler.Default).SelectMany(_ => GetFilesSafe(provider));
Is there better way to make SelectMany to call provider.GetFiles() even when an exception has been raised? I'm using error counter in such cases to repeat the reading operation N times and then fail (terminate the process).
Is there "try N time and wait Q seconds between attempts" in the Reactive Extensions?
There is a problem with GetFilesSafe also: it returns IEnumerable<MessageArg> for lazy reading however it can raise on iteration and exception will be thrown somewhere in the SelectMany
There's a Retry extension, that just subscribes to the observable again if the current one errors, but it sounds like that won't offer the flexibility you want.
You could build something using Catch, which subscribes to the observable you give it if an error occurs on the outer one. Something like the following (untested):
IObservable<Thing> GetFilesObs(int times, bool delay) {
return Observable
.Return(0)
.Delay(TimeSpan.FromSeconds(delay ? <delay_time> : 0))
.SelectMany(_ => Observable.Defer(() => GetFilesErroringObservable()))
.Catch(Observable.Defer(() => GetFilesObs(times - 1, true)));
}
// call with:
GetFilesObs(<number_of_tries>, false);
As written, this doesn't do anything with the errors other than trigger a retry. In particular, when enough errors have happened, it will just complete without an error, which might not be what you want.
In the script task:
else if (val == 0)
{
Dts.Events.FireError(0, "", "Custom Message ", "", 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
When we have Dts.Events.FireError() in the script task, and when it gets invoked it fails the task as well as displaying the custom error message. So is is good to write the
Dts.TaskResult = (int)ScriptResults.Failure;
to fail the task as in the above code.
Is it not like calling the fail twice?
Any use case we should have this both.
Should we use both
"It depends." How do you want to handle error handling?
My experience has been that it is cleaner to set the TaskResult to Failure and then use precedence constraints out of the task to drive control flow behavior. That is, "yes, this task failed but this package still has work to do." e.g. The file we expected isn't there - that's the error, but I'm going to take an error path to drive the next action (send email alert about missing file)
Otherwise, you get to use the Event Handlers which is totally a valid approach but for all the shops I've consulted in, maybe two have used them well. Many people get tripped up over the possibility that an event is raised several times due to container nesting and reraising of events.
If I know I am killing out execution from a Task, then FireError event can be helpful as it helps me log exactly why I'm aborting processing (File not found exception).
So I have a lambda that makes a point-to-point call to another lambda. We have AWS X-Ray set up so we can monitor performance. However, X-Ray show this odd result where even though the invocation itself takes only a second, the "invoke" call from the original takes a minute and a half.
This makes no sense, since we are calling the lambda as an event (ACK and forget) and using an async call on which we do not await. It really causes problems because even though all lambdas successfully complete and do their work (as we can see from Cloudwatch logs and resulting data in our data store), occasionally that secondary lambda call takes so long that X-Ray times out, which bombs the whole rest of the trace.
Other notes:
We have Active tracing enabled on both lambdas
We do occasionally have cold start times, but as you can see from the screenshot, there is no "initialization" step here, so both lambdas are warm
This particular example was a singular action with no other activity in the system, so it's not like there was a bottleneck due to high load
Does anyone have an explanation for this, and hopefully what we can do to fix it?
Our invocation code (simplified):
var assetIds = new List<Guid> { Guid.NewGuid() };
var request= new AddBulkAssetHistoryRequest();
request.AssetIds = assetIds.ToList();
request.EventType = AssetHistoryEventTypeConstants.AssetDownloaded;
request.UserId = tokenUserId.Value;
var invokeRequest = new InvokeRequest
{
FunctionName = "devkarl02-BulkAddAssetHistory",
InvocationType = InvocationType.Event,
Payload = JsonConvert.SerializeObject(request)
};
var region = RegionEndpoint.GetBySystemName("us-east-1");
var lambdaClient= new AmazonLambdaClient(region)
_ = lambdaClient.InvokeAsync(invokeRequest);
This is also posted over in the AWS Forums (for whatever that is worth): https://forums.aws.amazon.com/thread.jspa?threadID=307615
So it turns out the issue was that we weren't using the await operator. For some reason, that made the calls interminably slow. Making this small change:
_ = await lambdaClient.InvokeAsync(invokeRequest);
made everything else behave properly, both in logs and in x-ray. Not sure why, but hey, it solved the issue.
As far as I understand, not adding the await, causes the call to execute synchronously while adding the await causes the call to happen async.