I've setup a chrono Job using Quartz library for .NET. Basically, I want a job scheduled every hour and every day at a specific minute (i.e. 10). This is done simply with this chrono expression 0 10 * ? * * *, but how can I achieve the following behavior?:
if the program at XX:12, then run it immediately
if the program is started before XX:10, then wait for the chrono scheduler
I've tried this solution but It does not work.
q.AddTrigger(opts =>
opts.ForJob(job)
.WithIdentity("CalcServiceJob-trigger")
.StartNow()
.WithCronSchedule("0 10 * ? * * *")
);
I've also tried this configuration without relevant results
q.AddTrigger(opts =>
opts.ForJob(job)
.WithIdentity("CalcServiceJob-trigger")
.WithCronSchedule("0 10 * ? * * *", x => x.WithMisfireHandlingInstructionFireAndProceed())
);
You can add another trigger that fires immediately and only once against the job:
q.AddTrigger(t => t
.ForJob(job)
.StartNow()
.WithSimpleSchedule(x => x.WithRepeatCount(0).WithInterval(TimeSpan.Zero))
);
Related
I have a list of tasks that I want to execute in parallel using Parallel.ForEach. It starts fine with 4 tasks running in parallel but in the end it decreases to only one task at a time.
Here is the count of parallel tasks in time:
1 2 3 4 4 3 4 4 ... 4 4 4 3 3 1 1 1 1 1 1 1
Max degree of parallelism is set to 4. At the end of execution only one task is executed at one time and all executions run on the same thread. My question is why I am getting this one task at a time execution in the end? How can I avoid this?
Here is the code:
var threadCount = 4;
ThreadPool.SetMinThreads(threadCount, threadCount);
Parallel.ForEach(taskDataList,
new ParallelOptions() {MaxDegreeOfParallelism = threadCount},
(x) => { RunOne(x); });
RunOne function starts external process and waits for it to end. Some suspected that RunOne could have been the problem of lack of parallel execution. To make sure that this is not the case I recreated situation by replacing this function with a sleep call of identical duration.
The code is below. Here t is the list of seconds each task takes. activeCount is the number of currently running tasks and remaining is the number of tasks that still remain in the list.
var t = new List<int>()
{2,2,2,1,1,1,1,1,1,1,
1,1,1,1,1,3,1,1,1,1,
1,1,1,1,1,1,1,1,5,4,
26,12,11,16,44,4,37,26,13,36};
int activeCount = 0;
int remaining = t.Count;
Parallel.ForEach(t, new ParallelOptions() {MaxDegreeOfParallelism = 4},
(x) =>
{
Console.WriteLine($"Active={Interlocked.Increment(ref activeCount)}"+
$"Remaining={Interlocked.Decrement(ref remaining)} " +
$"Run thread={Thread.CurrentThread.ManagedThreadId}");
Thread.Sleep(x * 1000); //Sleep x seconds
Interlocked.Decrement(ref activeCount);
});
At the very end it produces output like this:
Active=2 Remaining=7 Run thread=3
Active=1 Remaining=6 Run thread=3
Active=1 Remaining=5 Run thread=3
Active=1 Remaining=4 Run thread=3
Active=1 Remaining=3 Run thread=3
Active=1 Remaining=2 Run thread=3
Active=1 Remaining=1 Run thread=3
Active=1 Remaining=0 Run thread=3
This output shows that in the end only 1 task is running when 6 tasks still remain. With limit of 4 parallel tasks it does not make any sense. When 6 tasks are still available I would expect to see 4 tasks running in parallel.
Should I use Parallel.ForEach differently or is it a bug/feature?
After looking at reference source of Parallel.ForEach I found out that instead of distributing elements to different threads one by one it splits the list of tasks into chunks and then gives the list of tasks to each thread. It is very inefficient approach for long running tasks
var t = new List<int>()
{2,2,2,1,1,1,1,1,1,1,
1,1,1,1,1,3,1,1,1,1,
1,1,1,1,1,1,1,1,5,4,
26,12,11,16,44,4,37,26,13,36};
int activeCount = 0;
int remaining = t.Count;
var cq = new ConcurrentQueue<int>(t);
var tasks = new List<Task>();
for (int i = 0; i < 4; i++) tasks.Add(Task.Factory.StartNew(() =>
{
int x;
while (cq.TryDequeue(out x))
{
Console.WriteLine($"Active={Interlocked.Increment(ref activeCount)} " +
$"Remaining={Interlocked.Decrement(ref remaining)} " +
$"Run thread={Thread.CurrentThread.ManagedThreadId}");
Thread.Sleep(x * 1000); //Sleep x seconds
Interlocked.Decrement(ref activeCount);
}
}));
Task.WaitAll(tasks.ToArray());
I used 4 parallel tasks as in the first code example. Execution time in this case was 83 seconds when using Parallel.ForEach took 211 seconds. This just proves that Parallel.ForEach is very inefficient in certain cases and that it should be used with caution.
I need to schedule a job to run at 9:00AM, 12:00PM and 5:00PM on Monday to Friday only. Did not find any documentation on FluentScheduler.
I can do it by having multiple(separately for 5 days) Schedule of the job but can we have single Schedule to do this repeatedly on the given time and days?
I would have thought the simplest solution would be to have the Execute() method in your IJob check the day of the week at its entry point and bail out immediately on Saturdays or Sundays...
You could use Weekdays i.e:
var schedule = Schedule(yourJob);
schedule.ToRunEvery(0).Weekdays().At(9, 0);
schedule.ToRunEvery(0).Weekdays().At(12, 0);
schedule.ToRunEvery(0).Weekdays().At(17, 0);
ToRunEvery(0) means we need to start now.
ToRunEvery(1) would wait one interval for first execution - in our case 1 week day.
I ran into the same issue. FluentScheduler isn't robust enough to handle very complex schedules. A better solution would be to use http://www.quartz-scheduler.net/ It is very flexible, is supported by Topshelf, and has support for most IoC containers.
For example in my service I used:
config.Service<Service>(sc =>
{ sc.ScheduleQuartzJob(configurator =>
configurator.WithJob(
() => JobBuilder.Create<DataLoadJob>().WithIdentity("DataLoad", "Job1").Build())
.AddTrigger(() => TriggerBuilder.Create().WithIdentity("DataLoadSchedule", "Job1")
.WithSimpleSchedule(builder => builder.WithIntervalInSeconds(10).RepeatForever()).Build()));
sc.ScheduleQuartzJob(configurator =>
configurator.WithJob(
() => JobBuilder.Create<DataMergeJob>().WithIdentity("DataMerge", "Job1").Build())
.AddTrigger(() => TriggerBuilder.Create().WithIdentity("DataMergeSchedule", "Job1")
.WithCronSchedule("0 30 7-20/3 ? * MON-FRI").Build()));
sc.ConstructUsingSimpleInjector();
sc.WhenStarted((s, h) => s.Start(h));
sc.WhenStopped((s, h) => s.Stop(h));
});
This is a fragment from a Topshelf service using SimpleInjector along with Quartz.
public class Example: Registry
{
public Example()
{
Schedule(() =>
{
DayOfWeek[] available = new DayOfWeek[] { DayOfWeek.Monday, DayOfWeek.Tuesday, DayOfWeek.Wednesday, DayOfWeek.Thursday, DayOfWeek.Friday };
if (DateTime.Now.DayOfWeek.IsOn(available) && (DateTime.Now.Hour == 8 && DateTime.Now.Minute == 0))//etc
{
//code
}
}).WithName("Example").ToRunEvery(0).Hours().At(0).Between(8, 0, 17, 0);
}
}
I'm trying to "throttle" an IObservable in (what I think is) a different way of the standard throttle methods.
I want to ignore values for 1s following a first non ignored value in the stream.
For example, if 1s=5 dashes
source: --1-23--45-----678901234
result: --1-----4------6----1---
Any ideas on how to achieve this?
Here is an idiomatic way to do this in Rx, as an extension method - an explanation and example using your scenario follows.
The desired function works a lot like Observable.Throttle but emits qualifying events as soon as they arrive rather than delaying for the duration of the throttle or sample period. For a given duration after a qualifying event, subsequent events are suppressed:
public static IObservable<T> SampleFirst<T>(
this IObservable<T> source,
TimeSpan sampleDuration,
IScheduler scheduler = null)
{
scheduler = scheduler ?? Scheduler.Default;
return source.Publish(ps =>
ps.Window(() => ps.Delay(sampleDuration,scheduler))
.SelectMany(x => x.Take(1)));
}
The idea is to use the overload of Window that creates non-overlapping windows using a windowClosingSelector that uses the source time-shifted back by the sampleDuration. Each window will therefore: (a) be closed by the first element in it and (b) remain open until a new element is permitted. We then simply select the first element from each window.
In the following example, I have repeated exactly your test scenario modelling one "dash" as 100 ticks. Note the delay is specified as 499 ticks rather than 500 due to the resolution of passing events between multiple schedulers causing 1 tick drifts - in practice you wouldn't need to dwell on this as single tick resolutions is unlikely to be meaningful. The ReactiveTest class and OnNext helper methods are made available by including the Rx testing framework nuget package rx-testing:
public class Tests : ReactiveTest
{
public void Scenario()
{
var scheduler = new TestScheduler();
var test = scheduler.CreateHotObservable<int>(
// set up events as per the OP scenario
// using 1 dash = 100 ticks
OnNext(200, 1),
OnNext(400, 2),
OnNext(500, 3),
OnNext(800, 4),
OnNext(900, 5),
OnNext(1500, 6),
OnNext(1600, 7),
OnNext(1700, 8),
OnNext(1800, 9),
OnNext(1900, 0),
OnNext(2000, 1),
OnNext(2100, 2),
OnNext(2200, 3),
OnNext(2300, 4)
);
test.SampleFirst(TimeSpan.FromTicks(499), scheduler)
.Timestamp(scheduler)
.Subscribe(x => Console.WriteLine(
"Time: {0} Value: {1}", x.Timestamp.Ticks, x.Value));
scheduler.Start();
}
}
Note that output is as per your scenario:
Time: 200 Value: 1
Time: 800 Value: 4
Time: 1500 Value: 6
Time: 2000 Value: 1
This should do the trick. There may be a shorter implementation.
The accumulate in the Scan stores the Timestamp of the last kept Item and marks whether to Keep each item.
public static IObservable<T> RateLimit<T>(this IObservable<T> source, TimeSpan duration)
{
return observable
.Timestamp()
.Scan(
new
{
Item = default(T),
Timestamp = DateTimeOffset.MinValue,
Keep = false
},
(a, x) =>
{
var keep = a.Timestamp + duration <= x.Timestamp;
return new
{
Item = x.Value,
Timestamp = keep ? x.Timestamp : a.Timestamp,
Keep = keep
};
}
})
.Where(a => a.Keep)
.Select(a => a.Item);
}
I'm trying to turn an IEnumerable into an IObservable that delivers its items in chunks one second apart.
var spartans = Enumerable.Range(0, 300).ToObservable();
spartans
.Window(30)
.Zip(Observable.Timer(DateTimeOffset.Now, TimeSpan.FromMilliseconds(1000)), (x, _) => x)
.SelectMany(w => w)
.Subscribe(
n => Console.WriteLine("{0}", n),
() => Console.WriteLine("all end"));
With this code, the only thing that is printed is "all end" after ten seconds. If I remove the .Zip then the entire sequence prints instantaneously, and if I remove the .Window and .SelectMany then the entire sequence prints one item per second. If I peek into the "windowed" observable inside the lambda passed to SelectMany, I can see that it is empty. My question is, why?
The problem is occurring because of how Window works with a count - and this one isn't particularly intuitive!
As you know, Window serves a stream of streams. However, with a count, the child streams are "warm" - i.e. when an observer of this stream receives a new window in it's OnNext handler, it must subscribe to it before it cedes control back to the observable, or the events are lost.
Zip doesn't "know" it's dealing with this situation, and doesn't give you the opportunity to subscribe to each child window before it grabs the next.
If you remove the Zip, you see all the events because the SelectMany does subscribe to all the child windows as it receives them.
The easiest fix is to use Buffer instead of Window - make that one change and your code works. That's because Buffer works very similarly to SelectMany, effectively preserving the windows by doing this:
Window(30).SelectMany(x => x.ToList())
The elements are no longer warm windows but are crystallized as lists, and your Zip will now work as expected, with the following SelectMany flattening the lists out.
Important Performance Consideration
It's important to note that this approach will cause the entire IEnumerable<T> to be run through in one go. If the source enumerable should be lazily evaluated (which is usually desirable), you'll need to go a different way. Using a downstream observable to control the pace of an upstream one is tricky ground.
Let's replace your enumerable with a helper method so we can see when each batch of 30 is evaluated:
static IEnumerable<int> Spartans()
{
for(int i = 0; i < 300; i++)
{
if(i % 30 == 0)
Console.WriteLine("30 More!");
yield return i;
}
}
And use it like this (with the Buffer "fix" here, but the behaviour is similar with Window):
Spartans().ToObservable()
.Buffer(30)
.Zip(Observable.Timer(DateTimeOffset.Now,
TimeSpan.FromMilliseconds(1000)),
(x, _) => x)
.SelectMany(w => w)
.Subscribe(
n => Console.WriteLine("{0}", n),
() => Console.WriteLine("all end"));
Then you see this kind of output demonstrating how the source enumerable is drained all at once:
30 More!
0
1
...miss a few...
29
30 More!
30 More!
30 More!
30 More!
30 More!
30 More!
30 More!
30 More!
30 More!
30
31
32
...etc...
To truly pace the source, rather than using ToObservable() directly you could do the following. Note the Buffer operation on the Spartans() IEnumerable<T> comes from nuget package Ix-Main - added by the Rx team to plug a few holes on the IEnumerable<T> monad:
var spartans = Spartans().Buffer(30);
var pace = Observable.Timer(DateTimeOffset.Now, TimeSpan.FromMilliseconds(1000));
pace.Zip(spartans, (_,x) => x)
.SelectMany(x => x)
.Subscribe(
n => Console.WriteLine("{0}", n),
() => Console.WriteLine("all end"));
And the output becomes a probably much more desirable lazily evaluated output:
30 More!
0
1
2
...miss a few...
29
30 More!
30
31
32
...miss a few...
59
30 More!
60
61
62
...etc
I'm not sure how to get this working with Window, but what about this:
var spartans = Enumerable.Range(0, 300).ToObservable();
spartans
.Select(x => Observable.Timer(TimeSpan.FromSeconds(1)).Select(_ => x))
.Merge(30);
Due to twitter rating limitation of 180 request per 15 minute. I made this implementation and delay to the task. But it doesn't seems to work. Whats an issue with this?
What i implemented is actually am giving a 15 minutes wait after 180 request. Whether my implementation correct?
var currentRequestIndex = 1;
var timeToDelay = 0;
foreach (var item in items)
{
var contactFeed = item;
if(currentRequestIndex % 180 == 0)
{
timeToDelay = currentRequestIndex*5000;
}
Delay(timeToDelay * 5000).ContinueWith(_ => Task.Factory.StartNew(
() =>
-- call to twitter api here
));
currentRequestIndex++;
}
public Task Delay(int milliseconds)
{
var tcs = new TaskCompletionSource<object>();
new Timer(_ => tcs.SetResult(null)).Change(milliseconds, -1);
return tcs.Task;
}
Well, you set timeToDelay to 0 and then wait timeToDelay * 5000 which given the former is also 0.
Solution 1 - Spread them evenly
Let's assume your network has no lag and all requests are send to twitter immediately. Then in order to spread your requests evenly during the 15 minutes interval you should delay the ith request by precisely i * 15 * 6000 / 180
foreach (var item in items)
{
var contactFeed = item;
delayTime = currentRequestIndex * 15 * 6000 / 180;
Delay(timeToDelay).ContinueWith(_ => Task.Factory.StartNew(
() =>
-- call to twitter api here
));
currentRequestIndex++;
}
Solution 2 - Send them all at once, wait for the rest of the 15 minutes to pass
I'll just post the code, it's pretty much self-explanatory.
Action makeRequests = () =>
{
DateTime start = DateTime.Now;
foreach (var item in items)
{
// Call twitter api here
}
TimeSpan diff = DateTime.Now - start;
Delay(15 * 6000 - diff.Milliseconds).ContinueWith(_ => Task.StartNew(makeRequests));
};
makeRequests();
P. S. By the looks of it, are using .NET v4.0, but if I'm mistaken and you are compiling against v4.5 you can use the built-in Task.Delay method.