I am having Parallel For loop for following statements but I want to use async task not Parallel. Any idea how can i use async task on same statements? I don't need fully working code but just an idea about how to replace async task with Parallel. Happy coding
Parallel.For(0, allRequests.Count(), i =>
{
var rand = new Random();
var token = allTokens.ElementAt(rand.Next(allTokens.Count()));
var accessKey = token.AccessKey;
var secretKey = token.SecretKey;
using (var ctx = new db_mytestdb())
{
var firstRequest = allRequests[i];
Console.WriteLine("Started scan for: " + firstRequest.SearchedUser.EbayUsername + " and using token: " + allTokens[i % allTokens.Count].TokenName);
var bulkScannedItems = new ConcurrentBag<BulkScannedItems>();
var userPreferences = ctx.UserPreferences.FirstOrDefault(x => x.UserId == firstRequest.UserId);
var userBrekEven = userPreferences.BreakEven;
var intoPast = DateTime.Now.Subtract(TimeSpan.FromDays(firstRequest.Range));
var filteredProducts = ctx.EbayUserTransactions.Where(x => x.SearchedUserID == firstRequest.SearchedUserID && x.TransactionDate >= intoPast && x.TransactionDate <= firstRequest.SearchedUser.LastUpdatedAt)
.ToList()
.GroupBy(x => x.ItemID).Select(x => new ResultItem()
{
ItemID = x.Key,
SaleNumber = x.Sum(y => y.QuantityPurchased)
})
.Where(x => x.SaleNumber >= firstRequest.MinSales)
.ToList();
var itemSpecifics = ctx.SearchedUserItems.Where(x => x.SearchedUserID == firstRequest.SearchedUserID).ToList();
foreach (var item in itemSpecifics)
{
foreach (var filtered in filteredProducts)
{
if (item.ItemID == filtered.ItemID)
{
if (item.UPC != null)
{
filtered.UPC = item.UPC;
}
else
{
filtered.UPC = "does not apply";
}
if (item.EAN != null)
{
filtered.EAN = item.EAN;
}
else
{
filtered.EAN = "does not apply";
}
if (item.MPN != null)
{
filtered.MPN = item.MPN;
}
else
{
filtered.MPN = "does not apply";
}
}
}
}
var bulkScanner = new BulkScannerAlgorithm();
foreach (var dbItem in filteredProducts)
{
var amazonItem = bulkScanner.Found(dbItem.UPC, dbItem.ItemID, accessKey, secretKey);
if (amazonItem.Found)
{
bulkScanner.InsertAmazonData(firstRequest, bulkScannedItems, userBrekEven, amazonItem);
continue;
}
amazonItem = bulkScanner.Found(dbItem.EAN, dbItem.ItemID, accessKey, secretKey);
if (amazonItem.Found)
{
bulkScanner.InsertAmazonData(firstRequest, bulkScannedItems, userBrekEven, amazonItem);
continue;
}
amazonItem = bulkScanner.Found(dbItem.MPN, dbItem.ItemID, accessKey, secretKey);
if (amazonItem.Found)
{
bulkScanner.InsertAmazonData(firstRequest, bulkScannedItems, userBrekEven, amazonItem);
continue;
}
}
List<BulkScannedItems> filteredCompleteBulk;
if (firstRequest.IsPrimeOnly == true)
{
filteredCompleteBulk = bulkScannedItems.Where(x => x.CalculatedProfit >= firstRequest.MinProfit && x.IsPrime == true && x.EbayPrice >= firstRequest.minPrice && x.EbayPrice <= firstRequest.maxPrice).DistinctBy(x => x.ASIN).ToList();
}
else
{
filteredCompleteBulk = bulkScannedItems.Where(x => x.CalculatedProfit >= firstRequest.MinProfit && x.EbayPrice >= firstRequest.minPrice && x.EbayPrice <= firstRequest.maxPrice).DistinctBy(x => x.ASIN).ToList();
}
EFBatchOperation.For(ctx, ctx.BulkScannedItems).InsertAll(filteredCompleteBulk);
ctx.user_scanReq_update(firstRequest.UserSellerScanRequestId);
Console.WriteLine("Scan complete for user: " + firstRequest.SearchedUser.EbayUsername);
}
});
Parallelism and asynchrony are both forms of concurrency, but parallelism works by dividing the problem among multiple threads, and asynchrony works by freeing up threads. So they're kind of opposites in how they work.
That said, to make the code asynchronous, you'd start from your lowest-level I/O calls, e.g., the EF ToList and presumably also whatever APIs are used in the implementation of InsertAll. Replace those with asynchronous equivalents (e.g., ToListAsync) and call them with await.
Next, you'd need to replace the Parallel.For loop with code that creates a collection of asynchronous tasks and then (asynchronously) waits for them all, something like:
var tasks = allRequests.Select(async request => { ... });
await Task.WhenAll(tasks);
That's the basic pattern for asynchronous concurrency.
If you find that you do need true parallelism (multiple threads) in addition to asynchrony, consider using TPL Dataflow.
A little clarification, whether you are using Parallel.For/Foreach, Tasks, or await/async, they are all using the same thing behind the scenes (albeit slightly differently). You should always pick the one that fits your problem the best.
If you want to replace the parallel.for with a method of return type Task, that is straight forward enough but you would end up waiting for this piece to be done before you continued your processing.
Async/Await is generally used when dealing with UIs and web calls, it doesn't appear to be useful here.
What is it that you are trying to accomplish? Why the need to 'replace async task with Parallel'?
The general way you would off load a method to a task would be
Task<T> task = Task<T>.Factory.StartNew(() =>
{
});
or
public Task<T> task()
{
.....
}
Related
I've got a search box that I'm providing autocomplete suggestions for but it's really slow, it takes multiple seconds for suggestions to appear. I'm pretty sure my code is inefficient but I'm not sure the best way to improve it, any suggestions?
[HttpPost]
[Route("search")]
public virtual JsonResult Search(string term)
{
var result = new List<SearchResult>();
if (!String.IsNullOrWhiteSpace(term))
{
var searchTerms = term.ToLower().Split(' ');
List<Card> resultList = null;
foreach (var query in searchTerms)
{
if (resultList == null)
{
resultList = CardRepository.FindAll().Where(x => x.Name.ToLower().Contains(query) || x.Set.SetName.ToLower().Contains(query) || x.Variant.ToLower().Contains(query)
|| x.CardNumber.ToLower().Contains(query) || (query == "holo" && x.IsHolo)).ToList();
}
else
{
resultList = resultList.Where(x => x.Name.ToLower().Contains(query) || x.Set.SetName.ToLower().Contains(query) || x.Variant.ToLower().Contains(query)
|| x.CardNumber.ToLower().Contains(query) || (query == "holo" && x.IsHolo)).ToList();
}
}
foreach (var item in resultList.Take(10))
{
result.Add(new SearchResult()
{
label = item.FullCardName,
value = item.CardId.ToString()
});
}
}
return Json(result);
}
EDIT: Added the FindAll() code.
private readonly IDatabase _database;
public IQueryable<Card> FindAll()
{
return _database.CardDataSource.OrderBy(a => a.Name).AsQueryable();
}
SOLUTION: Going on the advice from the comments and with reference to this post Full Text Search with LINQ I moved my searching to the repository as a method and the result is almost instant autocomplete suggestions. I'm not sure how much better I could make the performance but it's easily usable in its current state.
public Card[] Search(string[] searchTerms)
{
IQueryable<Card> cardQuery = _database.CardDataSource;
foreach(var term in searchTerms)
{
var currentTerm = term.Trim();
cardQuery = cardQuery.Where(p => (p.Name.Contains(currentTerm) ||
p.Variant.Contains(currentTerm) ||
p.CardNumber.Contains(currentTerm) ||
p.Set.SetName.Contains(currentTerm) ||
(term == "holo" && p.IsHolo) ||
(term == "reverse" && p.IsHolo))
);
}
return cardQuery.Take(10).ToArray();
}
[HttpPost]
[Route("search")]
public virtual JsonResult Search(string term)
{
var result = new List<SearchResult>();
if (!String.IsNullOrWhiteSpace(term))
{
var searchTerms = term.ToLower().Split(' ');
var resultList = CardRepository.Search(searchTerms);
foreach (var item in resultList)
{
result.Add(new SearchResult()
{
label = item.FullCardName,
value = item.CardId.ToString()
});
}
}
return Json(result);
}
I think that the main problem is that you're using .FindAll() which returns a List<T>.
This means that when you say CardRepository.FindAll() it gets all of the records into an in-memory list and then your subsequent refining queries (e.g. Where(x => x.Name.ToLower().Contains(query)) and so on) are all run against the entire list. So it's right that it's returning really slowly.
You could try rewriting it by simply removing the .FindAll() and see what happens.
Please note, I'm just giving you the main problem, there are other issues, but none is as important as this one.
You could use multi-threading like this (pseudo-C# code):
var allCards = CardRepository.FindAll().ToArray(); // Ensure array.
query = query.ToUpper();
var nameTask = Task.StartNew(() => allCards.Where(x => x.Name.ToUpper().Contains(query)).ToArray());
var setTask = Task.StartNew(() => allCards.Where(x => x.Set.SetName.ToUpper().Contains(query)).ToArray());
var variantTask = Task.StartNew(() => allCards.Where(x => x.Variant.ToUpper().Contains(query)).ToArray());
var cardNumberTask = Task.StartNew(() => allCards.Where(x => x.CardNumber.ToUpper().Contains(query)).ToArray());
var holoTask = Task.StartNew(() => allCards.Where(x => query == "holo" && x.IsHolo).ToArray());
Task.WaitAll(new Task[] {nameTask, setTask, variantTask, cardNumberTask, holoTask});
var result = (nameTask.Result + setTask.Result + variantTask.Result + cardNumberTask.Result + halaTask.Result).Distinct().ToArray();
I have the following code:
Task.Factory.ContinueWhenAll(items.Select(p =>
{
return CreateItem(p);
}).ToArray(), completedTasks => { Console.WriteLine("completed"); });
Is it possible to convert ContinueWhenAll to a synchronous method? I want to switch back between async and sync.
Edit: I should metnion that each of the "tasks" in the continuewhenall method should be executing synchronously.
If you want to leave your existing code intact and have a variable option of executing synchronously you should make these changes:
bool isAsync = false; // some flag to check for async operation
var batch = Task.Factory.ContinueWhenAll(items.Select(p =>
{
return CreateItem(p);
}).ToArray(), completedTasks => { Console.WriteLine("completed"); });
if (!isAsync)
batch.Wait();
This way you can toggle it programmatically instead of by editing your source code. And you can keep the continuation code the same for both methods.
Edit:
Here is a simple pattern for having the same method represented as a synchronous and async version:
public Item CreateItem(string name)
{
return new Item(name);
}
public Task<Item> CreateItemAsync(string name)
{
return Task.Factory.StartNew(() => CreateItem(name));
}
Unless am mistaken this is what you're looking for
Task.WaitAll(tasks);
//continuation code here
i think you can try this.
using TaskContinuationOptions for a simple scenario.
var taskFactory = new TaskFactory(TaskScheduler.Defau
var random = new Random();
var tasks = Enumerable.Range(1, 30).Select(p => {
return taskFactory.StartNew(() => {
var timeout = random.Next(5, p * 50);
Thread.Sleep(timeout / 2);
Console.WriteLine(#" 1: ID = " + p);
return p;
}).ContinueWith(t => {
Console.WriteLine(#"* 2: ID = " + t.Result);
}, TaskContinuationOptions.ExecuteSynchronously);
}).ToArray();
Task.WaitAll(tasks);
or using TPL Dataflow for a complex scenario.
var step2 = new ActionBlock<int>(i => {
Thread.Sleep(i);
Console.WriteLine(#"* 2: ID = " + i);
}, new ExecutionDataflowBlockOptions {
MaxDegreeOfParallelism = 1,
//MaxMessagesPerTask = 1
});
var random = new Random();
var tasks = Enumerable.Range(1, 50).Select(p => {
return Task.Factory.StartNew(() => {
var timeout = random.Next(5, p * 50);
Thread.Sleep(timeout / 2);
Console.WriteLine(#" 1: ID = " + p);
return p;
}).ContinueWith(t => {
Thread.Sleep(t.Result);
step2.Post(t.Result);
});
}).ToArray();
await Task.WhenAll(tasks).ContinueWith(t => step2.Complete());
await step2.Completion;
I'm optimizing a method with a number of Linq queries. So far the execution time is around 3 seconds and I'm trying to reduce it. There is quite a lot of operations and calculations happening in the method, but nothing too complex.
I will appreciate any suggections and ideas how the performance can be improved and code optimized.
The whole code of the method(Below I'll point where I have the biggest delay):
public ActionResult DataRead([DataSourceRequest] DataSourceRequest request)
{
CTX.Configuration.AutoDetectChangesEnabled = false;
var repoKomfortaktion = new KomfortaktionRepository();
var komfortaktionen = CTX.Komfortaktionen.ToList();
var result = new List<AqGeplantViewModel>();
var gruppen = new HashSet<Guid?>(komfortaktionen.Select(c => c.KomfortaktionsGruppeId).ToList());
var hochgeladeneKomplettabzuege = CTX.Komplettabzug.Where(c => gruppen.Contains(c.KomfortaktionsGruppeId)).GroupBy(c => new { c.BetriebId, c.KomfortaktionsGruppeId }).Select(x => new { data = x.Key }).ToList();
var teilnehmendeBetriebe = repoKomfortaktion.GetTeilnehmendeBetriebe(CTX, gruppen);
var hochgeladeneSperrlistenPlz = CTX.SperrlistePlz.Where(c => gruppen.Contains(c.KomfortaktionsGruppeId) && c.AktionsKuerzel != null)
.GroupBy(c => new { c.AktionsKuerzel, c.BetriebId, c.KomfortaktionsGruppeId }).Select(x => new { data = x.Key }).ToList();
var hochgeladeneSperrlistenKdnr = CTX.SperrlisteKdnr.Where(c => gruppen.Contains(c.KomfortaktionsGruppeId) && c.AktionsKuerzel != null)
.GroupBy(c => new { c.AktionsKuerzel, c.BetriebId, c.KomfortaktionsGruppeId }).Select(x => new { data = x.Key }).ToList();
var konfigsProAktion = CTX.Order.GroupBy(c => new { c.Vfnr, c.AktionsId }).Select(c => new { count = c.Count(), c.Key.AktionsId, data = c.Key }).ToList();
foreach (var komfortaktion in komfortaktionen)
{
var item = new AqGeplantViewModel();
var zentraleTeilnehmer = teilnehmendeBetriebe.Where(c => c.TeilnahmeStatus.Any(x => x.KomfortaktionId == komfortaktion.Id && x.AktionsTypeId == 1)).ToList();
var lokaleTeilnehmer = teilnehmendeBetriebe.Where(c => c.TeilnahmeStatus.Any(x => x.KomfortaktionId == komfortaktion.Id && x.AktionsTypeId == 2)).ToList();
var hochgeladeneSperrlistenGesamt =
hochgeladeneSperrlistenPlz.Count(c => c.data.AktionsKuerzel == komfortaktion.Kuerzel && c.data.KomfortaktionsGruppeId == komfortaktion.KomfortaktionsGruppeId) +
hochgeladeneSperrlistenKdnr.Count(c => c.data.AktionsKuerzel == komfortaktion.Kuerzel && c.data.KomfortaktionsGruppeId == komfortaktion.KomfortaktionsGruppeId);
item.KomfortaktionId = komfortaktion.KomfortaktionId;
item.KomfortaktionName = komfortaktion.Aktionsname;
item.Start = komfortaktion.KomfortaktionsGruppe.StartAdressQualifizierung.HasValue ? komfortaktion.KomfortaktionsGruppe.StartAdressQualifizierung.Value.ToString("dd.MM.yyyy") : string.Empty;
item.LokalAngemeldet = lokaleTeilnehmer.Count();
item.ZentralAngemeldet = zentraleTeilnehmer.Count();
var anzHochgelandenerKomplettabzuege = hochgeladeneKomplettabzuege.Count(c => zentraleTeilnehmer.Count(x => x.BetriebId == c.data.BetriebId) == 1) +
hochgeladeneKomplettabzuege.Count(c => lokaleTeilnehmer.Count(x => x.BetriebId == c.data.BetriebId) == 1);
item.KomplettabzugOffen = (zentraleTeilnehmer.Count() + lokaleTeilnehmer.Count()) - anzHochgelandenerKomplettabzuege;
item.SperrlisteOffen = (zentraleTeilnehmer.Count() + lokaleTeilnehmer.Count()) - hochgeladeneSperrlistenGesamt;
item.KonfigurationOffen = zentraleTeilnehmer.Count() - konfigsProAktion.Count(c => c.AktionsId == komfortaktion.KomfortaktionId && zentraleTeilnehmer.Any(x => x.Betrieb.Vfnr == c.data.Vfnr));
item.KomfortaktionsGruppeId = komfortaktion.KomfortaktionsGruppeId;
result.Add(item);
}
return Json(result.ToDataSourceResult(request));
}
The first half (before foreach) takes half a second which is okay. The biggest delay is inside foreach statement in the first iteration and in particular in these lines, execution of zentraleTeilnehmer takes 1.5 second for the first time.
var zentraleTeilnehmer = teilnehmendeBetriebe.Where(c => c.TeilnahmeStatus.Any(x => x.KomfortaktionId == komfortaktion.Id && x.AktionsTypeId == 1)).ToList();
var lokaleTeilnehmer = teilnehmendeBetriebe.Where(c => c.TeilnahmeStatus.Any(x => x.KomfortaktionId == komfortaktion.Id && x.AktionsTypeId == 2)).ToList();
TeilnehmendeBetriebe has over 800 lines, where TeilnahmeStatus property has normally around 4 items. So, maximum 800*4 iterations, which is not a huge number afterall...
Thus, I'm mostly interected in optimizing these lines, hoping to reduce execution time to half a second or so.
What I tried:
Rewrite Linq to foreach: didn't help, same time... probably not surprising, but was worth a try.
foreach (var tb in teilnehmendeBetriebe) //836 items
{
foreach (var ts in tb.TeilnahmeStatus) //3377 items
{
if (ts.KomfortaktionId == komfortaktion.Id && ts.AktionsTypeId == 1)
{
testResult.Add(tb);
break;
}
}
}
Selecting particular columns for teilnehmendeBetriebe with .Select(). Didn't help either.
Neither helped other small manipulations I tried.
What is interesting - while the first iteration of foreach can take up to 2 seconds, the second and further take just milisecons, so .net is capable of optimizing or reusing calculation data.
Any advice on what can be changed in order to improve performance is very welcome!
Edit:
TeilnahmeBetriebKomfortaktion.TeilnahmeStatus is loaded eagerly in the method GetTeilnehmendeBetriebe:
public List<TeilnahmeBetriebKomfortaktion> GetTeilnehmendeBetriebe(Connection ctx, HashSet<Guid?> gruppen)
{
return ctx.TeilnahmeBetriebKomfortaktion.Include(
c => c.TeilnahmeStatus).ToList();
}
Edit2:
The query which is sent when executing GetTeilnehmendeBetriebe:
SELECT
[Extent1].[Id] AS [Id],
[Extent1].[BetriebId] AS [BetriebId],
[Extent1].[MandantenId] AS [MandantenId],
[Extent1].[CreatedUser] AS [CreatedUser],
[Extent1].[UpdatedUser] AS [UpdatedUser],
[Extent1].[CreatedDate] AS [CreatedDate],
[Extent1].[UpdatedDate] AS [UpdatedDate],
[Extent1].[IsDeleted] AS [IsDeleted]
FROM [Semas].[TeilnahmeBetriebKomfortaktion] AS [Extent1]
WHERE [Extent1].[IsDeleted] <> cast(1 as bit)
My assumption is that TeilnahmeBetriebKomfortaktion.TeilnahmeStatus is a lazy loaded collection, resulting in the N + 1 problem. You should eagerly fetch that collection to improve your performance.
The following iterations of the foreach loop are fast, because after the first iteration those objects are no longer requested from the database server but are server from memory.
I was looking for this and I couldn't figure how to do it.
I have a some threats (tasks) in an App.
foreach (string targetMachine in targetMachines)
{
Task<int> task = Task.Run(() => Magic(targetMachine));
}
I created a task for every object in an array.
The task returns a value.
I want to process the values and act based on them. As example, if task return 0 dont do anything, if returns 1 write a log, if returns 2 run a process.
How can I acoomplish this?
If I process the return values inside the foreach:
foreach (string targetMachine in targetMachines)
{
Task<int> task = Task.Run(() => Magic(targetMachine));
Task.Waitforexit()
if (task.result == 2)
{
do something
}
}
I think, task are not going to be useful and the programa will wait each task to be completed to continue.
It can be something like this:
foreach (string targetMachine in targetMachines)
{
Task.Run(() =>
{
var result = Magic(targetMachine);
if (result == 2)
{
DoSomething();
}
});
}
OR (using async/await)
foreach (string targetMachine in targetMachines)
{
var result = await Task.Run(() => Magic(targetMachine));
if (result == 2)
{
DoSomething();
}
}
OR (using ContinueWith)
foreach (string targetMachine in targetMachines)
{
Task<int>.Run(()=>Magic(targetMachine))
.ContinueWith(t =>
{
if (t.Result == 2)
{
DoSomething();
}
});
}
If you want to wait to finish all of your tasks
Parallel.ForEach(targetMachines, targetMachine =>
{
var result = Magic(targetMachine);
if (result == 2)
{
DoSomething();
}
});
Have a look at Task.ContinueWith()
http://msdn.microsoft.com/en-us/library/dd321405.aspx
When each task is complete it passes the result to the ContinueWith which can then either do nothing, log or call a method like you want.
Additionally you could run the foreach .AsParallel() and remove the tasks altogether.
http://msdn.microsoft.com/en-us/library/dd413602.aspx
I'll add a new variant in addition to the variants given in L.B.'s answer:
var tasks = targetMachines.Select(x => Magic(x));
Task.WaitAll(tasks); //fork-join
If you're using async, you can write:
var tasks = targetMachines.Select(x => Magic(x));
await Task.WhenAll(tasks); //fork-join
And instead of
var tasks = targetMachines.Select(x => Magic(x));
you can always write:
var tasks = targetMachines.Select(x =>
{
var result = Magic(x);
if (result == 2) DoSomething();
});
which liberates you from having to use ContinueWith.
I'm using reactive extensions to collate data into buffers of 100ms:
this.subscription = this.dataService
.Where(x => !string.Equals("FOO", x.Key.Source))
.Buffer(TimeSpan.FromMilliseconds(100))
.ObserveOn(this.dispatcherService)
.Where(x => x.Count != 0)
.Subscribe(this.OnBufferReceived);
This works fine. However, I want slightly different behavior than that provided by the Buffer operation. Essentially, I want to reset the timer if another data item is received. Only when no data has been received for the entire 100ms do I want to handle it. This opens up the possibility of never handling the data, so I should also be able to specify a maximum count. I would imagine something along the lines of:
.SlidingBuffer(TimeSpan.FromMilliseconds(100), 10000)
I've had a look around and haven't been able to find anything like this in Rx? Can anyone confirm/deny this?
This is possible by combining the built-in Window and Throttle methods of Observable. First, let's solve the simpler problem where we ignore the maximum count condition:
public static IObservable<IList<T>> BufferUntilInactive<T>(this IObservable<T> stream, TimeSpan delay)
{
var closes = stream.Throttle(delay);
return stream.Window(() => closes).SelectMany(window => window.ToList());
}
The powerful Window method did the heavy lifting. Now it's easy enough to see how to add a maximum count:
public static IObservable<IList<T>> BufferUntilInactive<T>(this IObservable<T> stream, TimeSpan delay, Int32? max=null)
{
var closes = stream.Throttle(delay);
if (max != null)
{
var overflows = stream.Where((x,index) => index+1>=max);
closes = closes.Merge(overflows);
}
return stream.Window(() => closes).SelectMany(window => window.ToList());
}
I'll write a post explaining this on my blog. https://gist.github.com/2244036
Documentation for the Window method:
http://leecampbell.blogspot.co.uk/2011/03/rx-part-9join-window-buffer-and-group.html
http://enumeratethis.com/2011/07/26/financial-charts-reactive-extensions/
I wrote an extension to do most of what you're after - BufferWithInactivity.
Here it is:
public static IObservable<IEnumerable<T>> BufferWithInactivity<T>(
this IObservable<T> source,
TimeSpan inactivity,
int maximumBufferSize)
{
return Observable.Create<IEnumerable<T>>(o =>
{
var gate = new object();
var buffer = new List<T>();
var mutable = new SerialDisposable();
var subscription = (IDisposable)null;
var scheduler = Scheduler.ThreadPool;
Action dump = () =>
{
var bts = buffer.ToArray();
buffer = new List<T>();
if (o != null)
{
o.OnNext(bts);
}
};
Action dispose = () =>
{
if (subscription != null)
{
subscription.Dispose();
}
mutable.Dispose();
};
Action<Action<IObserver<IEnumerable<T>>>> onErrorOrCompleted =
onAction =>
{
lock (gate)
{
dispose();
dump();
if (o != null)
{
onAction(o);
}
}
};
Action<Exception> onError = ex =>
onErrorOrCompleted(x => x.OnError(ex));
Action onCompleted = () => onErrorOrCompleted(x => x.OnCompleted());
Action<T> onNext = t =>
{
lock (gate)
{
buffer.Add(t);
if (buffer.Count == maximumBufferSize)
{
dump();
mutable.Disposable = Disposable.Empty;
}
else
{
mutable.Disposable = scheduler.Schedule(inactivity, () =>
{
lock (gate)
{
dump();
}
});
}
}
};
subscription =
source
.ObserveOn(scheduler)
.Subscribe(onNext, onError, onCompleted);
return () =>
{
lock (gate)
{
o = null;
dispose();
}
};
});
}
With Rx Extensions 2.0, your can answer both requirements with a new Buffer overload accepting a timeout and a size:
this.subscription = this.dataService
.Where(x => !string.Equals("FOO", x.Key.Source))
.Buffer(TimeSpan.FromMilliseconds(100), 1)
.ObserveOn(this.dispatcherService)
.Where(x => x.Count != 0)
.Subscribe(this.OnBufferReceived);
See https://msdn.microsoft.com/en-us/library/hh229200(v=vs.103).aspx for the documentation.
I guess this can be implemented on top of Buffer method as shown below:
public static IObservable<IList<T>> SlidingBuffer<T>(this IObservable<T> obs, TimeSpan span, int max)
{
return Observable.CreateWithDisposable<IList<T>>(cl =>
{
var acc = new List<T>();
return obs.Buffer(span)
.Subscribe(next =>
{
if (next.Count == 0) //no activity in time span
{
cl.OnNext(acc);
acc.Clear();
}
else
{
acc.AddRange(next);
if (acc.Count >= max) //max items collected
{
cl.OnNext(acc);
acc.Clear();
}
}
}, err => cl.OnError(err), () => { cl.OnNext(acc); cl.OnCompleted(); });
});
}
NOTE: I haven't tested it, but I hope it gives you the idea.
Colonel Panic's solution is almost perfect. The only thing that is missing is a Publish component, in order to make the solution work with cold sequences too.
/// <summary>
/// Projects each element of an observable sequence into a buffer that's sent out
/// when either a given inactivity timespan has elapsed, or it's full,
/// using the specified scheduler to run timers.
/// </summary>
public static IObservable<IList<T>> BufferUntilInactive<T>(
this IObservable<T> source, TimeSpan dueTime, int maxCount,
IScheduler scheduler = default)
{
if (maxCount < 1) throw new ArgumentOutOfRangeException(nameof(maxCount));
scheduler ??= Scheduler.Default;
return source.Publish(published =>
{
var combinedBoundaries = Observable.Merge
(
published.Throttle(dueTime, scheduler),
published.Skip(maxCount - 1)
);
return published
.Window(() => combinedBoundaries)
.SelectMany(window => window.ToList());
});
}
Beyond adding the Publish, I've also replaced the original .Where((_, index) => index + 1 >= maxCount) with the equivalent but shorter .Skip(maxCount - 1). For completeness there is also an IScheduler parameter, which configures the scheduler where the timer is run.