Develop Reference

IAsyncEnumerable like a Source for akka streams

I want to use IAsyncEnumerable like a Source for akka streams. But I not found, how do it.
No sutiable method in Source class for this code.
using System.Collections.Generic;
using System.Threading.Tasks;
using Akka.Streams.Dsl;
namespace ConsoleApp1
{
class Program
{
static async Task Main(string[] args)
{
Source.From(await AsyncEnumerable())
.Via(/*some action*/)
//.....
}
private static async IAsyncEnumerable<int> AsyncEnumerable()
{
//some async enumerable
}
}
}
How use IAsyncEnumerbale for Source?

This has been done in the past as a part of Akka.NET Streams contrib package, but since I don't see it there anymore, let's go through on how to implement such source. The topic can be quite long, as:
Akka.NET Streams is really about graph processing - we're talking about many-inputs/many-outputs configurations (in Akka.NET they're called inlets and outlets) with support for cycles in graphs.
Akka.NET is not build on top of .NET async/await or even on top of .NET standard thread pool library - they're both pluggable, which means that the lowest barier is basically using callbacks and encoding what C# compiler sometimes does for us.
Akka.NET streams is capable of both pushing and pulling values between stages/operators. IAsyncEnumerable<T> can only pull data while IObservable<T> can only push it, so we get more expressive power here, but this comes at a cost.
The basics of low level API used to implement custom stages can be found in the docs.
The starter boilerplate looks like this:
public static class AsyncEnumerableExtensions {
// Helper method to change IAsyncEnumerable into Akka.NET Source.
public static Source<T, NotUsed> AsSource<T>(this IAsyncEnumerable<T> source) =>
Source.FromGraph(new AsyncEnumerableSource<T>(source));
}
// Source stage is description of a part of the graph that doesn't consume
// any data, only produce it using a single output channel.
public sealed class AsyncEnumerableSource<T> : GraphStage<SourceShape<T>>
{
private readonly IAsyncEnumerable<T> _enumerable;
public AsyncEnumerableSource(IAsyncEnumerable<T> enumerable)
{
_enumerable = enumerable;
Outlet = new Outlet<T>("asyncenumerable.out");
Shape = new SourceShape<T>(Outlet);
}
public Outlet<T> Outlet { get; }
public override SourceShape<T> Shape { get; }
/// Logic if to a graph stage, what enumerator is to enumerable.
protected override GraphStageLogic CreateLogic(Attributes inheritedAttributes) =>
new Logic(this);
sealed class Logic: OutGraphStageLogic
{
public override void OnPull()
{
// method called whenever a consumer asks for new data
}
public override void OnDownstreamFinish()
{
// method called whenever a consumer stage finishes,used for disposals
}
}
}
As mentioned, we don't use async/await straight away here: even more, calling Logic methods in asynchronous context is unsafe. To make it safe we need to register out methods that may be called from other threads using GetAsyncCallback<T> and call them via returned wrappers. This will ensure, that not data races will happen when executing asynchronous code.
sealed class Logic : OutGraphStageLogic
{
private readonly Outlet<T> _outlet;
// enumerator we'll call for MoveNextAsync, and eventually dispose
private readonly IAsyncEnumerator<T> _enumerator;
// callback called whenever _enumerator.MoveNextAsync completes asynchronously
private readonly Action<Task<bool>> _onMoveNext;
// callback called whenever _enumerator.DisposeAsync completes asynchronously
private readonly Action<Task> _onDisposed;
// cache used for errors thrown by _enumerator.MoveNextAsync, that
// should be rethrown after _enumerator.DisposeAsync
private Exception? _failReason = null;
public Logic(AsyncEnumerableSource<T> source) : base(source.Shape)
{
_outlet = source.Outlet;
_enumerator = source._enumerable.GetAsyncEnumerator();
_onMoveNext = GetAsyncCallback<Task<bool>>(OnMoveNext);
_onDisposed = GetAsyncCallback<Task>(OnDisposed);
}
// ... other methods
}
The last part to do are methods overriden on `Logic:
OnPull used whenever the downstream stage calls for new data. Here we need to call for next element of async enumerator sequence.
OnDownstreamFinish called whenever the downstream stage has finished and will not ask for any new data. It's the place for us to dispose our enumerator.
Thing is these methods are not async/await, while their enumerator's equivalent are. What we basically need to do there is to:
Call corresponding async methods of underlying enumerator (OnPull → MoveNextAsync and OnDownstreamFinish → DisposeAsync).
See, if we can take their results immediately - it's important part that usually is done for us as part of C# compiler in async/await calls.
If not, and we need to wait for the results - call ContinueWith to register our callback wrappers to be called once async methods are done.
sealed class Logic : OutGraphStageLogic
{
// ... constructor and fields
public override void OnPull()
{
var hasNext = _enumerator.MoveNextAsync();
if (hasNext.IsCompletedSuccessfully)
{
// first try short-path: values is returned immediately
if (hasNext.Result)
// check if there was next value and push it downstream
Push(_outlet, _enumerator.Current);
else
// if there was none, we reached end of async enumerable
// and we can dispose it
DisposeAndComplete();
}
else
// we need to wait for the result
hasNext.AsTask().ContinueWith(_onMoveNext);
}
// This method is called when another stage downstream has been completed
public override void OnDownstreamFinish() =>
// dispose enumerator on downstream finish
DisposeAndComplete();
private void DisposeAndComplete()
{
var disposed = _enumerator.DisposeAsync();
if (disposed.IsCompletedSuccessfully)
{
// enumerator disposal completed immediately
if (_failReason is not null)
// if we close this stream in result of error in MoveNextAsync,
// fail the stage
FailStage(_failReason);
else
// we can close the stage with no issues
CompleteStage();
}
else
// we need to await for enumerator to be disposed
disposed.AsTask().ContinueWith(_onDisposed);
}
private void OnMoveNext(Task<bool> task)
{
// since this is callback, it will always be completed, we just need
// to check for exceptions
if (task.IsCompletedSuccessfully)
{
if (task.Result)
// if task returns true, it means we read a value
Push(_outlet, _enumerator.Current);
else
// otherwise there are no more values to read and we can close the source
DisposeAndComplete();
}
else
{
// task either failed or has been cancelled
_failReason = task.Exception as Exception ?? new TaskCanceledException(task);
FailStage(_failReason);
}
}
private void OnDisposed(Task task)
{
if (task.IsCompletedSuccessfully) CompleteStage();
else {
var reason = task.Exception as Exception
?? _failReason
?? new TaskCanceledException(task);
FailStage(reason);
}
}
}

As of Akka.NET v1.4.30 this is now natively supported inside Akka.Streams via the RunAsAsyncEnumerable method:
var input = Enumerable.Range(1, 6).ToList();
var cts = new CancellationTokenSource();
var token = cts.Token;
var asyncEnumerable = Source.From(input).RunAsAsyncEnumerable(Materializer);
var output = input.ToArray();
bool caught = false;
try
{
await foreach (var a in asyncEnumerable.WithCancellation(token))
{
cts.Cancel();
}
}
catch (OperationCanceledException e)
{
caught = true;
}
caught.ShouldBeTrue();
I copied that sample from the Akka.NET test suite, in case you're wondering.

You can also use an existing primitive for streaming large collections of data. Here is an example of using Source.unfoldAsync to stream pages of data - in this case github repositories using Octokit - until there is no more.
var source = Source.UnfoldAsync<int, RepositoryPage>(startPage, page =>
{
var pageTask = client.GetRepositoriesAsync(page, pageSize);
var next = pageTask.ContinueWith(task =>
{
var page = task.Result;
if (page.PageNumber * pageSize > page.Total) return Option<(int, RepositoryPage)>.None;
else return new Option<(int, RepositoryPage)>((page.PageNumber + 1, page));
});
return next;
});
To run
using var sys = ActorSystem.Create("system");
using var mat = sys.Materializer();
int startPage = 1;
int pageSize = 50;
var client = new GitHubClient(new ProductHeaderValue("github-search-app"));
var source = ...
var sink = Sink.ForEach<RepositoryPage>(Console.WriteLine);
var result = source.RunWith(sink, mat);
await result.ContinueWith(_ => sys.Terminate());
class Page<T>
{
public Page(IReadOnlyList<T> contents, int page, long total)
{
Contents = contents;
PageNumber = page;
Total = total;
}
public IReadOnlyList<T> Contents { get; set; } = new List<T>();
public int PageNumber { get; set; }
public long Total { get; set; }
}
class RepositoryPage : Page<Repository>
{
public RepositoryPage(IReadOnlyList<Repository> contents, int page, long total)
: base(contents, page, total)
{
}
public override string ToString() =>
$"Page {PageNumber}\n{string.Join("", Contents.Select(x => x.Name + "\n"))}";
}
static class GitHubClientExtensions
{
public static async Task<RepositoryPage> GetRepositoriesAsync(this GitHubClient client, int page, int size)
{
// specify a search term here
var request = new SearchRepositoriesRequest("bootstrap")
{
Page = page,
PerPage = size
};
var result = await client.Search.SearchRepo(request);
return new RepositoryPage(result.Items, page, result.TotalCount);
}
}

I can not handle these classes c# uwp

until now I only have small applications, but now I can not handle these classes c# uwp
I want to break the code into several classes, each class will deal with a few parameters. classes will send parameter values to the main page. but when I try to display those parameters, they are always zero, even though the text file appears modified.
i have main page
namespace airflow
{
public sealed partial class MainPage : Page
{
}
public MainPage()
{
}
private async void main_page_Loaded(object sender, RoutedEventArgs e)
{
param_perimetrala read = new param_perimetrala();
ora_start_perimetrala = read.start_perimetrala;
var mesaj = new MessageDialog(ora_start_perimetrala.ToString());
var res = await mesaj.ShowAsync();
}
}
and a class
namespace airflow
{
public class param_perimetrala
{
public static int ora_start_perimetrala;
public int minut_start_perimetrala;
public int ora_stop_perimetrala;
public int minut_stop_perimetrala;
public int ore_ciclu_perimetrala;
public int minut_ciclu_perimetrala;
public int contor_ore_perimetrala = 0;
public int contor_minute_perimetrala = 0;
public int contor_sec_perimetrala = 0;
public async void readfile_perimetrala()
{
StorageFolder folder = ApplicationData.Current.LocalFolder;
StorageFile perimetrala_file = await folder.CreateFileAsync("parametrii_perimetrala.txt", CreationCollisionOption.OpenIfExists);
var readFile_perimetrala = await FileIO.ReadLinesAsync(perimetrala_file);
int count = 0;
foreach (var line in readFile_perimetrala)
{
string[] split_perimetrala = line.Split(new string[] { "=" }, StringSplitOptions.RemoveEmptyEntries);
var temp = split_perimetrala[1];
if (count == 0)
{
ora_start_perimetrala = Int32.Parse(temp);
}
if (count == 1)
{
minut_start_perimetrala = Int32.Parse(temp);
}
if (count == 2)
{
ora_stop_perimetrala = Int32.Parse(temp);
}
count = count + 1;
}
}
public int start_perimetrala
{
get { return ora_start_perimetrala; }
set { ora_start_perimetrala = value; }
}
}
}
how to send ora_start_perimetrala value in mainpage?
enter image description here

In your main_page_Loaded event handler method you are calling read_file_perimetrala but read_file_perimetrala is async void. This means as soon as you get to an await statement, and the await is actually awaitable (not returned immediately) then the method will put the rest of the work aside until the awaited portion is complete. Then it picks back up and runs the rest. During that time of placing it aside the calling thread then continues to run wherever it's needed.
You need to make this a Task and await the call to is also so that you can insure the parameters are filled before continuing work.
Change your readFile_perimetrala to be like so:
public async Task readfile_perimetralaAsync()
Change the entire main_page_Loaded event handler to read like so...
private async void main_page_Loaded(object sender)
{
param_perimetrala read = new param_perimetrala();
await read.readfile_perimetralaAsync();
var mesaj = new MessageDialog(read.start_perimetrala.ToString());
var res = await mesaj.ShowAsync();
}
In your code you're assigning the value to a method but I can tell that you want the int value.
Just some light weight examples to help get started understanding Task and async await. Don't use this as a resource but just enough to get you curious to dig deeper because it's a simple but rather deep subject.
public class TaskExamples
{
public async void DoAsyncVoid()
{
await Task.Delay(200);
}
public async Task DoAsyncTask()
{
await Task.Delay(200);
}
public async Task<int> DoReturnValueTask()
{
await Task.Delay(200);
return 50;
}
public async void CallingTasks()
{
DoAsyncVoid(); //This can't use await because it is 'void' so the next line is ran as soon as this command reaches the first 'true awaitable' await.
await DoAsyncTask(); //This runs before DoAsyncVoid is complete.
var value = await DoReturnValueTask(); //This waits until 'DoAsyncTask' is complete because it is a Task and awaited.
await new MessageDialog(value.ToString()).ShowAsync(); //This waits until 'DoReturnValueTask' is complete and value will be 50 in this case.
//All code here waits until the Dialog is closed because it is also awaited.
}
}

How to implement a continuous producer-consumer pattern inside a Windows Service

Here's what I'm trying to do:
Keep a queue in memory of items that need processed (i.e. IsProcessed = 0)
Every 5 seconds, get unprocessed items from the db, and if they're not already in the queue, add them
Continuous pull items from the queue, process them, and each time an item is processed, update it in the db (IsProcessed = 1)
Do this all "as parallel as possible"
I have a constructor for my service like
public MyService()
{
Ticker.Elapsed += FillQueue;
}
and I start that timer when the service starts like
protected override void OnStart(string[] args)
{
Ticker.Enabled = true;
Task.Run(() => { ConsumeWork(); });
}
and my FillQueue is like
private static async void FillQueue(object source, ElapsedEventArgs e)
{
var items = GetUnprocessedItemsFromDb();
foreach(var item in items)
{
if(!Work.Contains(item))
{
Work.Enqueue(item);
}
}
}
and my ConsumeWork is like
private static void ConsumeWork()
{
while(true)
{
if(Work.Count > 0)
{
var item = Work.Peek();
Process(item);
Work.Dequeue();
}
else
{
Thread.Sleep(500);
}
}
}
However this is probably a naive implementation and I'm wondering whether .NET has any type of class that is exactly what I need for this type of situation.

Though #JSteward' answer is a good start, you can improve it with mixing up the TPL-Dataflow and Rx.NET extensions, as a dataflow block may easily become an observer for your data, and with Rx Timer it will be much less effort for you (Rx.Timer explanation).
We can adjust MSDN article for your needs, like this:
private const int EventIntervalInSeconds = 5;
private const int DueIntervalInSeconds = 60;
var source =
// sequence of Int64 numbers, starting from 0
// https://msdn.microsoft.com/en-us/library/hh229435.aspx
Observable.Timer(
// fire first event after 1 minute waiting
TimeSpan.FromSeconds(DueIntervalInSeconds),
// fire all next events each 5 seconds
TimeSpan.FromSeconds(EventIntervalInSeconds))
// each number will have a timestamp
.Timestamp()
// each time we select some items to process
.SelectMany(GetItemsFromDB)
// filter already added
.Where(i => !_processedItemIds.Contains(i.Id));
var action = new ActionBlock<Item>(ProcessItem, new ExecutionDataflowBlockOptions
{
// we can start as many item processing as processor count
MaxDegreeOfParallelism = Environment.ProcessorCount,
});
IDisposable subscription = source.Subscribe(action.AsObserver());
Also, your check for item being already processed isn't quite accurate, as there is a possibility to item get selected as unprocessed from db right at the time you've finished it's processing, yet didn't update it in database. In this case item will be removed from Queue<T>, and after that added there again by producer, this is why I've added the ConcurrentBag<T> to this solution (HashSet<T> isn't thread-safe):
private static async Task ProcessItem(Item item)
{
if (_processedItemIds.Contains(item.Id))
{
return;
}
_processedItemIds.Add(item.Id);
// actual work here
// save item as processed in database
// we need to wait to ensure item not to appear in queue again
await Task.Delay(TimeSpan.FromSeconds(EventIntervalInSeconds * 2));
// clear the processed cache to reduce memory usage
_processedItemIds.Remove(item.Id);
}
public class Item
{
public Guid Id { get; set; }
}
// temporary cache for items in process
private static ConcurrentBag<Guid> _processedItemIds = new ConcurrentBag<Guid>();
private static IEnumerable<Item> GetItemsFromDB(Timestamped<long> time)
{
// log event timing
Console.WriteLine($"Event # {time.Value} at {time.Timestamp}");
// return items from DB
return new[] { new Item { Id = Guid.NewGuid() } };
}
You can implement cache clean up in other way, for example, start a "GC" timer, which will remove processed items from cache on regular basis.
To stop events and processing items you should Dispose the subscription and, maybe, Complete the ActionBlock:
subscription.Dispose();
action.Complete();
You can find more information about Rx.Net in their guidelines on github.

You could use an ActionBlock to do your processing, it has a built in queue that you can post work to. You can read up on tpl-dataflow here: Intro to TPL-Dataflow also Introduction to Dataflow, Part 1. Finally, this is a quick sample to get you going. I've left out a lot but it should at least get you started.
using System;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Tasks.Dataflow;
namespace MyWorkProcessor {
public class WorkProcessor {
public WorkProcessor() {
Processor = CreatePipeline();
}
public async Task StartProcessing() {
try {
await Task.Run(() => GetWorkFromDatabase());
} catch (OperationCanceledException) {
//handle cancel
}
}
private CancellationTokenSource cts {
get;
set;
}
private ITargetBlock<WorkItem> Processor {
get;
}
private TimeSpan DatabasePollingFrequency {
get;
} = TimeSpan.FromSeconds(5);
private ITargetBlock<WorkItem> CreatePipeline() {
var options = new ExecutionDataflowBlockOptions() {
BoundedCapacity = 100,
CancellationToken = cts.Token
};
return new ActionBlock<WorkItem>(item => ProcessWork(item), options);
}
private async Task GetWorkFromDatabase() {
while (!cts.IsCancellationRequested) {
var work = await GetWork();
await Processor.SendAsync(work);
await Task.Delay(DatabasePollingFrequency);
}
}
private async Task<WorkItem> GetWork() {
return await Context.GetWork();
}
private void ProcessWork(WorkItem item) {
//do processing
}
}
}

how to fire a list of task in parallel c#

What would be he correct way to fire a list of task in parallel in a fire and forget manner.
What I've got below makes me believe that .WhenAll is blocking until all is done.
I've got quit a few like these, I need to learn how to loop and store all the called functions and then fire them off where the all run at the same time, it does not matter what function gets called first or last.
What is the correct approach for this?
I wish MS would put in their intellisense a little bit more info to help us out, because I've got more needs for async calls especially a lot of work calls at one time and they're all fire and forget.
Here is what I've got now.
public async static Task UpdateBayPositionAsync(string cadCNN, string bayPositions)
{
List<Task> myTask = new List<Task>();
string[] bps = bayPositions.Split(',');
int bID; byte pos;
for (int i = 0; i < bps.Length; i++)
{
bID = int.Parse(bps[i].Split(':')[0].ToString());
pos = byte.Parse(bps[i].Split(':')[1].ToString());
myTask.Add(Task.Run(() => { ElevationManagerDL.UpdateBayPosition(cadCNN, bID, pos); }));
};
await Task.WhenAll(myTask.ToList());
}

It looks like you are interested in both asynchronicity and parallelism
I would recommend solving the asynchronicity by a Task (not awaited) and the parallelism with Parallel.ForEach(..)
Parallel.ForEach is a lot more performant than to create one task per position,
especially if there are many positions, see Parallel.ForEach vs Task.Factory.StartNew
Something like this
public async static Task UpdateBayPositionAsync(string cadCnn, string serializedBayPositions)
{
string[] bayPositionsAsStrings = serializedBayPositions.Split(',');
List<BayPosition> bayPositions = bayPositionsAsStrings.Select(bp => new BayPosition(cadCnn, bp)).ToList();
Task.Factory.StartNew( () => Parallel.ForEach(bayPositions, item => item.Update()));
}
public class BayPosition
{
public int BId { get; private set; }
public byte Pos { get; private set; }
public string CadCnn { get; private set; }
public BayPosition(string cadCnn, string bayPosition)
{
string[] parameters = bayPosition.Split(':');
BId = Int32.Parse(parameters[0]);
Pos = Byte.Parse(parameters[1]);
CadCnn = cadCnn;
}
public void Update()
{
ElevationManagerDL.UpdateBayPosition(CadCnn, BId, Pos);
}
}
And if you only want the parallelism and want to block until all Updates are run then you just replace:
Task.Factory.StartNew( () => Parallel.ForEach(bayPositions, item => item.Update()));
with
Parallel.ForEach(bayPositions, item => item.Update());

Get object from Task and return from non-async method

I've got a lower level library that one of the methods I've designed to be async and to return a Task.
I know it is not a good idea to have async methods in lower level libraries, but there is no way around this one, because of how heave the processing is.
In between my lower level library I've got a main library that exposes the lower level library method in question. "Just like a business layer".
The entry point of my app is a WEB Api it then calls the mid level library and the the mid level calls the lowest level library.
My relevant question is would it be better to unpack the result from the lowest level library where the async chain begin inside my mid library and then expose just a static method to the consumer, or should I just offer the consumer a async method and just offer async methods from the lowest to the highest?
From the entry into the app the first method in question is: CutSheet.CutSheet cutSheet = AlumCloudPlans.Manager.GetCutSheet(elev);
I know there must be a better way because on the mid level library I've got to create an entire new task and the return is value, which is a Task, and the return that Task value.
What would ya'll recommend that I do in the mid level library?
---Lowest level
namespace CutSheet
{
public class Manager
{
public async static Task<CutSheet> GetCutSheet(IElevation elevation)
{
return new CutSheet(await new PartsProcessor.PartProcessor()
.ProcessParts(elevation));
}
}
}
---Mid level
namespace AlumCloudPlans
{
public class Manager
{
public static CutSheet.CutSheet GetCutSheet(IElevation elevation)
{
var t = Task.Factory.StartNew(async delegate
{
return await CutSheet.Manager.GetCutSheet(elevation);
});
t.Wait();
return t.Result.Result;
}
}
}
---Highest level and entry to app
namespace StorefrontSystem.Controllers
{
[Authorize]
public class CutSheetController : AlumCloudWebApiBaseController
{
public async Task<HttpResponseMessage> Get(string a, string f, int id)
{
HttpResponseMessage r = Request.CreateResponse();
IElevation elev = await ElevationManager.GetElevation(id);
CutSheet.CutSheet cutSheet = AlumCloudPlans.Manager
.GetCutSheet(elev);
var ms = new MemoryStream();
bool isPDF = f.ToLower() == "pdf";
if (isPDF)
{
using (var pdf = await cutSheet.GetCutSheetPDF(elev))
{
pdf.Save(ms, false);
}
}
else
{
using (Bitmap canvas = await cutSheet.GetDrawing())
{
canvas.Save(ms, System.Drawing.Imaging.ImageFormat.Png);
}
}
ms.Position = 0;
r.Content = new StreamContent(ms);
if (isPDF)
{
r.Content.Headers.ContentType
= new Headers.MediaTypeHeaderValue("Application/pdf");
}
else
{
r.Content.Headers.ContentType
= new Headers.MediaTypeHeaderValue("image/png");
}
return r;
}
}
}

I know it is not a good idea to have async methods in lower level libraries
Who told you that? It's a perfectly good idea to have async methods at any level, as long as the operation you're doing is naturally asynchronous. However, it's not a good idea in the lower levels to pretend something is asynchronous by using Task.Run - and this is particularly true in ASP.NET.
Assuming that your operation is naturally asynchronous, you should use async all the way up.
Mid-level:
public class Manager
{
public static Task<CutSheet.CutSheet> GetCutSheetAsync(IElevation elevation)
{
return CutSheet.Manager.GetCutSheetAsync(elevation);
}
}
Highest-level:
[Authorize]
public class CutSheetController : AlumCloudWebApiBaseController
{
public async Task<HttpResponseMessage> Get(string a, string f, int id)
{
HttpResponseMessage r = Request.CreateResponse();
IElevation elev = await ElevationManager.GetElevationAsync(id);
CutSheet.CutSheet cutSheet = await AlumCloudPlans.Manager.GetCutSheetAsync(elev);
...
}
}