Develop Reference

porting a dequeue - add loop from C# to F#

I am trying to port this simple loop:
var messages = new List<string>();
while (_MessageQueue.TryDequeue(out var message))
messages.Add(message);
where the message queue is concurrent. It is used in a module that enqueues messages from several threads and they are then processed by a single thread.
Is there an idiomatic F# way to do the dequeue / add loop?

F# has a couple of ways of doing concurrency including a nice support for agent-based programming, so it is quite possible that the idiomatic F# version of what you are doing would not actually use conrrent queues, but would be instead based on agents or some other architecture.
However, to answer your specific question about looping - the C# version is quite terse thanks to the clever use of while and out var. In F#, you can call TryDequeue as a method that returns bool together with the value (so we can avoid mutation). I would use that, together with a recursive sequence expression:
let mq = System.Collections.Concurrent.ConcurrentQueue<int>()
let rec readAll () = seq {
let succ, msg = mq.TryDequeue()
if succ then
yield msg
yield! readAll() }
let messages = readAll() |> List.ofSeq
The readAll function defines a sequence (IEnumerable) that calls TryDequeue and if the operation succeeds, it adds the message to the result using yield msg and then recursively tries to read more messages by using yield!.

Here's a straight conversion:
open System.Collections.Concurrent
let _MessageQueue = ConcurrentQueue<string>()
let messages = ResizeArray<string>()
let mutable continueLooping = true
while continueLooping do
let success, message = _MessageQueue.TryDequeue()
if success then messages.Add(message)
continueLooping <- success

While this question already has an accepted answer, I wanted to contribute an alternative implementation using the library function Seq.unfold:
let getAllMessages (mq : _ ConcurrentQueue) =
mq |> Seq.unfold (fun q ->
match q.TryDequeue () with
| true, m -> Some (m, q)
| _ -> None)
let messages = getAllMessages _MessageQueue |> Seq.toList
Not sure if internally it is as complex as (or even more than) Tomas's solution but I find it is short, understandable, and elegant.

I'm providing a couple additional design options for giggles. The common parts:
open System.Collections.Concurrent
type Message = { I: int }
let queue = ConcurrentQueue<Message>()
drain1 invokes queue.GetEnumerator(), which has the condition that it's returning a snapshot at the time of the initial request. The snapshot is, substantially, the same race condition in the C# version.
let drain1 () = queue |> Seq.toList
drain2 returns an array instead, again a snapshot at the time of the initial request. In case you have the luxury of changing the return type.
let drain2 () = queue.ToArray()
This is an example of the idiomatic return from TryQueue, it avoids an 'out' arg, therefore it's not a mutable value as C# does/did it.
let example () =
let (success, message) = queue.TryDequeue()
() // ...
Finally, a recursively built, self-terminating sequence.
let drain3 () =
let rec drain () = seq {
let success, message = queue.TryDequeue()
if success then
yield message
yield! drain()
}
drain() |> Seq.toList
(Standard Internet warranty applies.)

Riffing on #Scott Hutchinson's answer
I'd define an efficient, straightforward helper that encapsulates the mutation and looping :-
[<AutoOpen>]
module ConcurrentQueueExtensions =
type System.Collections.Concurrent.ConcurrentQueue<'T> with
member this.Drain() =
let buffer = ResizeArray(this.Count)
let mutable more = true
while more do
match this.TryDequeue() with
| true, req -> buffer.Add req
| false, _ -> more <- false
buffer.ToArray()
or even leave that generic helper in C#:
class ConcurrentQueueExtensions
{
public static T[] Drain<T>(this System.Collections.Concurrent.ConcurrentQueue<T> that)
{
var buffer = new List<T>(that.Count);
while (that.TryDequeue(out var req))
buffer.Add(req);
return buffer.ToArray();
}
}
and then apply it with no mixing of paradigms:
let messages = queue.Drain()

Getting the latest item in an observable sequence using RX in C#

take the following as an example:
var ob = Observable.Interval(TimeSpan.FromSeconds(1)).StartWith(500).Replay(1).RefCount();
What I'm trying to achieve here is to obtain the value of the latest item in the sequence at any given time "synchronously". Which means extensions like FirstAsync can't make it up for me.
The StartWith and Replay bit ensures that there will always be a value, and the RefCount bit is necessary in my actual code to detect when I can do some disposal actions.
So to simulate this "any given time" part, let's try getting the latest value after 5 seconds:
Observable.Timer(TimeSpan.FromSeconds(5)).Subscribe(x =>
{
// Try to get latest value from "ob" here.
});
So with a 5 second delay, I need to get the value 5 out of the sequence and these are what I have tried so far with no success:
ob.First() - returns 500
ob.Latest().Take(1) - same as above
ob.MostRecent(-1).First() - same as above
ob.MostRecent(-1) - gives me an IEnumerable<long> full of "500"
ob.Last() - never returns because it's waiting for the sequence to complete which it never will
ob.Latest().Last() - same as above
ob.ToTask().Result - same as above
ob.ToEnumerable() - same as above
ob.MostRecent().Last() same as above
It seems there's not much resources around that people can actually do this. The closest I can find is this: "Rx: operator for getting first and most recent value from an Observable stream", but it is not a synchronous call after all (still using a subscription) so it doesn't work for me.
Does any body know if this is actually doable?

To point out why your code probably isn't working as you expect it to
var ob = Observable.Interval(TimeSpan.FromSeconds(1)).StartWith(500).Replay(1).RefCount();
//Note at this point `ob` has never been subscribed to,
// so the Reference-count is 0 i.e. has not be connected.
Observable.Timer(TimeSpan.FromSeconds(5)).Subscribe(x =>
{
// Try to get latest value from "ob" here.
//Here we make our first subscription to the `ob` sequence.
// This will connect the sequence (invoke subscribe)
// which will
// 1) invoke StartWith
// 2) invoke onNext(500)
// 3) invoke First()
// 4) First() will then unsubscribe() as it has the single value it needs
// 5) The refCount will now return to 0
// 6) The sequence will be unsubscribed to.
ob.First().Dump();
//Any future calls like `ob.First()` will thus always get the value 500.
});
Potentially what you want is
var ob = Observable.Interval(TimeSpan.FromSeconds(1))
.Publish(500);
var connection = ob.Connect();
//Note at this point `ob` has never been subscribed to, so the ReferenceCount is 0 i.e. has not be connected.
var subscription = Observable.Timer(TimeSpan.FromSeconds(5)).Subscribe(x =>
{
// Try to get latest value from "ob" here.
ob.First().Dump();
});
//Sometime later
subscription.Dispose();
connection.Dispose()
HOWEVER, You really don't want to be mixing Synchronous calls with Rx. You also generally don't want to be subscribing within a subscription (as .First() is a subscription). What you probably mean to be doing is getting the latest value, and stashing it somewhere. Using .First() is just a slippery slope. You probably would be better writing something like
var subscription = Observable.Timer(TimeSpan.FromSeconds(5))
.SelectMany(_=>ob.Take(1))
.Subscribe(x =>
{
//Do something with X here.
x.Dump();
});

You need to do something like this:
var ob = Observable.Interval(TimeSpan.FromSeconds(1)).StartWith(500);
var latestAndThenTheRest =
Observable
.Create<long>(o =>
{
var bs = new BehaviorSubject<long>(1);
var s1 = ob.Subscribe(bs);
var s2 = bs.Subscribe(o);
return new CompositeDisposable(s1, s2);
});
The only thing that you need to consider here is that ob must be a hot observable for this to even make sense. If it were cold then every subscriber would get a brand new subscription to the start of the ob sequence.

Just to clarify this a bit, and thanks for #LeeCampbell's answer.
What was not working:
var ob = Observable.Interval(TimeSpan.FromSeconds(1)).StartWith(500).Replay(1).RefCount();
Observable.Timer(TimeSpan.FromSeconds(5)).Subscribe(x =>
{
ob.First().Dump();
// This gives you 500.
// Because this is the first time any one subscribes to the observable,
// so it starts right here and gives you the initial value.
});
What would actually work:
var ob = Observable.Interval(TimeSpan.FromSeconds(1)).StartWith(500).Replay(1).RefCount();
ob.Subscribe(); // Subscribe to start the above hot observable immediately.
Observable.Timer(TimeSpan.FromSeconds(5)).Subscribe(x =>
{
ob.First().Dump();
// This would give you either 3 or 4, depending on the speed and timing of your computer.
});

I'm not sure if this answer helps you, but have you looked into BehaviorSubject? It's an IObservable that remembers its latest value. It's a bit like a combination of a regular variable and an observable in one.
Otherwise, why don't you subscribe to 'ob' and store the latest value in a variable yourself?

Learning Rx: How can I parse an observable sequence of characters into an observable sequence of strings?

This is probably really simple but I'm at the bottom of the learning curve with Rx. I've spent several hours reading articles, watching videos and writing code but I seem to have a mental block on something that seems like it should be really simple.
I'm gathering data from a serial port. I have used Observable.FromEventPattern to capture the SerialDataReceived event and convert it to an observable sequence of characters. So far so good.
Now, I want to parse that character sequence based on separator characters. There are no newlines involved, but each 'packet' of data is surrounded by a preamble and a terminator, both single characters. For the sake of argument, lets say they are braces { and }.
So if I get the character sequence
j u n k { H e l l o } j u n k on my character sequence,
then I want to emit either Hello or {Hello} on my string sequence.
I'm probably missing something simple but I can't even begin to figure out how to approach this. Any suggestions please?

This can be easily accomplished using Publish and Buffer:
var source = "junk{Hello}junk{World}junk".ToObservable();
var messages = source
.Publish(o =>
{
return o.Buffer(
o.Where(c => c == '{'),
_ => o.Where(c => c == '}'));
})
.Select(buffer => new string(buffer.ToArray()));
messages.Subscribe(x => Console.WriteLine(x));
Console.ReadLine();
The output of this is:
{Hello}
{World}
The idea is that you can use the following opening and closing selectors in the call to Buffer. The use of Publish is to make sure that all three of Buffer, the opening selector, and the closing selector share the same subscription.
source: junk{Hello}junk{World}junk|
opening: ----{----------{----------|
closing: ------}|
closing: ------}|

Use Scan to aggregate your so-far-received values into the aggregated string (TAccumulate is a string), and reset that string to "" every time you get an end brace. (I'll leave the work of implementing the aggregation function up to you). This will produce observables like
j
ju
jun
junk
junk{
junk{h
junk{hi
junk{hi}
j
ju
...
Then you can use Where to only emit the ones that end with }
Then finally use Select to get rid of the junk.
So in full, should be
IObservable<string> packetReceived =
serialPort.CharReceived
.Scan(YourAggregationFunction)
.Where(s => s.EndsWith("}"))
.Select(s => s.EverythingAfter("{"));
(I leave EverythingAfter up to you to implement as well).
Just a note, as you're experimenting with the aggregation function, it may be easier to use the IEnumerable interface of string to test it, i.e.
foreach (s in "junk{hi}hunk{ji}blah".Scan(YourAggregationFunction))
Console.WriteLine(s);
Okay, here's a full working example
static void Main(string[] args) {
var stuff = "junk{hi}junk{world}junk".ToObservable()
.Scan("", (agg, c) => agg.EndsWith("}") ? c.ToString() : agg + c)
.Where(s => s.EndsWith("}"))
.Select(s => s.Substring(s.IndexOf('{')));
foreach (var thing in stuff.ToEnumerable()) {
Console.WriteLine(thing);
}
}

How can a TPL Dataflow block downstream get data produced by a source?

I'm processing images using TPL Dataflow. I receive a processing request, read an image from a stream, apply several transformations, then write the resulting image to another stream:
Request -> Stream -> Image -> Image ... -> Stream
For that I use the blocks:
BufferBlock<Request>
TransformBlock<Request,Stream>
TransformBlock<Stream,Image>
TransformBlock<Image,Image>
TransformBlock<Image,Image>
...
writerBlock = new ActionBlock<Image>
The problem is the initial Request is what contains some data necessary to create the resulting Stream along with some additional info I need at that point. Do I have to pass the original Request (or some other context object) down the line to the writerBlock across all the other blocks like this:
TransformBlock<Request,Tuple<Request,Stream>>
TransformBlock<Tuple<Request,Stream>,Tuple<Request,Image>>
TransformBlock<Tuple<Request,Image>,Tuple<Request,Image>>
...
(which is ugly), or is there a way to link the first block to the last one (or, generalizing, to the ones that need the additional data)?

Yes, you pretty much need to do what you described, passing the additional data from every block to the next one.
But using a couple of helper methods, you can make this much simpler:
public static IPropagatorBlock<TInput, Tuple<TOutput, TInput>>
CreateExtendedSource<TInput, TOutput>(Func<TInput, TOutput> transform)
{
return new TransformBlock<TInput, Tuple<TOutput, TInput>>(
input => Tuple.Create(transform(input), input));
}
public static IPropagatorBlock<Tuple<TInput, TExtension>, Tuple<TOutput, TExtension>>
CreateExtendedTransform<TInput, TOutput, TExtension>(Func<TInput, TOutput> transform)
{
return new TransformBlock<Tuple<TInput, TExtension>, Tuple<TOutput, TExtension>>(
tuple => Tuple.Create(transform(tuple.Item1), tuple.Item2));
}
The signatures look daunting, but they are actually not that bad.
Also, you might want to add overloads that pass options to the created block, or overloads that take async delegates.
For example, if you wanted to perform some operations on a number using separate blocks, while passing the original number along the way, you could do something like:
var source = new BufferBlock<int>();
var divided = CreateExtendedSource<int, double>(i => i / 2.0);
var formatted = CreateExtendedTransform<double, string, int>(d => d.ToString("0.0"));
var writer = new ActionBlock<Tuple<string, int>>(tuple => Console.WriteLine(tuple));
source.LinkTo(divided);
divided.LinkTo(formatted);
formatted.LinkTo(writer);
for (int i = 0; i < 10; i++)
source.Post(i);
As you can see, your lambdas (except for the last one) deal only with the “current” value (int, double or string, depending on the stage of the pipeline), the “original” value (always int) is passed automatically. At any moment, you can use block created using the normal constructor to access both values (like the final ActionBlock in the example).
(That BufferBlock isn't actually necessary, but I added it to more closely match your design.)

I may be going over my head since I am only starting to play with TPL Dataflow.
But I believe you can accomplish that using a BroadcastBlock as an intermediary between your source and your first target.
BroadcastBlock can offer the message to many targets, so you use it to offer to your target, and also to a JoinBlock, at the end that will merge the result with the original message.
source -> Broadcast ->-----------------------------------------> JoinBlock <source, result>
-> Transformation1 -> Transformation 'n' ->
For example:
var source = new BufferBlock<int>();
var transformation = new TransformBlock<int, int>(i => i * 100);
var broadCast = new BroadcastBlock<int>(null);
source.LinkTo(broadCast);
broadCast.LinkTo(transformation);
var jb = new JoinBlock<int, int>();
broadCast.LinkTo(jb.Target1);
transformation.LinkTo(jb.Target2);
jb.LinkTo(new ActionBlock<Tuple<int, int>>(
c => Console.WriteLine("Source:{0}, Target Result: {1}", c.Item1, c.Item2)));
source.Post(1);
source.Post(2);
source.Complete();
yields...
Source:1, Target Result: 100
Source:2, Target Result: 200
I am just not too sure about how it would behave in an asynchronous environment.

Reactive: Trying to understand how Subject<T> work

Trying to understand how the Subject<T>, ReplaySubject<T> and other work. Here is example:
(Subject is Observable and observer)
public IObservable<int> CreateObservable()
{
Subject<int> subj = new Subject<int>(); // case 1
ReplaySubject<int> subj = new ReplaySubject<int>(); // case 2
Random rnd = new Random();
int maxValue = rnd.Next(20);
Trace.TraceInformation("Max value is: " + maxValue.ToString());
subj.OnNext(-1); // specific value
for(int iCounter = 0; iCounter < maxValue; iCounter++)
{
Trace.TraceInformation("Value: " + iCounter.ToString() + " is about to publish");
subj.OnNext(iCounter);
}
Trace.TraceInformation("Publish complete");
subj.OnComplete();
return subj;
}
public void Main()
{
//
// First subscription
CreateObservable()
.Subscribe(
onNext: (x)=>{
Trace.TraceInformation("X is: " + x.ToString());
});
//
// Second subscribe
CreateObservable()
.Subscribe(
onNext: (x2)=>{
Trace.TraceInformation("X2 is: " + x.ToString());
});
Case 1: The strange situation is - when I use Subject<T> no subscription is made (???) - I never see the "X is: " text - I only see the "Value is: " and "Max value is"... Why does Subject<T> does not push values to subscription ?
Case 2: If I use ReplaySubject<T> - I do see the values in Subscription but I could not apply Defer option to anything. Not to Subject and not to Observable.... So every subscription will receive different values because CreateObservable function is cold observable. Where is Defer ?

Whenever you need to create an observable out of thin air, Observable.Create should be the first thing to think of. Subjects enter the picture in two cases:
You need some kind of "addressable endpoint" to feed data to in order for all subscribers to receive it. Compare this to a .NET event which has both an invocation side (through delegate invocation) and a subscription side (through delegate combine with +- and -= syntax). You'll find in a lot of cases, you can achieve the same effect using Observable.Create.
You need multicasting of messages in a query pipeline, effectively sharing an observable sequence by many forks in your query logic, without triggering multiple subscriptions. (Think of subscribing to your favorite magazine once for your dorm and putting a photo copier right behind the letter box. You still pay one subscription, though all of your friends can read the magazine delivered through OnNext on the letter box.)
Also, in a lot of cases, there's already a built-in primitive in Rx that does exactly what you need. For example, there's From* factory methods to bridge with existing concepts (such as events, tasks, asynchronous methods, enumerable sequence), some of which using a subject under the covers. For the second case of multicasting logic, there's the Publish, Replay, etc. family of operators.

You need to be mindful of when code is executed.
In "Case 1", when you use a Subject<T>, you'll notice that the all of the calls to OnNext & OnCompleted finish before the observable is returned by the CreateObservable method. Since you are using a Subject<T> this means that any subsequent subscription will have missed all of the values so you should expect to get what you got - nothing.
You have to delay the operations on the subject until you have the observer subscribed. To do that using the Create method. Here's how:
public IObservable<int> CreateObservable()
{
return Observable.Create<int>(o =>
{
var subj = new Subject<int>();
var disposable = subj.Subscribe(o);
var rnd = new Random();
var maxValue = rnd.Next(20);
subj.OnNext(-1);
for(int iCounter = 0; iCounter < maxValue; iCounter++)
{
subj.OnNext(iCounter);
}
subj.OnCompleted();
return disposable;
});
}
I've removed all the trace code for succinctness.
So now, for every subscriber, you get a new execution of the code inside the Create method and you would now get the values from the internal Subject<T>.
The use of the Create method is generally the correct way to create observables that you return from methods.
Alternatively you could use a ReplaySubject<T> and avoid the use of the Create method. However this is unattractive for a number of reasons. It forces the computation of the entire sequence at creation time. This give you a cold observable that you could have produced more efficiently without using a replay subject.
Now, as an aside, you should try to avoid using subjects at all. The general rule is that if you're using a subject then you're doing something wrong. The CreateObservable method would be better written as this:
public IObservable<int> CreateObservable()
{
return Observable.Create<int>(o =>
{
var rnd = new Random();
var maxValue = rnd.Next(20);
return Observable.Range(-1, maxValue + 1).Subscribe(o);
});
}
No need for a subject at all.
Let me know if this helps.