Develop Reference

DataflowBlock ITargetSource.AsObservable() not triggering OnNext()

I'm trying to use a dataflowblock and I need to spy the items passing through for unit testing.
In order to do this, I'm using the AsObservable() method on ISourceBlock<T> of my TransformBlock<Tinput, T>,
so I can check after execution that each block of my pipeline have generated the expected values.
Pipeline
{
...
var observer = new MyObserver<string>();
_block = new TransformManyBlock<string, string>(MyHandler, options);
_block.LinkTo(_nextBlock);
_block.AsObservable().Subscribe(observer);
_block.Post("Test");
...
}
MyObserver
public class MyObserver<T> : IObserver<T>
{
public List<Exception> Errors = new List<Exception>();
public bool IsComplete = false;
public List<T> Values = new List<T>();
public void OnCompleted()
{
IsComplete = true;
}
public void OnNext(T value)
{
Values.Add(value);
}
public void OnError(Exception e)
{
Errors.Add(e);
}
}
So basically I subscribe my observer to the transformblock, and I expect that each value passing through get registered in my observer "values" list.
But, while the IsComplete is set to true, and the OnError() successfully register exception,
the OnNext() method never get called unless it is the last block of the pipeline...
I can't figure out why, because the "nextblock" linked to this sourceBlock successfully receive the data, proving that some data are exiting the block.
From what I understand, the AsObservable is supposed to report every values exiting the block and not only the values that have not been consumed by other linked blocks...
What am I doing wrong ?

Your messages are being consumed by _nextBlock before you get a chance to read them.
If you comment out this line _block.LinkTo(_nextBlock); it would likely work.
AsObservable sole purpose is just to allow a block to be consumed from RX. It doesn't change the internal working of the block to broadcast messages to multiple targets. You need a special block for that BroadcastBlock
I would suggest broadcasting to another block and using that to Subscribe
BroadcastBlock’s mission in life is to enable all targets linked from
the block to get a copy of every element published
var options = new DataflowLinkOptions {PropagateCompletion = true};
var broadcastBlock = new BroadcastBlock<string>(x => x);
var bufferBlock = new BufferBlock<string>();
var actionBlock = new ActionBlock<string>(s => Console.WriteLine("Action " + s));
broadcastBlock.LinkTo(bufferBlock, options);
broadcastBlock.LinkTo(actionBlock, options);
bufferBlock.AsObservable().Subscribe(s => Console.WriteLine("peek " + s));
for (var i = 0; i < 5; i++)
await broadcastBlock.SendAsync(i.ToString());
broadcastBlock.Complete();
await actionBlock.Completion;
Output
peek 0
Action 0
Action 1
Action 2
Action 3
Action 4
peek 1
peek 2
peek 3
peek 4

How do I save result in string of executed command

As the title says how can I save in string the result of the executed command?
SendCommand("server.hostname");
My code:
public void SendCommand(string command)
{
PacketModel packet = new PacketModel()
{
Identifier = 1,
Message = command,
Name = "RustManager"
};
string packetString = JsonConvert.SerializeObject(packet);
_webSocket.SendAsync(packetString, null);
}
public void GetServerHostname()
{
SendCommand("server.hostname");
}

Due to my small reputation I cannot comment - which is what I would have done before that.
Normally methods that end on Async are async and return a Task<T> type.
Using the await keyword makes your method async which is why you have to mark it as async in the method head.
Link to C#-Documentation on the await keyword
It is really hard to say how to get your code running since I don't have alot of information but maybe this helps:
public async void SendCommand(string command)
{
PacketModel packet = new PacketModel()
{
Identifier = 1,
Message = command,
Name = "RustManager"
};
string packetString = JsonConvert.SerializeObject(packet);
var result = await _webSocket.SendAsync(packetString, null);
}
EDIT 1:
After getting some new information here is my new answer:
You use this class for your websocket. If you look at the signiture of the "SendAsync" method you can see, that it returns void (which means "nothing"). So you will not be able to "Store some kind of information" here.
The method looks like this:
public void SendAsync (string data, Action<bool> completed)
{ [...] }
You will have to listen to the WebSocket and wait for a server-side response. It seems, that the library supports that via events:
ws.OnMessage += (sender, e) => {
...
};
So you can define an eventhandler to process the server-response.
If you would like to get the message data, you should access e.Data or e.RawData property.
e.Data property returns a string, so it is mainly used to get the text message data.
(source (GitHub Readme))
So to fullfil your wishes try the following:
1.) At initialization of your _websocket instance subscribe to the .OnMessageevent with a corresponding event handler. (Some information about that)
2.) Send your message as you do it now with SendAsync
3.) If your server responds with a message to the network socket, the OnMessageevent will fire and you will be able to get the Data from the Eventargument e
(I did not test this - but it should work since it is used this way in the examples)

"yield return" from event handler

I have a class which takes a stream in the constructor. You can then set up callbacks for various events, and then call StartProcessing. The issue is that I want to use it from a function which should return an IEnumerable.
Example:
public class Parser
{
public Parser(System.IO.Stream s) { // saves stream and does some set up }
public delegate void OnParsedHandler(List<string> token);
public event OnParsedHandler OnParsedData;
public void StartProcessing()
{
// reads stream and makes callback when it has a whole record
}
}
public class Application
{
public IEnumerable<Thing> GetThings(System.IO.Stream s)
{
Parser p = new Parser(s);
p.OnParsedData += (List<string> str) =>
{
Thing t = new Thing(str[0]);
// here is where I would like to yield
// but I can't
yield return t;
};
p.StartProcessing();
}
}
Right now my solution, which isn't so great, is to put them all the Things into a List which is captured by the lambda, and then iterate over them after calling StartProcessing.
public class Application
{
public IEnumerable<Thing> GetThings(System.IO.Stream s)
{
Parser p = new Parser(s);
List<Thing> thingList = new List<Thing>();
p.OnParsedData += (List<string> str) =>
{
Thing t = new Thing(str[0]);
thingList .Add(t);
};
p.StartProcessing();
foreach(Thing t in thingList )
{
yield return t;
}
}
}
The issue here is that now I have to save all of the Thing objects into list.

The problem you have here is that you don't fundamentally have a "pull" mechanic here, you're trying to push data from the parser. If the parser is going to push data to you, rather than letting the caller pull the data, then GetThings should return an IObservable, rather than an IEnumerable, so the caller can consume the data when it's ready.
If it really is important to have a pull mechanic here then Parser shouldn't fire an event to indicate that it has new data, but rather the caller should be able to ask it for new data and have it get it; it should either return all of the parsed data, or itself return an IEnumerable.

Interesting question. I would like to build upon what #servy has said regarding push and pull. In your implementation above, you are effectively adapting a push mechanism to a pull interface.
Now, first things first. You have not specified whether the call to the StartProcessing() method is a blocking call or not. A couple of remarks regarding that:
If the method is blocking (synchronous), then there is really no point in adapting it to a pull model anyway. The caller will see all the data processed in a single blocking call.
In that regard, receiving the data indirectly via an event handler scatters into two seemingly unrelated constructs what should otherwise be a single, cohesive, explicit operation. For example:
void ProcessAll(Action<Thing> callback);
On the other hand, if the StartProcessing() method actually spawns a new thread (maybe better named BeginProcessing() and follow the Event-based Asynchronous Pattern or another async processing pattern), you could adapt it to a pull machanism by means of a synchronization construct using a wait handle: ManualResetEvent, mutex and the like. Pseudo-code:
public IEnumerable<Thing> GetThings(System.IO.Stream s)
{
var parser = new Parser(s);
var waitable = new AutoResetEvent(false);
Thing item = null;
parser.OnParsedData += (Thing thing) =>
{
item = thing;
waitable.Set();
};
IAsyncResult result = parser.BeginProcessing();
while (!result.IsCompleted)
{
waitable.WaitOne();
yield return item;
}
}
Disclaimer
The above code serves only as a means for presenting an idea. It is not thread-safe and the synchronization mechanics do not work properly. See the producer-consumer pattern for more information.

How to Implement Progress Bar for Long Running HTTP Request

I have an HTTP server written in C# based off the HttpListenerContext class. The server is for processing binary log files and converting them to text, and can take quite a long time to do the conversion. I would like to indicate progress back to the user, but I am unsure on the best way to do this. On the server side, in handling my HTTP request, I essentially have this function:
public async Task HandleRequest()
{
try
{
await ProcessRequest();
}
catch (HttpListenerException)
{
// Something happened to the http connection, don't try to send anything
}
catch (Exception e)
{
SendFailureResponse(500);
}
}
Currently, ProcessRequest() sends the HTML response when finished, but I would like to essentially add the IProgress interface to the function and somehow indicate that progress back to the Web client. What is the best way to do this?

One way of doing it would be to store progress on server side and periodically pull the information from client.
However, if you want the server to notify the client ( push ), then you will need to implement some kind of bi-directional communication between the server and client (I am currently using ASP.NET Web API and SignalR to achieve this at work).

Here is what I got I'll try to explain and I hope you notice its not FULL FULL complete, you'll have to understand the logic behind this and accept or not as a plausible option.
The Method: Set a custom object to store progress of your ongoing operations, make a global static list containing this metadata. Notice how I track them with Ids: I don't store that on DB, the natural act of instantiating the class will auto_increment their Id.
Then, you can add a new controller to respond the progress of a particular ongoing process.
Now that you have a controller to respond the progress of an ongoing process by Id, you can create a javascript timer to call it and update the DOM.
When creating your process, dont hold the htmlrequest until its over, open a background operation instead and just respond with the newly created ProgressTracker.Id, through that class/list you can keep track of the progress and reply accordingly.
As said in another answer, when an operation finishes you can send a push notification and the clientside javascript will interrupt the timers and proceed to the next view/result/page, or you can increment the looping timer to detect when its done and call the results from another controller. (this way you can avoid using push if needed.)
Here is the partial code:
public class ProgressTracker {
private static GlobalIdProvider = 0;
public int _id = ++GlobalIdProvider;
public int Id { get { return _id; } }
bool IsInProgress = false;
bool IsComplete = false;
float Progress;
public YourProgressObject Data;
}
public class GlobalStatic {
public static List<ProgressTracker> FooOperations = new List<ProgressTracker>();
}
public class SomeWebApiController {
[HttpGet]
[Authorize]
public HttpResponseMessage GetProgress(int Id) {
var query = (from a in GlobalStatic.FooOperations where a.Id==Id select a);
if(!query.Any()) {
return Request.CreateResponse(HttpStatusCode.NotFound, "No operation with this Id found.");
} else {
return Request.CreateResponse(HttpStatusCode.Ok, query.First());
}
}
}
// this is javascript
// ... Your code until it starts the process.
// You'll have to get the ProgressTracker Id from the server somehow.
var InProgress = true;
window.setTimeout(function(e) {
var xmlhttp = new XMLHttpRequest();
var url = "<myHostSomething>/SomeWebApiController/GetProgress?Id="+theId;
xmlhttp.setRequestHeader("Authentication","bearer "+localStorage.getItem("access_token"));
xmlhttp.onreadystatechange = function() {
if (xmlhttp.readyState == 4 && xmlhttp.status == 200) {
var data = JSON.parse(xmlhttp.responseText);
updateProgressBar(data);
}
}
xmlhttp.open("GET", url, true);
xmlhttp.send();
function updateProgressBar(data) {
document.getElementById("myProgressText").innerHTML = data.Progress;
}
}, 3000);
Disclaimer: If my javascript is shitty, pardon me but I'm too used to using jQuery and all this fancy stuff x_x

How to manage observable subscription for dependent observables?

This sample console application has 2 observables. The first one pushes numbers from 1 to 100. This observable is subscribed by the AsyncClass which runs a long running process for each number it gets. Upon completion of this new async process I want to be able to 'push' to 2 subscribers which would be doing something with this new value.
My attempts are commented in the source code below.
AsyncClass:
class AsyncClass
{
private readonly IConnectableObservable<int> _source;
private readonly IDisposable _sourceDisposeObj;
public IObservable<string> _asyncOpObservable;
public AsyncClass(IConnectableObservable<int> source)
{
_source = source;
_sourceDisposeObj = _source.Subscribe(
ProcessArguments,
ExceptionHandler,
Completed
);
_source.Connect();
}
private void Completed()
{
Console.WriteLine("Completed");
Console.ReadKey();
}
private void ExceptionHandler(Exception exp)
{
throw exp;
}
private void ProcessArguments(int evtArgs)
{
Console.WriteLine("Argument being processed with value: " + evtArgs);
//_asyncOpObservable = LongRunningOperationAsync("hello").Publish();
// not going to work either since this creates a new observable for each value from main observer
}
// http://rxwiki.wikidot.com/101samples
public IObservable<string> LongRunningOperationAsync(string param)
{
// should not be creating an observable here, rather 'pushing' values?
return Observable.Create<string>(
o => Observable.ToAsync<string, string>(DoLongRunningOperation)(param).Subscribe(o)
);
}
private string DoLongRunningOperation(string arg)
{
return "Hello";
}
}
Main:
static void Main(string[] args)
{
var source = Observable
.Range(1, 100)
.Publish();
var asyncObj = new AsyncClass(source);
var _asyncTaskSource = asyncObj._asyncOpObservable;
var ui1 = new UI1(_asyncTaskSource);
var ui2 = new UI2(_asyncTaskSource);
}
UI1 (and UI2, they're basically the same):
class UI1
{
private IConnectableObservable<string> _asyncTaskSource;
private IDisposable _taskSourceDisposable;
public UI1(IConnectableObservable<string> asyncTaskSource)
{
_asyncTaskSource = asyncTaskSource;
_asyncTaskSource.Connect();
_taskSourceDisposable = _asyncTaskSource.Subscribe(RefreshUI, HandleException, Completed);
}
private void Completed()
{
Console.WriteLine("UI1: Stream completed");
}
private void HandleException(Exception obj)
{
Console.WriteLine("Exception! "+obj.Message);
}
private void RefreshUI(string obj)
{
Console.WriteLine("UI1: UI refreshing with value "+obj);
}
}
This is my first project with Rx so let me know if I should be thinking differently. Any help would be highly appreciated!

I'm going to let you know you should be thinking differently... :) Flippancy aside, this looks like a case of bad collision between object-oriented and functional-reactive styles.
It's not clear what the requirements are around timing of the data flow and caching of results here - the use of Publish and IConnectableObservable is a little confused. I'm going to guess you want to avoid the 2 downstream subscriptions causing the processing of a value being duplicated? I'm basing some of my answer on that premise. The use of Publish() can achieve this by allowing multiple subscribers to share a subscription to a single source.
Idiomatic Rx wants you to try and keep to a functional style. In order to do this, you want to present the long running work as a function. So let's say, instead of trying to wire your AsyncClass logic directly into the Rx chain as a class, you could present it as a function like this contrived example:
async Task<int> ProcessArgument(int argument)
{
// perform your lengthy calculation - maybe in an OO style,
// maybe creating class instances and invoking methods etc.
await Task.Delay(TimeSpan.FromSeconds(1));
return argument + 1;
}
Now, you can construct a complete Rx observable chain calling this function, and through the use of Publish().RefCount() you can avoid multiple subscribers causing duplicate effort. Note how this separates concerns too - the code processing the value is simpler because the reuse is handled elsewhere.
var query = source.SelectMany(x => ProcessArgument(x).ToObservable())
.Publish().RefCount();
By creating a single chain for subscribers, the work is only started when necessary on subscription. I've used Publish().RefCount() - but if you want to ensure values aren't missed by the second and subsequent subscribers, you could use Replay (easy) or use Publish() and then Connect - but you'll want the Connect logic outside the individual subscriber's code because you just need to call it once when all subscribers have subscribed.