How can I convert IObservable to byte[]?
I want to convert IObservable to a wav file and save it on the disk.
DisOutput d;
File.WriteAllBytes("outputsend.wav", d);
I have an array IObservable which I want to convert to byte[] so that I can write to a file. How can I convert IObservable to byte[] because WriteAllBytes takes byte[] as input
I assume that you have a framework that generates a WAV file on the fly in the form of an IObservable<Byte>. Presumably, when the generation is complete the IObservable<Byte> will fire OnCompleted to signal this.
IObservable<Byte> observable;
The simplest way to do what you want is to use ToList which will generate an IObservable<IList<Byte>> that fires when the source observable sequence completes:
observable.ToList().Subscribe(list => File.WriteAllBytes("outputsend.wav", list.ToArray());
What happens is that the ToList operator will collect all incoming bytes in a list that grows over time. When the incoming sequence completes the subscribers are notified and in this case the list of bytes is written to a file.
However, there is no need to buffer the bytes in memory. Instead they can be written directly to the file. It is important that the file is closed when the incoming stream of bytes completes and this can be achieved using this somewhat more complicated but also more efficient extension method:
static class ObservableExtensions {
public static IObservable<Unit> WriteToFile(this IObservable<Byte> source, String fileName) {
return Observable.Create<Unit>(
observer => {
var fileStream = new SerialDisposable();
return new CompositeDisposable(
source.Subscribe(
value => {
try {
if (fileStream.Disposable == null)
fileStream.Disposable = File.Create(fileName);
((FileStream) fileStream.Disposable).WriteByte(value);
}
catch (SystemException ex) {
observer.OnError(ex);
}
},
observer.OnError,
() => {
observer.OnNext(Unit.Default);
observer.OnCompleted();
}
),
fileStream
);
}
);
}
}
You can use this extension method like this:
observable.WriteToFile("outputsend.wav").Subscribe(_ => Console.WriteLine("Done"));
To handle errors you can use another overload of Subscribe:
observable.WriteToFile("outputsend.wav").Subscribe(
_ => Console.WriteLine("Done"),
ex => Console.WriteLine(ex)
);
This will write exceptions to the console. A more sophisticated approach would be required in a production quality application.
Related
I am not all that familiar with IObservable<T>, but a package I am using forced it on me in a way.
I am expected to return an IObservable<T> on which at a later point Subscribe is called, and then the result of that gets disposed immediately after. And my intent is to fill it with data from a massive dataset. It reads line by line from text files, for GB's worth of data.
But I can't seem to find a good example of how I can make my hour long while loop, into an observable in a way that it doesn't expect all data to be read up front.
I see that there are some variants of Observable.FromAsync, but Tasks aren't my strong suit either, and can't seem to get that to work either.
The best I managed so far is below. Which compiles and runs, but does absolutely nothing. Never even calls the code inside the Create as far as I can tell.
public static IObservable<Data> GetHistoricalData(DateTime startDate, DateTime endDate)
{
return Observable.Create<Data>(async (subject, token) =>
{
try
{
FileStream stream = null;
StreamReader sr = null;
DateTime date = startDate;
string path = string.Format("C:\\MYPATH\\{0}-{1}-{2}.csv", date.Year,
date.Month.ToString("00"), date.Day.ToString("00"));
while (date < endDate)
{
if (!File.Exists(path))
{
date.AddDays(1);
continue;
}
stream = File.Open(path, FileMode.Create, FileAccess.Read);
sr = new StreamReader(stream);
while (!sr.EndOfStream)
{
string line = await sr.ReadLineAsync();
Data d = ParseData(line);
subject.OnNext(d);
}
if (stream != null)
{
sr.Close();
stream.Close();
}
}
}
catch (Exception ex)
{
try
{
subject.OnError(ex);
}
catch (Exception)
{
Console.WriteLine("An exception was thrown while trying to call" +
" OnError on the observable subject -- means you're not" +
" catching exceptions");
throw;
}
}
}).Publish();
}
I am not even sure if what I want to do is even technically possible, as I am not sure how the Observable pattern works. But due to the context it seems to expect a server connection feeding it a DataStream normally. So I assume it can be done. But only with the right combination of Observable creation methods.
If anyone has some good documentation to read up on that explains this in a starter friendly way, that would be nice as well.
As requested, how the method is called below, but it mostly goes into a blackbox library.
IObservable<Data> data = GetHistoricalData(
new DateTime(2021, 1, 1, 0, 0, 0, DateTimeKind.Utc),
new DateTime(2021, 1, 5, 0, 0, 0, DateTimeKind.Utc));
// Build charts from data
IObservable<(Data, Chart)> dataWithChart = data.GenerateCharts(TimeFrame);
// Generate signals from the charts
IObservable<(Signal, Chart)> signalsWithChart = dataWithChart.GenerateSignals(
Symbol, strategy);
// We're interested in the signals only
IObservable<Signal> signals = signalsWithChart.SelectSignals();
// Show information from each signal
IDisposable subscription = signals.Subscribe(ShowSignal);
subscription.Dispose();
I think you should read "Introduction to Rx", especially the section about hot and cold observables, and publish and connect.
There are several smaller issues with your code.
I made a simpler version of your code, and it worked after I removed the call to .Publish(). And I'm fairly certain you don't want that here.
Publish makes a wrapper that supports multiple observers. You can make it work for multiple observers by using Publish(), and then calling Connect once all observers are subscribed. But publishing works better for "hot" streams, like mouse/keyboard events, for example. Your data is only read once. Any observable that connects after you call Connect will not get the data that has already been read. If you need multiple subscribers like this, I would return IConnectableObservable instead of IObservable, so you can Connect() to it once all observers are subscribed.
As for your code:
Keep it simple. Use using whenever you work with a stream, unless you know really well why you shouldn't.
Calculate the path inside the loop. Use string interpolation instead of string.Format().
Call subject.OnCompleted() at the end.
Re-assign the date variable. DateTime values in c# are immutable. date.AddDays() does not modify date, but returns a new DateTime.
Use the FileMode you actually want. You don't want .Create if you don't intend to call the code if the file doesn't exist.
This works for me:
public IObservable<Data> GetHistoricalData(DateTime startDate, DateTime endDate)
{
return Observable.Create<Data>(async (subject, token) =>
{
var date = startDate;
try
{
while (date < endDate)
{
string path = $#"C:\MYPATH\{date.Year}-{date.Month:00}-{date.Day:00}.csv";
if (File.Exists(path))
{
using (var stream = File.Open(path, FileMode.Open, FileAccess.Read))
using (var sr = new StreamReader(stream))
{
while (!sr.EndOfStream)
{
var line = await sr.ReadLineAsync();
if (!string.IsNullOrWhiteSpace(line))
{
var data = ParseData(line);
subject.OnNext(data);
}
}
}
}
date = date.AddDays(1);
}
} catch (Exception e)
{
subject.OnError(e);
}
subject.OnCompleted();
});
}
It seems that your code is consumed by a package that misuses the IObservable<T> monad, by treating it as an IEnumerable<T>. When someone Subscribes to an observable sequence and then immediately unsubscribes, they are only going to observe the notifications pushed synchronously during the subscription. Which is equivalent to enumerating synchronously an IEnumerable<T>. So my suggestion is to simplify your life by writing IEnumerable<T>-based code, and then pass the sequence to the package after converting it to observable with the ToObservable Rx operator:
public static IEnumerable<Data> GetHistoricalData(DateTime startDate, DateTime endDate)
{
//...
foreach (var line in File.ReadLines(path))
{
yield return ParseData(line);
}
//...
}
Conversion from enumerable to observable:
IObservable<Data> data = GetHistoricalData(date1, date2).ToObservable();
Here you go. One nice query:
public static IObservable<Data> GetHistoricalData(DateTime startDate, DateTime endDate) =>
from n in Observable.Range(0, int.MaxValue)
let d = startDate.AddDays(n)
where d < endDate
let path = $"C:\\MYPATH\\{d.ToString("yyyy-MM-dd")}.csv"
where File.Exists(path)
from l in Observable.Using(
() => File.Open(path, FileMode.Open, FileAccess.Read),
s => Observable.Using(
() => new StreamReader(s),
sr => Observable.While(
() => !sr.EndOfStream,
Observable.Defer(
() => Observable.FromAsync(
() => sr.ReadLineAsync())))))
select ParseData(l);
I have a class which takes a stream in the constructor. You can then set up callbacks for various events, and then call StartProcessing. The issue is that I want to use it from a function which should return an IEnumerable.
Example:
public class Parser
{
public Parser(System.IO.Stream s) { // saves stream and does some set up }
public delegate void OnParsedHandler(List<string> token);
public event OnParsedHandler OnParsedData;
public void StartProcessing()
{
// reads stream and makes callback when it has a whole record
}
}
public class Application
{
public IEnumerable<Thing> GetThings(System.IO.Stream s)
{
Parser p = new Parser(s);
p.OnParsedData += (List<string> str) =>
{
Thing t = new Thing(str[0]);
// here is where I would like to yield
// but I can't
yield return t;
};
p.StartProcessing();
}
}
Right now my solution, which isn't so great, is to put them all the Things into a List which is captured by the lambda, and then iterate over them after calling StartProcessing.
public class Application
{
public IEnumerable<Thing> GetThings(System.IO.Stream s)
{
Parser p = new Parser(s);
List<Thing> thingList = new List<Thing>();
p.OnParsedData += (List<string> str) =>
{
Thing t = new Thing(str[0]);
thingList .Add(t);
};
p.StartProcessing();
foreach(Thing t in thingList )
{
yield return t;
}
}
}
The issue here is that now I have to save all of the Thing objects into list.
The problem you have here is that you don't fundamentally have a "pull" mechanic here, you're trying to push data from the parser. If the parser is going to push data to you, rather than letting the caller pull the data, then GetThings should return an IObservable, rather than an IEnumerable, so the caller can consume the data when it's ready.
If it really is important to have a pull mechanic here then Parser shouldn't fire an event to indicate that it has new data, but rather the caller should be able to ask it for new data and have it get it; it should either return all of the parsed data, or itself return an IEnumerable.
Interesting question. I would like to build upon what #servy has said regarding push and pull. In your implementation above, you are effectively adapting a push mechanism to a pull interface.
Now, first things first. You have not specified whether the call to the StartProcessing() method is a blocking call or not. A couple of remarks regarding that:
If the method is blocking (synchronous), then there is really no point in adapting it to a pull model anyway. The caller will see all the data processed in a single blocking call.
In that regard, receiving the data indirectly via an event handler scatters into two seemingly unrelated constructs what should otherwise be a single, cohesive, explicit operation. For example:
void ProcessAll(Action<Thing> callback);
On the other hand, if the StartProcessing() method actually spawns a new thread (maybe better named BeginProcessing() and follow the Event-based Asynchronous Pattern or another async processing pattern), you could adapt it to a pull machanism by means of a synchronization construct using a wait handle: ManualResetEvent, mutex and the like. Pseudo-code:
public IEnumerable<Thing> GetThings(System.IO.Stream s)
{
var parser = new Parser(s);
var waitable = new AutoResetEvent(false);
Thing item = null;
parser.OnParsedData += (Thing thing) =>
{
item = thing;
waitable.Set();
};
IAsyncResult result = parser.BeginProcessing();
while (!result.IsCompleted)
{
waitable.WaitOne();
yield return item;
}
}
Disclaimer
The above code serves only as a means for presenting an idea. It is not thread-safe and the synchronization mechanics do not work properly. See the producer-consumer pattern for more information.
I have a simple IObservable that provides NMEA strings from a serial device:
var source = Observable.Create<string>(
observer =>
{
var port = new SerialPort("COM4", 4800, Parity.None, 8, StopBits.One);
string output = string.Empty;
port.DataReceived += (sender, eventArgs) =>
{
var input = (SerialPort)sender;
var buffer = new byte[input.BytesToRead];
try
{
input.Read(buffer, 0, buffer.Length);
}
catch (Exception exception)
{
observer.OnError(exception);
}
var encoding = Encoding.ASCII;
var data = encoding.GetString(buffer);
if (data.StartsWith("$") && output != string.Empty)
{
if (output.StartsWith("$") || data.EndsWith(Environment.NewLine))
{
output = output.TrimEnd(Environment.NewLine.ToCharArray());
observer.OnNext(output);
}
output = data;
}
else if ((output == string.Empty || data.StartsWith("$")) || output.StartsWith("$"))
{
output += data;
}
};
try
{
port.Open();
}
catch (Exception error)
{
observer.OnError(error);
}
return port;
}
);
Now I want to parse these strings to concrete message types that can be filtered later on.
Thus the IObservable needs to be more specialized as IObservable.
This can be solved by .Select() statements where the Result is reprojected into NMEAMessage...
(the following is just an example for a simple reprojection)
var selected = source.Select(s => { return new NmeaMessage(s); });
... but what happens if a reprojection can't be made (e.g. unknown message type or parse error for the delivered string). How to handle that? I can't call OnError here (because it is an Observable not an Observer. Just suppress the parse error and return nothing in this case? How to state that the source may be not a valid NMEA source? Should I create a virtual "Device" class (that uses the string source inside) instead of cascading or filtering Observables? The "Device" class could use events and then an Observable on top (Observable.FromEventPattern<>) could be created again.
I also want the "observing" Parser to be able to subscribe to different sources of IObservable. What is the best way to integrate the parser and reprojection into this scenario?
Wrap your NmeaMessage constructor in a static function like this:
public static NmeaMessage TryParseNmeaMessage(TInputData d)
{
if (IsValidInput(d))
return new NmeaMessage(d);
else
return null;
}
Then you can do something like this:
var inputData = Observable.Create(...);
var parsed = inputData.Select(d => TryParseNmeaMessage(d))
.Where(d => d != null);
Obviously you need to define IsValidInput() too.
Instead of null you could also return some BadNmeaMessage (subclass of NmeaMessage containing error info) from ParseMessage.
Then you can react on the good/bad messages separately using .OfType<(Bad)NmeaMessage>().
Or you could signal OnError (throw inside ParseMessage) and then restart the sequence.
See here for advanced error handling.
There is an excellent article about this sort of error handling in F#: Railway oriented programming.
The same principles can be used in C# with RX.
I have this class for explaining my problem:
public class DataObserver: IDisposable
{
private readonly List<IDisposable> _subscriptions = new List<IDisposable>();
private readonly SomeBusinessLogicServer _server;
public DataObserver(SomeBusinessLogicServer server, IObservable<SomeData> data)
{
_server = server;
_subscriptions.Add(data.Subscribe(TryHandle));
}
private void TryHandle(SomeData data)
{
try
{
_server.MakeApiCallAsync(data).Wait();
}
catch (Exception)
{
// Handle exceptions somehow!
}
}
public void Dispose()
{
_subscriptions.ForEach(s => s.Dispose());
_subscriptions.Clear();
}
}
A) How can I avoid blocking inside the TryHandle() function?
B) How would you publish exceptions caught inside that function for handling them properly?
The Rx Design Guidelines provide a lot of useful advice when writing your own Rx operators:
http://go.microsoft.com/fwlink/?LinkID=205219
I'm sure I'll get lambasted for linking to an external article, but this link has been good for a couple of years and it's too big to republish on SO.
First, take a look at CompositeDisposable instead of re-implementing it yourself.
Other than that, there are many answers to your question. I have found that the best insight I've had when working with Rx is realizing that most cases where you want to subscribe are really just more chains in the observable you are building and you don't really want to subscribe but instead want to apply yet another transform to the incoming observable. And let some code that is further "on the edge of the system" and has more knowledge of how to handle errors do the actual subscribing
In the example you have presented:
A) Don't block by just transforming the IObservable<SomeData> into an IObservable<Task> (which is really better expressed as an IObservable<IObservable<Unit>>).
B) Publish exceptions by just ending the observable with an error or, if you don't want the exception to end the observable, exposing an IObservable<Exception>.
Here's how I'd re-write your example, assuming you did not want the stream to end on error, but instead just keep running after reporting the errors:
public static class DataObserver
{
public static IObservable<Exception> ApplyLogic(this IObservable<SomeData> source, SomeBusinessLogicServer server)
{
return source
.Select(data =>
{
// execute the async method as an observable<Unit>
// ignore its results, but capture its error (if any) and yield it.
return Observable
.FromAsync(() => server.MakeApiCallAsync(data))
.IgnoreElements()
.Select(_ => (Exception)null) // to cast from IObservable<Unit> to IObservable<Exception>
.Catch((Exception e) => Observable.Return(e));
})
// runs the Api calls sequentially (so they will not run concurrently)
// If you prefer to let the calls run in parallel, then use
// .Merge() instead of .Concat()
.Concat() ;
}
}
// Usage (in Main() perhaps)
IObservable<SomeData> dataStream = ...;
var subscription = dataStream.ApplyLogic(server).Subscribe(error =>
{
Console.WriteLine("An error occurred processing a dataItem: {0}", error);
}, fatalError =>
{
Console.WriteLine("A fatal error occurred retrieving data from the dataStream: {0}", fatalError);
});
I'm struggling here. Normally I'd read a book but there aren't any yet. I've found countless examples of various things to do with reading streams using RX but I'm finding it very hard to get my head around.
I know I can use Observable.FromAsyncPattern to create a wrapper of the Stream's BeginRead/EndRead or BeginReadLine/EndReadLine methods.
But this only reads once -- when the first observer subscribes.
I want an Observable which will keep reading and pumping OnNext until the stream errors or ends.
In addition to this, I'd also like to know how I can then share that observable with multiple subscribers so they all get the items.
You can use Repeat in order to keep reading lines until the end of the stream and Publish or Replay in order to control sharing across multiple readers.
An example of a simple, full Rx solution for reading lines from any stream until the end would be:
public static IObservable<string> ReadLines(Stream stream)
{
return Observable.Using(
() => new StreamReader(stream),
reader => Observable.FromAsync(reader.ReadLineAsync)
.Repeat()
.TakeWhile(line => line != null));
}
This solution also takes advantage of the fact that ReadLine returns null when the end of the stream is reached.
Adding to Lee's answer, using rxx:
using (new FileStream(#"filename.txt", FileMode.Open)
.ReadToEndObservable()
.Subscribe(x => Console.WriteLine(x.Length)))
{
Console.ReadKey();
}
The length of the read buffers will be outputted.
Heh - gonna reuse one of my other answers here (well, part of it, anyways):
Ref: Reading from NetworkStream corrupts the buffer
In that, I've got an extension method like this:
public static class Ext
{
public static IObservable<byte[]> ReadObservable(this Stream stream, int bufferSize)
{
// to hold read data
var buffer = new byte[bufferSize];
// Step 1: async signature => observable factory
var asyncRead = Observable.FromAsyncPattern<byte[], int, int, int>(
stream.BeginRead,
stream.EndRead);
return Observable.While(
// while there is data to be read
() => stream.CanRead,
// iteratively invoke the observable factory, which will
// "recreate" it such that it will start from the current
// stream position - hence "0" for offset
Observable.Defer(() => asyncRead(buffer, 0, bufferSize))
.Select(readBytes => buffer.Take(readBytes).ToArray()));
}
}
You can probably use this as written in a form like so:
// Note: ToEnumerable works here because your filestream
// has a finite length - don't do this with infinite streams!
var blobboData = stream
.ReadObservable(bufferSize)
// take while we're still reading data
.TakeWhile(returnBuffer => returnBuffer.Length > 0)
.ToEnumerable()
// mash them all together
.SelectMany(buffer => buffer)
.ToArray();
The solution is to use Observable.Create
Here is an example which can be adapated for reading any kind of stream
public static IConnectableObservable<Command> GetReadObservable(this CommandReader reader)
{
return Observable.Create<Command>(async (subject, token) =>
{
try
{
while (true)
{
if (token.IsCancellationRequested)
{
subject.OnCompleted();
return;
}
//this part here can be changed to something like this
//int received = await Task.Factory.FromAsync<int>(innerSocket.BeginReceive(data, offset, size, SocketFlags.None, null, null), innerSocket.EndReceive);
Command cmd = await reader.ReadCommandAsync();
subject.OnNext(cmd);
}
}
catch (Exception ex)
{
try
{
subject.OnError(ex);
}
catch (Exception)
{
Debug.WriteLine("An exception was thrown while trying to call OnError on the observable subject -- means you're not catching exceptions everywhere");
throw;
}
}
}).Publish();
}
Don't forget to call Connect() on the returned IConnectableObservable