Develop Reference

Buffering of observable to stabilize variable delays for slower observer

I have one observable that produces a sequence of numbers with delays in between the numbers that range from 0 to 1 second (randomly):
var random = new Random();
var randomDelaysObservable = Observable.Create<int>(async observer =>
{
var value = 0;
while (true)
{
// delay from 0 to 1 second
var randomDelay = TimeSpan.FromSeconds(random.NextDouble());
await Task.Delay(randomDelay);
observer.OnNext(value++);
}
return Disposable.Empty;
// ReSharper disable once FunctionNeverReturns
});
I would like to have a consumer that consumes those numbers and writes them out to the console, but only to take a single number every 2 seconds (exactly every two seconds).
Right now, I have this code for the observer (although I know it isn't correct with the use of await):
var delayedConsoleWritingObserver = Observer.Create<int>(async value =>
{
// fixed delay of 2 seconds
var fixedDelay = TimeSpan.FromSeconds(2);
await Task.Delay(fixedDelay);
Console.WriteLine($"[{DateTime.Now:O}] Received value: {value}.");
});
randomDelaysObservable.Subscribe(delayedConsoleWritingObserver);
If the producer produces numbers every 0 to 1 second, and the consumer is only able to consume single number every 2 seconds, it's clear that the producer produces the numbers faster than the consumer can consume them (backpressure). What I would like to do is to be able to "preload" e.g. 10 or 20 of the numbers from the producer in advance (if the consumer cannot process them fast enough) so that the consumer could consume them without the random delays (but not all of them as the observable sequence is infinite, and we'd run out of memory if it was running for some time).
This would sort of stabilize the variable delays from the producer if I have a slower consumer. However, I cannot think of a possible solution how to do this with the operators in ReactiveX, I've looked at the documentation of Buffer, Sample, Debounce and Window, and none of them look like the thing I'm looking for.
Any ideas on how this would be possible? Please note that even my observer code isn't really correct with the async/await, but I wasn't able to think of a better way to illustrate what I'm trying to achieve.
EDIT:
As pointed out by Schlomo, I might not have formulated the question well, as it looks a bit more like an XY-problem. I'm sorry about that. I'll try to illustrate the process I'm trying to model on another example. I don't really care that much about exact time delays on the producer or consumer side. The time delays were really just a placeholder for some asynchronous work that takes some time.
I'm more thinking about a general pattern, where producer produces items at some variable rate, I want to process all the items, and consumer also can only consume the items at some variable rate. And I'm trying to do this more effectively.
I'll try to illustrate this on a more real-world example of a pizza place 🍕.
Let's say that I'm the owner of a pizza place, and we serve just one kind of pizza, e.g. pizza Margherita.
I have one cook employed in the kitchen who makes the pizzas.
Whenever an order comes in for a pizza, he takes the order and prepares the pizza.
Now when I look at this as the owner, I see that it's not efficient. Every time a new order comes in, he has to start preparing the pizza. I think we can increase the throughput and serve the orders faster.
We only make one kind of pizza. I'm thinking that maybe if the cook has free time on his hands and there are no currently pending orders, he could prepare a couple of pizzas in advance. Let's say I'd let him prepare up to 10 pizzas in advance -- again, only when he has free time and is not busy fulfilling orders.
When we open the place in the morning, we've got no pizzas prepared in advance, and we just serve the orders as they come in. As soon as there's just a little bit of time and no orders are pending, the cook starts putting the pizzas aside in a queue. And he only stops once there are 10 pizzas in the queue. If there is an incoming order, we just fulfill it from the queue, and the cook needs to fill in the queue from the other end. For example, if we've got the queue completely filled with all 10 pizzas, and we take 1 pizza out, leaving 9 pizzas in the queue, the cook should immediately start preparing the 1 pizza to fill the queue again to 10 pizzas.
I see the generalized problem as a producer-consumer where the producer produces each item in some time, and consumer consumes each item in some time. And by adding this "buffer queue" between them, we can improve the throughput, so they wouldn't have to wait for each other that much. But I want to limit the size of the queue to 10 to avoid making too many pizzas in advance.
Now to the possible operators from Rx:
Throttle and Sample won't work because they are discarding items produced by the producer. Throughout the day, I don't want to throw away any pizzas that the cook makes. Maybe at the end of the day if few uneaten pizzas are left, it's ok, but I don't want to discard anything during the day.
Buffer won't work because that would just basically mean to prepare the pizzas in batches of 10. That's not what I want to do because I would still need to wait for every batch of 10 pizzas whenever the previous batch is gone. Also, I would still need to prepare the first batch of 10 pizzas first thing in the morning, and I couldn't just start fulfilling orders. So if there would be 10 people waiting in line before the place opens, I would serve all those 10 people at once. That's not how I want it to work, I want "first come first served" as soon as possible.
Window is a little bit better than Buffer in this sense, but I still don't think it works completely like the queue that I described above. Again, when the queue is filled with 10 pizzas, and one pizza gets out, I immediately want to start producing new pizza to fill the queue again, not wait until all 10 pizzas are out.
Hope this helps in illustrating my idea a little bit better. If it's still not clear, maybe I can come up with some better code samples and start a new question later.

Here's what your observables could look like using pure Rx:
var producer = Observable.Generate(
(r: new Random(), i: 0), // initial state
_ => true, // condition
t => (t.r, t.i + 1), // iterator
t => t.i, // result selector
t => TimeSpan.FromSeconds(t.r.NextDouble()) // timespan generator
);
var consumer = producer.Zip(
Observable.Interval(TimeSpan.FromSeconds(2)),
(_, i) => i
);
However, that isn't an easy thing to 'grab the first n without delay'. So we can instead create a non-time-gapped producer:
var rawProducer = Observable.Range(0, int.MaxValue);
then create the time gaps separately:
var timeGaps = Observable.Repeat(TimeSpan.Zero).Take(10) //or 20
.Concat(Observable.Generate(new Random(), r => true, r => r, r => TimeSpan.FromSeconds(r.NextDouble())));
then combine those two:
var timeGappedProducer = rawProducer.Zip(timeGaps, (i, ts) => Observable.Return(i).Delay(ts))
.Concat();
the consumer looks basically the same:
var lessPressureConsumer = timeGappedProducer .Zip(
Observable.Interval(TimeSpan.FromSeconds(2)),
(_, i) => i
);
Given all of that, I don't really understand why you want to do this. It's not a good way to handle back-pressure, and the question sounds like a bit of an XY-problem. The operators you mention (Sample, Throttle, etc.) are better ways of handling back-pressure.

Your problem as described is well suited to a simple bounded buffer shared between the producer and the consumer. The producer must have a condition associate with writing to the buffer stating that the buffer must not be full. The consumer must have a condition stating that the buffer cannot be empty.
See the following example using the Ada language.
with Ada.Text_IO; use Ada.Text_IO;
procedure Main is
type Order_Nums is range 1..10_000;
Type Index is mod 10;
type Buf_T is array(Index) of Order_Nums;
protected Orders is
entry Prepare(Order : in Order_Nums);
entry Sell(Order : out Order_Nums);
private
Buffer : Buf_T;
P_Index : Index := Index'First;
S_Index : Index := Index'First;
Count : Natural := 0;
end Orders;
protected body Orders is
entry Prepare(Order : in Order_Nums) when Count < Index'Modulus is
begin
Buffer(P_Index) := Order;
P_Index := P_Index + 1;
Count := Count + 1;
end Prepare;
entry Sell(Order : out Order_Nums) when Count > 0 is
begin
Order := Buffer(S_Index);
S_Index := S_Index + 1;
Count := Count - 1;
end Sell;
end Orders;
task Chef is
Entry Stop;
end Chef;
task Seller is
Entry Stop;
end Seller;
task body Chef is
The_Order : Order_Nums := Order_Nums'First;
begin
loop
select
accept Stop;
exit;
else
delay 1.0; -- one second
Orders.Prepare(The_Order);
Put_Line("Chef made order number " & The_Order'Image);
The_Order := The_Order + 1;
exit when The_Order = Order_Nums'Last;
end select;
end loop;
end Chef;
task body Seller is
The_Order : Order_Nums;
begin
loop
select
accept Stop;
exit;
else
delay 2.0; -- two seconds
Orders.Sell(The_Order);
Put_Line("Sold order number " & The_Order'Image);
end select;
end loop;
end Seller;
begin
delay 60.0; -- 60 seconds
Chef.Stop;
Seller.Stop;
end Main;
The shared buffer is named Orders. Orders contains a circular buffer of 10 Order_Nums. The index for the array containing the orders is declared as mod 10 which contains the values 0 through 9. Ada modular types exhibit wrap-around arithmetic, so incrementing past 9 wraps to 0.
The Prepare entry has a boundary condition requiring Count < Index'Moduluswhich evaluates to Count < 10 in this instance. The Sell entry has a boundary condition Count < 0.
The Chef task waits 1 second to produce a pizza, but waits until there is room in the buffer. As soon as there is room in the buffer Chef produces an order. Seller waits 2 seconds to consume an order.
Each task terminates when its Stop entry is called. Main waits 60 seconds and then calls the Stop entries for each task.
The output of the program is:
Chef made order number 1
Sold order number 1
Chef made order number 2
Chef made order number 3
Sold order number 2
Chef made order number 4
Chef made order number 5
Sold order number 3
Chef made order number 6
Chef made order number 7
Sold order number 4
Chef made order number 8
Chef made order number 9
Sold order number 5
Chef made order number 10
Chef made order number 11
Sold order number 6
Chef made order number 12
Chef made order number 13
Sold order number 7
Chef made order number 14
Chef made order number 15
Sold order number 8
Chef made order number 16
Chef made order number 17
Sold order number 9
Chef made order number 18
Chef made order number 19
Sold order number 10
Chef made order number 20
Sold order number 11
Chef made order number 21
Sold order number 12
Chef made order number 22
Chef made order number 23
Sold order number 13
Sold order number 14
Chef made order number 24
Sold order number 15
Chef made order number 25
Sold order number 16
Chef made order number 26
Chef made order number 27
Sold order number 17
Chef made order number 28
Sold order number 18
Chef made order number 29
Sold order number 19
Sold order number 20
Chef made order number 30
Sold order number 21
Chef made order number 31
Chef made order number 32
Sold order number 22
Sold order number 23
Chef made order number 33
Sold order number 24
Chef made order number 34
Sold order number 25
Chef made order number 35
Sold order number 26
Chef made order number 36
Chef made order number 37
Sold order number 27
Sold order number 28
Chef made order number 38
Sold order number 29
Chef made order number 39
Sold order number 30
Chef made order number 40
Sold order number 31

Profilling results - how to understand

I did profiling for my console application using Unity IOC and a lot of calls using HttpCLient. How to understand it?
Function Name, Inclusive Samples, Exclusive Samples, Inclusive Samples %, Exclusive Samples %
Microsoft.Practices.Unity.UnityContainer.Resolve 175 58 38.89 12.89
Microsoft.Practices.Unity.UnityContainer..ctor 29 29 6.44 6.44
System.Runtime.CompilerServices.AsyncTaskMethodBuilder1[System.DateTime].Start 36 13 8.00 2.89
Microsoft.Practices.Unity.UnityContainerExtensions.RegisterInstance 9 9 2.00 2.00
System.Net.Http.HttpClientHandler..ctor 9 9 2.00 2.00
System.Net.Http.HttpMessageInvoker.Dispose 9 9 2.00 2.00
System.Activator.CreateInstance 20 8 4.44 1.78
Microsoft.Practices.Unity.ObjectBuilder.NamedTypeDependencyResolverPolicy.Resolve 115 3 25.56 0.67
What means that inclusive samples for Microsoft.Practices.Unity.UnityContainer.Resolve are 38,89 but exclusive are 12,89? Is it ok? Not too much?

"Inclusive" means "exclusive time plus time spent in all callees".
Forget the "exclusive" stuff.
"Inclusive" is what it's costing you.
It says UnityContainer.Resolve is costing you 39% of time,
and Unity.ObjectBuilder.NamedTypeDependencyResolverPolicy.Resolve is costing you 26%.
It looks like the first one calls the second one, so you can't add their times together.
If you could avoid calling all that stuff, you would save at least 40%, giving you a speedup of at least 100/60 or 1.67 or 67%
By the way, that Unity stuff, while not exactly deprecated, is no longer being maintained.

Consistent number generator from multiple input variables

I wan't to generate a fictional job title from some information I have about the visitor.
For this, I have a table of about 30 different job titles:
01 CEO
02 CFO
03 Key Account Manager
...
29 Window Cleaner
30 Dishwasher
I'm trying to find a way to generate one of these titles from a few different variables like name, age, education history, work history and so on. I wan't it to be somewhat random but still consistent so that the same variables always result in the same title.
I also wan't the different variables to have some impact on the result. Lower numbers are "better" jobs and higher numbers are "worse" jobs, but it doesn't have to be very accurate, just not completely random.
So take these two people as an example.
Name: Joe Smith
Number of previous employers: 10
Number of years education: 8
Age: 56
Name: Samantha Smith
Number of previous employers: 1
Number of years education: 0
Age: 19
Now the reason I wan't the name in there is to have a bit of randomness, so that two co-workers of the same age with the same background doesn't get exactly the same title. So I was thinking of using the number of letters in the name to mix it up a bit.
Now I can generate consistent numbers in an infinite number of ways, like the number of letters in the name * age * years of education * number of employers. This would come out as 35 840 for Joe Smith and 247 for Samantha Smith. But I wan't it to be a number between 1-30 where Samantha is closer to 25-30 and Joe is closer to 1-5.
Maybe this is more of a math problem than a programming problem, but I have seen a lot of "What's your pirate name?" and similar apps out there and I can't figure out how they work. "What's your pirate name?" might be a bad example, since it's probably completely random and I wan't my variables to matter some, but the idea is the same.
What I have tried
I tried adding weights to variable groups so I would get an easier number to use in my calculations.
Age
01-20 5
20-30 4
30-40 3
40-50 2
...
Years of education
00-01 0
01-02 1
02-03 2
04-05 3
...
Add them together and play around with those numbers, but there was a lot of problems like everyone ending up in pretty much the same mid-range (no one got to be CEO or dishwasher, everyone was somewhere in the middle), not to mention how messy the code was.
Is there a good way to accomplish what I want to do without having to build a massive math engine?

int numberOfTitles = 30;
var semiRandomID = person.Name.GetHashCode()
^ person.NumberOfPreviousEmployers.GetHashCode()
^ person.NumberOfYearsEducation.GetHashCode()
^ person.Age.GetHashCode();
var semiRandomTitle = Math.Abs(semiRandomID) % numberOfTitles;
// adjust semiRandomTitle as you see fit
semiRandomTitle += ((person.Age / 10) - 2);
semiRandomTitle += (person.NumberOfYearsEducation / 2);
The semiRandomID is a number that is generated from unique hashes of each component. The numbers are unique so that you will always generate the same number for "Joe" for example, but they don't mean anything. It's just a number. So we take all those unique numbers and generate one job title out of the 30 available. Every person has the same chance to get each job title (probably some math freak will proof that there's egde cases to the contrary, but for all practical, non-cryptographic means, it's sufficient).
Now each person has one job title assigned that looks random. However, as it's math and not randomness, they will get the same every time.
Now lets assume Joe got Taxi-Driver, the number 20. However, he has 10 years of formal education, so you decide you want to have that aspect have some weight. You could just add the years onto the job title number, but that would make anyone with 30 years of college parties CEO, so you decide (arbitrarily) that each year of education counts for half a job title. You add (NumberOfYearsEducation / 2) to the job title.
Lets assume Jane got CIO, the number 5. However, she is only 22 years old, a little young to be that high on the list. Again, you could just add the years onto the job title number, but that would make anyone with 30 years of age a CEO, so you decide (arbitrarily) that each year counts as 1/10 of a job title. In addition, you think that being very young should instead subtract from the job title. All years below the first 20 should indeed be a negative weight. So the formula would be ((Age / 10) - 2). One point for each 10 years of age, with the first 2 counting as negative.

Musical representation of pi

I'm creating a program to output the values of pi as sound. I've seen a few videos and some other representations, and want to try to make my own for fun, except I'm unsure how to proceed.
There are the notes A,B,C,D,E,F and G.
Unfortunately, there are 10 possible values for a digit, 1,2,3,4,5,6,7,8,9, or 0. I read something about a rule of fifths, but I am unsure how I would map the 10 numbers to sound. I'm not sure if this is the right place to ask this question.
How would you go about mapping a sound to numbers 0-9? I'd like to keep the spread as even as possible, so if every possible sound was on a line, 0-9 would be evenly distributed on the line, if that makes sense.
Currently, I'm trying Console.beep(note frequency, 1000) with these frequencies http://www.phy.mtu.edu/~suits/notefreqs.html, and ignoring values that aren't 1-7. Any thoughts?

You either need to assign two extra notes to the two spare numbers or represent the digits with something else, how about a rest or an accent for a note or some other effect?Or of course you could include semi tones, you would have to miss one out though as there are 11 in an octave, or 12 including the octave (up or down) of the first note.The nicest sounding thing would be assign each digit to a note on a scale like E minor pentatonic, or G major or a sweet bluesy one or something.
Example blues scale in C minor:
0 = C //This is the root note
1 = E flat 2 = F 3 = G
flat 4 = G 5 = B flat 6 = C //This note is one octave
higher than the root note 7 = E flat 8 = F 9 = G flat
G flat in this scale is the blues note.

What if you consider the digits to be degrees of a scale rather than absolute note mappings?
In the key of C, as an example:
c = 1 // tonic or root
d = 2
e = 3 // third (major in this case)
f = 4
g = 5 // perfect fifth
a = 6
b = 7
c = 8 //octave
d = 9
As for the handling of 0 you could treat this as a rest.

You can use 8 to go up an octave, 9 to go down an octave and 0 for rest (silence).
Just a suggestion.
This will give variety to your song.
Hop this helped you.

When all of the sharps and flats are included, the western musical scale has 12 semitones. To move from one semitone to the next, you multiply the frequency by the twelfth root of 2 (which is 1.05946).
It might be interesting to invent a 10 semitone scale, where the next semitone is found by multiplying by the tenth root of 2 (which is 1.0718). Then map the digits to these ten semitones, and off we go...

Building up a matrix dynamically in the xy plane

I have to read some data sequentially(from a file) and put the data into a matrix. I don't know the rank of the matrix initially. For example consider the data is plotted on an x, y plane with years on the Y axis and increments in the x axis. At first the data came in for 1990 with 3 increments
year increment(1991) increment(1992) increment(1993)
1990 12 25 35
Note that I will only know about the increments after reading the data line. So next is 1989 with 4 increments. So it should be
year increment(1990) increment(1991) increment(1992) increment(1993)
1989 23 33 43 53
1990 0 12 25 35
Note that when the new data came in another increment year came in the y axis(1990).As there is no increment year of 1990 for year 1990 this has to be filled with zero or kept it empty, but the
In the end I have to create a matrix. For example
year increment(1990) increment(1991) increment(1992) increment(1993)
1989 23 33 43 53
1990 0 12 25 35
1991 0 0 23 33
To build up the matrix, the difficult part is I don't know the years/increments initially, I will only know after reading the entire data. I would like to plot the matrix while reading the data so that I can avoid more than one pass through the data.
The placement of the matrix in the xy axis will be only known after the entire data is processed!
Any suggestions?

I quite like the sparse matrix solution, but you could use a version of http://en.wikipedia.org/wiki/Dynamic_array. Dynamic arrays are arrays that you resize when they get too full. Resizing is expensive, but if you increase the size by a constant factor every time you resize the cost of resizing works out so that the total cost is still O(n) if the final size has n elements.
To use dynamic arrays for this, you could create two dynamic arrays for each row, one growing with years larger than those seen so far, and one growing with years smaller than those seen so far (so with years in decreasing order along the array).
Another way to do this would be to create a single area of storage for the matrix, with only the central section used, so there is always space to add entries in any direction. You would then have to check that increasing the size of this storage by a constant factor when you were about to run over the edges would lead to a total cost of at most O(n). I suspect that it would, but the constant factors might not be very good.

You can build it as a sparse matrix with SortedList<int, SortedList<int, int>>