Develop Reference

Keras.NET How to Use KerasIterator

I want to do the same as F. Chollet's notebook but in C#.
However, I can't find a way to iterate over my KerasIterator object:
def extract_features(directory, sample_count):
features = np.zeros(shape=(sample_count, 4, 4, 512))
labels = np.zeros(shape=(sample_count))
generator = datagen.flow_from_directory(
directory,
target_size=(150, 150),
batch_size=20,
class_mode='binary'
)
i = 0
for inputs_batch, labels_batch in generator:
features_batch = conv_base.predict(inputs_batch)
features[i * 20 : (i + 1) * 20] = features_batch
labels[i * 20 : (i + 1) * 20] = labels_batch
i += 1
if i * 20 >= sample_count:
break
return features, labels
My current C# function is below.
Details:
foreach loop gives me a 'Compiler Error CS1579' on KerasIterator
for loop is not possible because a KerasIterator doesn't have a Length (or Size, etc.)
private List<NDarray> ExtractFeatures(ImageDataGenerator datagen, String directory, int sample_count)
{
// create the return NDarrays
NDarray features = np.zeros(shape: (sample_count, 4, 4, 512));
NDarray labels = np.zeros(shape: (sample_count, 4, 4, 512));
KerasIterator generator = datagen.FlowFromDirectory(
directory,
target_size: (150, 150).ToTuple(),
batch_size: 20,
class_mode: "binary"
);
int i = 0;
/* how do I iterate on 'generator' here, in a similar way than in Python? */
// return the List element
return new List<NDarray> { features, labels };
}

As of April 19. 2020 it is not possible with the .NET Wrapper as documented in this issue on the GitHub page for Keras.NET

C# percentile rank algorithm

I'm trying to reverse engineer a calculation that was done on an old program but can't quite get it. I need a count of how many values are in the top 27%, middle 46%, and bottom 27%.
I have the following data sets with eleven values in each and what the program yields in regards to the percentages and the number of values that fall into those percentiles.
Upper 27%: 4, Middle 46%: 4, Lower 27%: 3
values: 8,9,10,11,11,11,11,12,12,12,13
Upper 27%: 5, Middle 46%: 4, Lower 27%: 2
values: 2,3,4,4,4,4,5,5,5,5,5
Upper 27%: 2, Middle 46%: 8, Lower 27%: 1
values: 2,4,4,4,4,4,4,4,4,5,5
Upper 27%: 2, Middle 46%: 6, Lower 27%: 3
values: 13,17,17,18,19,19,19,21,21,23,24
I have found formulas such as (n * p) where n is the number of values and p is the percentile but it doesn't seem to work across all of these data sets to give the same results. I'm a little lost and haven't found anything that produces the results here.
I have tested code that I have found on the internet but none have worked on the different data sets.
Code sample that I tried:
internal static double percentile(double[] sortedData, double p)
{
if (p >= 100.0d) return sortedData[sortedData.Length - 1];
double position = (double)(sortedData.Length + 1) * p / 100.0;
double leftNumber = 0.0d, rightNumber = 0.0d;
double n = p / 100.0d * (sortedData.Length - 1) + 1.0d;
if (position >= 1)
{
leftNumber = sortedData[(int)System.Math.Floor(n) - 1];
rightNumber = sortedData[(int)System.Math.Floor(n)];
}
else
{
leftNumber = sortedData[0]; // first data
rightNumber = sortedData[1]; // first data
}
if (leftNumber == rightNumber)
return leftNumber;
else
{
double part = n - System.Math.Floor(n);
return leftNumber + part * (rightNumber - leftNumber);
}
}
Is there a formula or a name for what I am trying to do? Am I on the right track with percentile ranks?

You're on the right path. This is indeed the percentile rank formula. My initial thought was to get the value at the 27th percentile (which looking at your code it looks like you started down this path as well) and figure out how many values were greater than or less than; but the values you provided don't support these numbers very well.
Therefore the approach I took was to calculate the percentile rank of each number and put it into a count if they matched the percentile you have above. This seems to be the approach they took.
Formula (check out this website for more info):
PR% = L + ( 0.5 x S ) / N
Where,
L = Number of below rank,
S = Number of same rank,
N = Total numbers.
Code:
var lower = 0;
var middle = 0;
var upper = 0;
// var values = new int[] { 8, 9, 10, 11, 11, 11, 11, 12, 12, 12, 13 };
// var values = new int[] { 2, 3, 4, 4, 4, 4, 5, 5, 5, 5, 5 };
// var values = new int[] { 2, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5 };
var values = new int[] { 13, 17, 17, 18, 19, 19, 19, 21, 21, 23, 24 };
var n = values.Length;
foreach(var i in values)
{
var pr = ((values.Count(v => v < i) + (.5 * values.Count(v => v == i))) / n);
if (pr < .27)
lower += 1;
else if (pr > .73)
upper += 1;
else
middle += 1;
}
Console.WriteLine("Upper: " + upper);
Console.WriteLine("Middle: " + middle);
Console.WriteLine("Lower: " + lower);

C# Iterating an If statement with new value each time

Is it possible to iterate nested if statements with a new value with every single iteration? I am trying to build a 1-dimensional cellular automata (for homework, I cannot deny it) and I'm completely new to C# as the following code will no doubt assure. I have tried to create this program using the most straightforward, basic, DIY methods available and have run myself into a rut.
I've got a string of 1's and 0's of length 8, say
string y;
y = "11110000";
I want to break this set up in 8 substring sets of 3 with each set comprising of a value in y together with a single value on either side of it. So counting from 0, the 3rd set would be 110, the 7th would be 001. However substrings will only provide the 1st to 6th set as I can't loop them around y to my liking so I defined the following-
y1=y.Substring(7,1)+y+y.Substring(0,1);
Using y1 I was able to get out all the substrings necessary. These were defined pretty basically as follows-
string a0, a1, a2, a3, a4, a5, a6, a7;
a0 = y1.Substring(0, 3);
a1 = y1.Substring(1, 3);
a2 = y1.Substring(2, 3);
a3 = y1.Substring(3, 3);
a4 = y1.Substring(4, 3);
a5 = y1.Substring(5, 3);
a6 = y1.Substring(6, 3);
a7 = y1.Substring(7, 3);
The rules for the next generation of cellular automata are up to the user in this program- that is to say the user can choose whether or not a substring, say 111->0 or 1 for all iterations. I used (an awful lot of) if tables in the following way for each substring
{
if (a0=="000")
{
Console.Write(a);
}
else if (a0=="001")
{
Console.Write(b);
}
else if (a0 =="010")
{
Console.Write(c);
}
else if (a0 == "011")
{
Console.Write(d);
}
else if (a0 == "100")
{
Console.Write(e);
}
else if (a0 == "101")
{
Console.Write(f);
}
else if (a0 == "110")
{
Console.Write(g);
}
else if (a0 == "111")
{
Console.Write(h);
}
}
where a,b,c,d,e,f,g,h are ints and are rules chosen by the user. So say for instance the user decides that each set 000 should result in a 1 value, then a=1. b corresponds to {0,0,1}, c to {0,1,0} and so on. However the fairly obvious problem with this method is that I end up with only 1 generation in ints that I can't get at. I'd love to replace y1 with this new generation (converted into a string). If this isn't possible let me know!
This link might also clear things up a bit

and here's how you COULD have gotten an A+ :D
private static int[,] HipPriestsHomework()
{
string y = "11110000";
Console.WriteLine(y);
var rules = new[]
{
new {pattern = 0, result = 0},
new {pattern = 1, result = 1},
new {pattern = 2, result = 1},
new {pattern = 3, result = 1},
new {pattern = 4, result = 1},
new {pattern = 5, result = 0},
new {pattern = 6, result = 0},
new {pattern = 7, result = 0},
};
Dictionary<int, int> rulesLookup = new Dictionary<int, int>();
foreach(var rule in rules)
{
rulesLookup.Add(rule.pattern, rule.result);
}
int numGenerations = 10;
int inputSize = y.Length;
int[,] output = new int[numGenerations, inputSize];
int[] items = new int[y.Length];
for(int inputIndex = 0; inputIndex< y.Length; inputIndex++)
{
string token = y.Substring(inputIndex, 1);
int item = Convert.ToInt32(token);
items[inputIndex] = item;
}
int[] working = new int[items.Length];
items.CopyTo(working, 0);
for (int generation = 0; generation < numGenerations; generation++)
{
for (uint y_scan = 0; y_scan < items.Length; y_scan++)
{
int a = items[(y_scan - 1) % items.Length];
int b = items[y_scan % items.Length];
int c = items[(y_scan + 1) % items.Length];
int pattern = a << 2 | b << 1 | c;
var match = rules[pattern];
output[generation, y_scan] = match.result;
working[y_scan] = match.result;
Console.Write(match.result);
}
working.CopyTo(items, 0);
Console.WriteLine();
}
return output;
}

How do you set DateTime range on X axis for System.Windows.Forms.DataVisualization.Charting?

Presently I am attempting to display a chart using windows forms that shows monthly data on the X axis and an integer value on the Y axis; however, I am not setting the range properly for the X Axis, where MonthYear is a DateTime:
var pnChart = new System.Windows.Forms.Panel();
pnChart.Controls.Clear();
DataTable dtChartData = myDatabaseLayer.BuildDataTable("SELECT Added, Modified FROM tblStatistics WHERE ApplicationID = " + intApplicationID + " ORDER BY MonthYear");
Chart chart = GenerateChart(dtChartData, pnChart.Width, pnChart.Height, "ActiveBorder", 6);
chart.Series[0].XValueType = ChartValueType.DateTime;
chart.ChartAreas[0].AxisX.LabelStyle.Format = "yyyy-MM-dd";
chart.ChartAreas[0].AxisX.Interval = 1;
chart.ChartAreas[0].AxisX.IntervalType = DateTimeIntervalType.Months;
chart.ChartAreas[0].AxisX.IntervalOffset = 1;
pnChart.Controls.Add(chart);
The problem is, when the chart is displayed, the X axis has the datetime "1900-01-01" so my question is, how do I set the date range to start at 2013-01-01?
Please note that I have searched the internet and tried the following settings, but they do not give me the correct range:
chart.ChartAreas[0].AxisX.Maximum = DateTime.Now.Ticks;
Or,
chart.ChartAreas[0].AxisX.Crossing = DateTime.Now.Ticks;
Or,
chart.ChartAreas[0].AxisX.Minimum = DateTime.Now.Ticks;
TIA.
UPDATE:
Please note that I found how to set the range properly using this:
chart.Series[0].XValueType = ChartValueType.DateTime;
DateTime minDate = new DateTime(2013, 01, 01);
DateTime maxDate = DateTime.Now;
chart.ChartAreas[0].AxisX.Minimum = minDate.ToOADate();
chart.ChartAreas[0].AxisX.Maximum = maxDate.ToOADate();
The above code sets the proper range now for the X axis; however, now chart itself is blank.
UPDATE 2:
Yes, thank you DasKrumelmonster--that fixed it! I was using code from http://www.codeproject.com/Articles/168056/Windows-Charting-Application, and simply should have looked more closely at the author's protected internal Chart GenerateChart(DataTable dtChartDataSource, int width, int height, string bgColor, int intType) function. To correct the issue, I changed these lines:
foreach (DataRow dr in dtChartDataSource.Rows)
{
double dataPoint = 0;
double.TryParse(dr[dc.ColumnName].ToString(), out dataPoint);
DataPoint objDataPoint = new DataPoint() { AxisLabel = "series", YValues = new double[] { dataPoint } };
chart.Series[series].Points.Add(dataPoint);
}
To this:
DateTime dtStart = new DateTime(2013, 01, 01);
int intMonthCounter = 0;
//Add data points to the series
foreach (DataRow dr in dtChartDataSource.Rows)
{
double dataPoint = 0;
double.TryParse(dr[dc.ColumnName].ToString(), out dataPoint);
DataPoint objDataPoint = new DataPoint() { AxisLabel = "series", YValues = new double[] { dataPoint } };
chart.Series[series].Points.AddXY(dtStart.AddMonths(intMonthCounter), dataPoint);
intMonthCounter++;
}
Thank you!

Cannot reproduce. I tried this code:
private void button1_Click(object sender, EventArgs e)
{
var s = new Series();
s.ChartType = SeriesChartType.Line;
var d = new DateTime(2013, 04, 01);
s.Points.AddXY(d, 3);
s.Points.AddXY(d.AddMonths(-1), 2);
s.Points.AddXY(d.AddMonths(-2), 1);
s.Points.AddXY(d.AddMonths(-3), 4);
chart1.Series.Clear();
chart1.Series.Add(s);
chart1.Series[0].XValueType = ChartValueType.DateTime;
chart1.ChartAreas.Add(new ChartArea()); // In some cases the winforms designer adds this already
chart1.ChartAreas[0].AxisX.LabelStyle.Format = "yyyy-MM-dd";
chart1.ChartAreas[0].AxisX.Interval = 1;
chart1.ChartAreas[0].AxisX.IntervalType = DateTimeIntervalType.Months;
chart1.ChartAreas[0].AxisX.IntervalOffset = 1;
chart1.Series[0].XValueType = ChartValueType.DateTime;
DateTime minDate = new DateTime(2013, 01, 01).AddSeconds(-1);
DateTime maxDate = new DateTime(2013, 05, 01); // or DateTime.Now;
chart1.ChartAreas[0].AxisX.Minimum = minDate.ToOADate();
chart1.ChartAreas[0].AxisX.Maximum = maxDate.ToOADate();
}
Maybe I fixed your error on the way.
It works as expected: One line with four data points, all x-axes labels are visible and so is the graph itself. If there is still an issue, please provide full testing code along with a description of what should happen vs. what actually happens.

LINQ to find the closest number that is greater / less than an input

Suppose I have this number list:
List<int> = new List<int>(){3,5,8,11,12,13,14,21}
Suppose that I want to get the closest number that is less than 11, it would be 8
Suppose that I want to get the closest number that is greater than 13 that would be 14.
The numbers in list can't be duplicated and are always ordered. How can I write Linq for this?

with Linq assuming that the list is ordered I would do it like this:
var l = new List<int>() { 3, 5, 8, 11, 12, 13, 14, 21 };
var lessThan11 = l.TakeWhile(p => p < 11).Last();
var greaterThan13 = l.SkipWhile(p => p <= 13).First();
EDIT:
As I have received negative feedback about this answer and for the sake of people that may see this answer and while it's accepted don't go further, I explored the other comments regarding BinarySearch and decided to add the second option in here (with some minor change).
This is the not sufficient way presented somewhere else:
var l = new List<int>() { 3, 5, 8, 11, 12, 13, 14, 21 };
var indexLessThan11 = ~l.BinarySearch(10) -1;
var value = l[indexLessThan11];
Now the code above doesn't cope with the fact that the value 10 might actually be in the list (in which case one shouldn't invert the index)! so the good way is to do it:
var l = new List<int>() { 3, 5, 8, 11, 12, 13, 14, 21 };
var indexLessThan11 = l.BinarySearch(10);
if (indexLessThan11 < 0) // the value 10 wasn't found
{
indexLessThan11 = ~indexLessThan11;
indexLessThan11 -= 1;
}
var value = l[indexLessThan11];
I simply want to note that:
l.BinarySearch(11) == 3
//and
l.BinarySearch(10) == -4;

Use Array.BinarySearch - no need for LINQ or visiting on average half the elements to find your target.
There are also a variety of SortedXXX classes that may be suitable for what you're doing [that will have such efficient O(log N) searches built-in]

You can do this using a binary search. If your searching for 11, well obviously you'll get the index your after. If you search for 10 and use the bitwise complement of the result, you'll get the closest match.
List<int> list = new List<int>(){3,5,8,11,12,13,14,21};
list.Sort();
int index = list.BinarySearch(10);
int found = (~index)-1;
Console.WriteLine (list[found]); // Outputs 8
The same goes searching in the other direction
int index = list.BinarySearch(15);
Console.WriteLine("Closest match : " + list[+~index]); // Outputs 21
Binary searches are also extremely fast.

closest number below 11:
int someNumber = 11;
List<int> list = new List<int> { 3, 5, 8, 11, 12, 13, 14, 21 };
var intermediate = from i in list
where i < someNumber
orderby i descending
select i;
var result = intermediate.FirstOrDefault();
closest number above 13:
int someNumber = 13;
List<int> list = new List<int> { 3, 5, 8, 11, 12, 13, 14, 21 };
var intermediate = from i in list
where i > someNumber
orderby i
select i;
var result = intermediate.FirstOrDefault();

This is my answer
List<int> myList = new List<int>() { 3, 5, 8, 11, 12, 13, 14, 21 };
int n = 11;
int? smallerNumberCloseToInput = (from n1 in myList
where n1 < n
orderby n1 descending
select n1).First();
int? largerNumberCloseToInput = (from n1 in myList
where n1 > n
orderby n1 ascending
select n1).First();

var list = new List<int> {14,2,13,11,5,8,21,12,3};
var tested = 11;
var closestGreater = list.OrderBy(n => n)
.FirstOrDefault(n => tested < n); // = 12
var closestLess = list.OrderByDescending(n => n)
.FirstOrDefault(n => tested > n); // = 8
if (closestGreater == 0)
System.Diagnostics.Debug.WriteLine(
string.Format("No number greater then {0} exists in the list", tested));
if (closestLess == 0)
System.Diagnostics.Debug.WriteLine(
string.Format("No number smaler then {0} exists in the list", tested));

Here is my way hope this helps somebody!
List<float> list = new List<float> { 4.0f, 5.0f, 6.0f, 10.0f, 4.5f, 4.0f, 5.0f, 6.0f, 10.0f, 4.5f, 4.0f, 5.0f, 6.0f, 10.0f };
float num = 4.7f;
float closestAbove = list.Aggregate((x , y) => (x < num ? y : y < num ? x : (Math.Abs(x - num)) < Math.Abs(y - num) ? x : y));
float closestBelow = list.Aggregate((x , y) => (x > num ? y : y > num ? x : (Math.Abs(x - num)) < Math.Abs(y - num) ? x : y));
Console.WriteLine(closestAbove);
Console.WriteLine(closestBelow);
This means you dont have to order the list
Credit: addapted from here: How to get the closest number from a List<int> with LINQ?
The Expanded Code
float closestAboveExplained = list.Aggregate((closestAbove , next) => {
if(next < num){
return closestAbove;
}
if(closestAbove < num){
return next;
}
else{
if(Math.Abs(closestAbove - num) < Math.Abs(next - num)){
return closestAbove;
}
}
return next;
});

You can use a query for this such as:
List<int> numbers = new List<int>() { 3, 5, 8, 11, 12, 13, 14, 21 };
List<int> output = (from n in numbers
where n > 13 // or whatever
orderby n ascending //or descending
select n).ToList();