Develop Reference

Accord.net NaiveBayesLearning "Index was outside the bounds of the array"

I am using Accord.net 3.7.0 in dot net core 1.1.
The algorithm I use is naive bayesian. And the source code of the learning mechanism is as follows:
public LearningResultViewModel NaiveBayes(int[][] inputs, int[] outputs)
{
// Create a new Naive Bayes learning
var learner = new NaiveBayesLearning();
// Learn a Naive Bayes model from the examples
NaiveBayes nb = learner.Learn(inputs, outputs);
#region test phase
// Compute the machine outputs
int[] predicted = nb.Decide(inputs);
// Use confusion matrix to compute some statistics.
ConfusionMatrix confusionMatrix = new ConfusionMatrix(predicted, outputs, 1, 0);
#endregion
LearningResultViewModel result = new LearningResultViewModel()
{
Distributions = nb.Distributions,
NumberOfClasses = nb.NumberOfClasses,
NumberOfInputs = nb.NumberOfInputs,
NumberOfOutputs = nb.NumberOfOutputs,
NumberOfSymbols = nb.NumberOfSymbols,
Priors = nb.Priors,
confusionMatrix = confusionMatrix
};
return result;
}
I have tested this piece of code on a little data but as data grew the
Index was outside the bounds of the array
error occurred.
As I can't navigate in the Learn method so I don't know what to do. the screen shot of the run-time is this:
No extra information, no inner exception no IDEA!!!
TG.
// UPDATE_1 ***
The inputs array is a 180 by 4 matrix (array) as the bellow image shows:
which has 4 columns in every row. checked by hand (I can share its video too if needed!!!)
The outputs array is a 180 one as shown here:
which only contains 0 and 1 (I can share its video too if needed!!!).
And about NaiveBayesinLearning doc is here:
NaiveBayesinLearning
More examples bottom of this page:
More examples
And the learn method docs here:
learn method doc

According to the comments and the Ideas from them I have suspected to the values of matrix. So I have investigated it:
As shown in image above, some rows have below zero values. The inputs matrix is generated by Codification which is used in the examples of here:
NaiveBayes
with the docs below:
Codification docs
the codification -1 was with the values of null. Like the screen shot below:
So my solution was replacing null values with "null". But may be there is better solutions.
Now the caller method that contains fixed data is as follows:
public LearningResultViewModel Learn(EMVDBContext dBContext, string userId, LearningAlgorithm learningAlgorithm)
{
var learningDataRaw = dBContext.Mutants
.Include(mu => mu.MutationOperator)
.Where(mu => mu.Equivalecy == 0 || mu.Equivalecy == 10);
string[] featureTitles = new string[] {
"ChangeType",
"OperatorName",
"OperatorBefore",
"OperatorAfter",
};
string[][] learningInputNotCodified = learningDataRaw.Select(ldr => new string[] {
ldr.ChangeType.ToString(),
ldr.MutationOperator.Name??"null",
ldr.MutationOperator.Before??"null",
ldr.MutationOperator.After??"null",
}).ToArray();
int[] learningOutputNotCodified = learningDataRaw.Select(ldr => ldr.Equivalecy == 0 ? 0 : 1).ToArray();
#region Codification phase
// Create a new codification codebook to
// convert strings into discrete symbols
Codification codebook = new Codification(featureTitles, learningInputNotCodified);
// Extract input and output pairs to train
int[][] learningInput = codebook.Transform(learningInputNotCodified);
switch (learningAlgorithm)
{
case LearningAlgorithm.NaiveBayesian:
return learningService.NaiveBayes(learningInput, learningOutputNotCodified);
break;
case LearningAlgorithm.SVM:
break;
default:
break;
}
#endregion
return null;
}
I wish this will help the others encountering same problem.

How to know which checkbox are selected if I have a number with the sum of all values checked?

I have an existing database, and my table has a field in which is stored a value, a sum of all "values" of my checkbox list.
Another programmer made this method, called it "logic sum". I don't know this method, but now I need to know the reverse method, so I can know which checkboxes are selected by using the Sum value of all their values.
I really don't know how to do this ... I am using C# with asp.net, but the logic should to be the same for all languages.
Can you help me?

Normally these things are done with powers of 2... The first checkbox has a "weight" of 1, the second one has a "weight" of 2, the third one a "weight" of 4 and so on.
You can check if a checkbox is checked by doing
if ((sum & 1) != 0) // first is checked
if ((sum & 2) != 0) // second is checked
if ((sum & 4) != 0) // third is checked
or
bool firstIsChecked = (sum & 1) != 0;
bool secondIsChecked = (sum & 2) != 0;
bool thirdIsChecked = (sum & 4) != 0;
and so on.
if you don't like binary math, you can go through an enum to leverage the Enum.HasFlag method:
[Flags]
public enum MyChechboxes
{
FooCheckbox = 1,
BarCheckbox = 2,
BazCheckbox = 4,
}
int sum = 5;
MyChechboxes checkeds = (MyChechboxes)sum;
bool firstIsChecked = checkeds.HasFlag(MyChechboxes.FooCheckbox);
bool secondIsChecked = checkeds.HasFlag(MyChechboxes.BarCheckbox);
bool thirdIsChecked = checkeds.HasFlag(MyChechboxes.BazCheckbox);

Let's assume you have a company with 5 features (A,B,C,D,E)
Saving..
if(A.isChecked)
features = features | 1;
if(B.isChecked)
features = features | 2;
if(C.isChecked)
features = features | 4;
if(D.isChecked)
features = features | 8;
if(E.isChecked)
features = features | 16;
Store features value in you db.
Loading..
Get the value off you db
if(features&1)
A is checked
if(features&2)
B is checked
if(features&4)
C is checked
if(features&8)
D is checked
if(features&16)
E is checked

Do you know how to work with flags?
For this logical sum method to be reversible it must use the same principle as flags enums: each possible combination of checkboxes must result in a single unique number.
I'm guessing there are not that many checkboxes, numbered like this:
1, 2, 4, 8, 16, 32... etc. This way any combination will result in a single number, and the logical sum will be reversible.

Combine check boxes without using if

I'm making a password generator and I have three check boxes: lowercase, uppercase, and number. Right now I have an IF statement that says:
IF (check1.checked == true & check2.checked == true & check3.checked == true)
{
length = Convert.ToInt32(lengthTextBox.Text);
string password = "";
int choice;
Generate gen = new Generate();
for (int i = 1; i <= length; i++)
{
choice = rnd.Next(1, 4);
if (choice == 1)
{
password = password + gen.lower();
}
else if (choice == 2)
{
password = password + gen.upper();
}
else if (choice == 3)
{
password = password + gen.number();
}
}
passwordTextBox.Text = password;
}
Else IF (check1.checked == true & check2.checked == true)
{
length = Convert.ToInt32(lengthTextBox.Text);
string password = "";
int choice;
Generate gen = new Generate();
for (int i = 1; i <= length; i++)
{
choice = rnd.Next(1, 3);
if (choice == 1)
{
password = password + gen.lower();
}
else if (choice == 2)
{
password = password + gen.upper();
}
}
passwordTextBox.Text = password;
}
and so on. This method is terribly inefficient. I ran into this problem while making a cash register program (10 choices of toppings, check the ones you want, and the program adds the prices together. If you uncheck a topping, the price will be subtracted by that amount.), so I just gave up on it. But now, I am determined to find a way to metaphorically go shopping.

Although it is not really clear what your code does, you seem to be able to call the methods consecutive. So why not do it like this;
if (check1.Checked)
Generate.lowercase();
if (check2.Checked)
Generate.uppercase();
if (check3.Checked)
Generate.number();
Also you should probably want to use more descriptive names for your controls and variables.

if(check1.checked) Generate.lowercase();
if(check2.checked) Generate.uppercase();
...

So what you really need to do here is find a way of making the selection of choice dynamic. One way of doing this is to create a list of functions in which each function is capable of generating a value. At the start you populate the list with the relevant functions based on what's checked, and then you can have a single loop that simply chooses one of the functions from that list:
List<Func<char>> generators = new List<Func<char>>();
if (lowercaseCheckbox.Checked)
generators.Add(() => gen.lowercase());
if (uppercaseCheckbox.Checked)
generators.Add(() => gen.uppercase());
if (numberCheckbox.Checked)
generators.Add(() => gen.number());
int length = Convert.ToInt32(lengthTextBox.Text);
StringBuilder password = new StringBuilder();
Random random = new Random();
for (int i = 0; i < length; i++)
{
int choice = random.Next(generators.Count);
password.Append(generators[choice]());
}
string result = password.ToString();
Note that rather than appending characters to a string repeatedly in a loop you should be using a StringBuilder to avoid constant re-allocation and copying of the values.
I also suggest using more meaningful names for your checkbox variables; it makes the code much more readable.

It depends on what framework you're using and what patterns you have applied, but there are better ways to do this. You could use bit flags and have the check boxes and them together.
FlagVariable output = FlagVariable.None;
if( topping1.IsChecked )
{
output &= FlagVariable.Topping1;
}
if( topping2.IsChecked )
{
output &= FlagVariable.Topping2;
}
// etc...
then later
if( output | FlagVariable.Topping1 )
{
// it has topping 1
}
This is pretty efficient and maintainable. You'll still have to write associative code between the check boxes and their corresponding flag value (this is the part that depends on your framework/patterns, because in WPF with MVVM you might be able to avoid a lot of if-then code).

You could simply split them up:
if (check1.checked) Generate.lowercase();
if (check2.checked) Generate.upperrcase();
if (check3.checked) Generate.number();
Logically, there is no between this way and the grouped way.

In case of password generation #Gerald Versluis 's answer, is definetely worth looking at.
In the case of the shopping, where you have flavours. "Checking" off flavours, to later add the price of the ones that are "checked off", it just doesn't scale very well, as you have already realised.
In that case it's better to add flavours, to a list, which then is calculated to a total.
This can be applied to the password generation as well. Where you add different "Password" generators, thereby letting it scale alot better, and use C# great OOP capabilities. The buzz-word here is: Polymorhism

Best Way to Check for Used Key with Nhibernate?

on my site I allow people to buy subscriptions to my site in bulk(I call them vouchers). Once they have these vouchers, they give them to whoever and they enter that code into their account to upgrade them.
Right now I am thinking of doing 4 alphanumeric code(upper case, lower case and digits) and will have something like this
var chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
var stringChars = new char[4];
var random = new Random();
for (int i = 0; i < stringChars.Length; i++)
{
stringChars[i] = chars[random.Next(chars.Length)];
}
var finalString = new String(stringChars);
For now I think that will give me more than enough combinations and if I ever do run out I can always up the length of the code. I want to keep it short because I don't want the user to have to type in huge as numbers.
I also don't have the time to make a more elegant solution maybe were they click a link or something in their email and it activates their account and of course this would cut down on someone trying to randomly guess a voucher number.
These are things I would deal with if the site every becomes more popular.
I am wondering though how can I handle the possible duplicate generation of the same voucher. My first thought was to check the database each time a voucher is created and if it exists then make a new one.
However that seems like it could be slow. So I thought also maybe getting all the keys first and store them in memory and they check there but if the list keeps growing I might run into out of memory exceptions and all that great stuff.
So does anyone have any ideas? Or am I stuck doing one of the 2 method I listed above?
I am using nhibernate, asp.net mvc and C#.
Edit
static void Main(string[] args)
{
List<string> hold = new List<string>();
for (int i = 0; i < 10000; i++)
{
HashAlgorithm sha = new SHA1CryptoServiceProvider();
byte[] result = sha.ComputeHash(BitConverter.GetBytes(i));
string hex = null;
foreach (byte x in result)
{
hex += String.Format("{0:x2}", x);
}
hold.Add(hex.Substring(0,3));
Console.WriteLine(hex.Substring(0, 4));
}
Console.WriteLine("Number of Distinct values {0}", hold.Distinct().Count());
}
above is my attempt to try to use hashing. However I think I am missing something as it seems to have quite a bit more duplicates then expected.
Edit 2
I think I added what I was missing but not sure if this is exactly what he meant. I am also not sure what to do in a situation when I moved it as far as I can move it(my has seems to give me a length of 40 places I can move it).
static void Main(string[] args)
{
int subStringLength = 4;
List<string> hold = new List<string>();
for (int i = 0; i < 10000; i++)
{
SHA1CryptoServiceProvider sha = new SHA1CryptoServiceProvider();
byte[] result = sha.ComputeHash(BitConverter.GetBytes(i));
string hex = null;
foreach (byte x in result)
{
hex += String.Format("{0:x2}", x);
}
int startingPositon = 0;
string possibleVoucherCode = hex.Substring(startingPositon,subStringLength);
string voucherCode = Move(subStringLength, hold, hex, startingPositon, possibleVoucherCode);
hold.Add(voucherCode);
}
Console.WriteLine("Number of Distinct values {0}", hold.Distinct().Count());
}
private static string Move(int subStringLength, List<string> hold, string hex, int startingPositon, string possibleVoucherCode)
{
if (hold.Contains(possibleVoucherCode))
{
int newPosition = startingPositon + 1;
if (newPosition <= hex.Length)
{
if ((newPosition + subStringLength) > hex.Length)
{
possibleVoucherCode = hex.Substring(newPosition, subStringLength);
return Move(subStringLength, hold, hex, newPosition, possibleVoucherCode);
}
// return something
return "0";
}
else
{
// return something
return "0";
}
}
else
{
return possibleVoucherCode;
}
}
}

It is going to be slow because you want to generate the vouchers randomly and then check the database for every generated code.
I would create a table vouchers with an id, the code and an is_used column. I would fill that table once with enough random codes. Since this can be done in a separate process, the performance won't be such a big problem. Let it run in the evening and the next day you get a fully filled vouchers-table.
If you want to prevent generating duplicate vouchers, that won't be a problem. You can generate them anyway and put them either in a System.Collections.Generic.HashSet (which prevents adding duplicates without throwing an exception) or call the Linq-method Distinct(), before adding them to that vouchers table.

If you insist on short codes:
Use a GUID as a primary key, generate one random number. How you might want to translate this in to alpha-num is up to you.
Use the last byte or two of the guid and the random number. 1234-684687 This should make it slightly less easy to bruteforce coupons. And handle any (rare) collisions with an exception.
Easy way to shorten an int, change it's base (from 10 to 62). (in VB, and this is old code)
This yields "2lkCB1" when given Int32.MaxValue
''//given intValue as your random integer
Dim result As String = String.Empty
Dim digits as String = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
Dim x As Integer
While (intValue > 0)
x = intValue Mod digits.Length
result = digits(x) & result
intValue = intValue - x
intValue = intValue \ digits.Length
End While
Return result
But now we're already answering more than one question.

For a bulk data operation like this, I would recommend not using NHibernate and just doing straight ADO.NET.
Batch Check
Since you anticipate generating big batches of codes at once, you should batch multiple code checks into a single round-trip to the database. If you're using SQL Server 2008 or higher, you could do this using table-valued parameters, checking a whole list of codes at once.
SELECT DISTINCT b.Code
FROM #batch b
WHERE NOT EXISTS (
SELECT v.Code
FROM dbo.Voucher v
WHERE v.Code = b.Code
);
Concurrency
Now, what about concurrency issues? What if two users generate the same code at roughly the same time? Or simply in-between the time when we check the code for uniqueness and when we insert it into the Voucher table?
We can take care of that by modifying the query as follows:
DECLARE #batchid uniqueidentifier;
SET #batchid = NEWID();
INSERT INTO dbo.Voucher (Code, BatchId)
SELECT DISTINCT b.Code, #batchid
FROM #batch b
WHERE NOT EXISTS (
SELECT Code
FROM dbo.Voucher v
WHERE b.Code = v.Code
);
SELECT Code
FROM dbo.Voucher
WHERE BatchId = #batchid;
Executing via .NET
Assuming that you have defined the following table-valued user type...
CREATE TYPE dbo.VoucherCodeList AS TABLE (
Code nvarchar(8) COLLATE SQL_Latin1_General_CP1_CS_AS NOT NULL
/* !!! Remember to specify the collation on your Voucher.Code column too, since you want upper and lower-case codes. */
);
... you could execute this query via .NET code like this:
public ICollection<string> GenerateCodes(int numberOfCodes)
{
var result = new List<string>(numberOfCodes);
while (result.Count < numberOfCodes)
{
var batchSize = Math.Min(_batchSize, numberOfCodes - result.Count);
var batch = Enumerable.Range(0, batchSize)
.Select(x => GenerateRandomCode());
var oldResultCount = result.Count;
result.AddRange(FilterAndSecureBatch(batch));
var filteredBatchSize = result.Count - oldResultCount;
var collisionRatio = ((double)batchSize - filteredBatchSize) / batchSize;
// Automatically increment length of random codes if collisions begin happening too frequently
if (collisionRatio > _collisionThreshold)
CodeLength++;
}
return result;
}
private IEnumerable<string> FilterAndSecureBatch(IEnumerable<string> batch)
{
using (var command = _connection.CreateCommand())
{
command.CommandText = _sqlQuery; // the concurrency-safe query listed above
var metaData = new[] { new SqlMetaData("Code", SqlDbType.NVarChar, 8) };
var param = command.Parameters.Add("#batch", SqlDbType.Structured);
param.TypeName = "dbo.VoucherCodeList";
param.Value = batch.Select(x =>
{
var record = new SqlDataRecord(metaData);
record.SetString(0, x);
return record;
});
using (var reader = command.ExecuteReader())
while (reader.Read())
yield return reader.GetString(0);
}
}
Performance
After implementing all of this (and moving the command and parameter creation out of the loop so it would be re-used between batches), I was able to insert 10,000 codes using a batch size of 500 consistently in approx. 0.5 to 2 seconds, or 5 to 20 codes per millisecond.
Code Density / Collisions / Guessability
The _collisionThreshold field limits the density of your codes. It's a value between 0 and 1. Actually, it must be less than 1 or else you would wind up in an infinite loop when the 4 digit codes were exhausted (probably should add an assertion for this in code). I would recommend never turning it above 0.5 for performance reasons. More than 50% collisions would mean it's spending more time testing already-used codes than actually generating new ones.
Keeping the collision threshold low is how you would control how hard-to-guess your codes are. Setting _collisionThreshold to 0.01 would generate codes such that there's approximately a 1% chance of someone guessing a code.
If collisions occur too frequently, CodeLength (which is used by the GenerateRandomCode() method) will be incremented. This value needs to be persisted somewhere. After executing GenerateCodes(), check CodeLength to see if it has changed and then save the new value.
Source Code
The full code is available here: https://gist.github.com/3217856. I am the author of this code, and am releasing it under the MIT license. I had fun with this little challenge, and also got to learn how to pass a table-valued parameter to an inline parametrized query. I hadn't ever done that before. I've only ever passed them to full-fledged stored procedures.

A possible solution for you is like this:
Find the maximum ID of a voucher (an integer). Then, run any hash function on it, take the first 32 bits and convert to the string you want to show the user (or use a 32bit hash function such as Jenkins hash function). This will probably work, hash collisions are pretty rare. But this solution is very similar to yours, in the point of randomness.
You could run a test which finds the first 10 or 100 collisions (this should be enough for you) and forces the algorithm to "skip" them and use a different starting value. Then, you don't need to check the database at all (well, at least until you reach about 4294967296 vouchers...)

how about utilizing nHibernate's HiLo algorithm?
Here is an example on how you can get the next value (without DB access).

datarow values comparision with c#

I have a sql database with a table that contains my grading scales and comment e.g
debut end comment
5 ---- 10 -- x
0 ---- 4 --- y
I have managed to iterate through the rows of my table with a foreach loop.
I want to supply a value, maybe with a text box control, then the program should check the range in my gradingScale table where the value follows and outputs a corresponding comment
for example
int number;
number=4
comment=y;

Not sure what you're looking for - and you didn't mention what database you're using - so here I'm just guess that you might be looking for something like this:
DECLARE #Number INT
SET #Number = 4
SELECT comment
FROM dbo.gradingScale
WHERE #Number BETWEEN debut AND end
Of course, you could also wrap this inside a stored procedure (if your database supports that):
CREATE PROCEDURE dbo.GetComment (#Number INT)
AS
SELECT comment
FROM dbo.gradingScale
WHERE #Number BETWEEN debut AND end
These code samples are for Microsoft SQL Server 2005 and up (T-SQL).

If I understand correctly, your database list ranges, each associated with a comment. In your example 0 to 4 map to x, while 5 to 10 map to y.
In that case, a very simple approach would be, assuming that your ranges are not overlapping, to sort your table by ascending debut, and then iterate over the rows until you find one which start is <= to your value.

Hard to make out what you want exactly, but here's an example implementation (You don't say how or in what form your Sql results are returned, so I've provided a DTO/List implementation:
static void SO6648999()
{
List<test> sample = new List<test>
{
new test { debut = 0,
end = 4,
comment = "y"},
new test { debut = 5,
end = 10,
comment = "x"}
};
int number = 4;
string comment = sample.Single(x => number >= x.debut && number <= x.end).comment;
}
class test
{
public int debut;
public int end;
public string comment;
}

I believe you are referring to DataTable
You can use a Select on the DataTable and filter out the records by providing an expression. It works similar to a where clause in Sql.
dt1.Select("end = 4")// assuming column holding int value
end is the column name of the value you are searching and this will return the datarow (array) satisfying the condition.