Develop Reference

Combine multiple realm query results .net

As a Contains implementation, I am using a bit tweaked method written by Andy Dent to query my realm database:
private IQueryable<Entry> FilterEntriesByIds(IQueryable<Entry> allEntries, int[] idsToMatch)
{
// Fancy way to invent Contains<>() for LINQ
ParameterExpression pe = Expression.Parameter(typeof(Entry), "Entry");
Expression chainedByOr = null;
Expression left = Expression.Property(pe, typeof(Entry).GetProperty("Id"));
for (int i = 0; i < idsToMatch.Count(); i++) {
Expression right = Expression.Constant(idsToMatch[i]);
Expression anotherEqual = Expression.Equal(left, right);
if (chainedByOr == null)
chainedByOr = anotherEqual;
else
chainedByOr = Expression.OrElse(chainedByOr, anotherEqual);
}
MethodCallExpression whereCallExpression = Expression.Call(
typeof(Queryable),
"Where",
new Type[] { allEntries.ElementType },
allEntries.Expression,
Expression.Lambda<Func<Entry, bool>>(chainedByOr, new ParameterExpression[] { pe }));
return allEntries.Provider.CreateQuery<Entry>(whereCallExpression);
}
It all works just fine as long as I pass less than 2-3K of ids, when I go with larger amounts the app just crashes with what seems to be a stackoverflow exception.
My first thought to solve this was to break the query into chunks and later combine the results, but the Concat and Union methods do not work on these realm IQueryables, so, how else can I merge such chunked results? Or is there any other workaround?
I can't just convert the results to a List or something and then merge, I have to return realm objects as IQueryable<>
The call stack:
=================================================================
Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
No native Android stacktrace (see debuggerd output).
=================================================================
Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7c326075c8):0x7c326075b8 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x7c326075c8 fd 7b bb a9 fd 03 00 91 a0 0b 00 f9 10 0e 87 d2 .{..............
0x7c326075d8 10 d0 a7 f2 90 0f c0 f2 b0 0f 00 f9 10 00 9e d2 ................
0x7c326075e8 f0 d5 a9 f2 90 0f c0 f2 b0 13 00 f9 a0 a3 00 91 ................
=================================================================
Managed Stacktrace:
============================
=====================================
at Realms.QueryHandle:GroupBegin <0x00000>
at Realms.RealmResultsVisitor:VisitCombination <0x0007f>
at Realms.RealmResultsVisitor:VisitBinary <0x003f7>
at System.Linq.Expressions.BinaryExpression:Accept <0x00073>
at System.Linq.Expressions.ExpressionVisitor:Visit <0x00087>
at Realms.RealmResultsVisitor:VisitCombination <0x000d7>
at Realms.RealmResultsVisitor:VisitBinary <0x003f7>
at System.Linq.Expressions.BinaryExpression:Accept <0x00073>
at System.Linq.Expressions.ExpressionVisitor:Visit <0x00087>
at Realms.RealmResultsVisitor:VisitCombination <0x000d7>
at Realms.RealmResultsVisitor:VisitBinary <0x003f7>
at System.Linq.Expressions.BinaryExpression:Accept <0x00073>
at System.Linq.Expressions.ExpressionVisitor:Visit <0x00087>
at Realms.RealmResultsVisitor:VisitCombination <0x000d7>
at Realms.RealmResultsVisitor:VisitBinary <0x003f7>
at System.Linq.Expressions.BinaryExpression:Accept <0x00073>
at System.Linq.Expressions.Ex
pressionVisitor:Visit <0x00087>
UPD
I found the exact number of elements after which this error gets thrown: 3939, if I pass anything larger than that, it crashes.

I'm not a database expert, just have worked at varying levels (inc Realm) usually on top of someone's lower-level engine. (The c-tree Plus ISAM engine or the C++ core by the real wizards at Realm).
My first impression is that you have mostly-static data and so this is a fairly classical problem where you want a better index generated up front.
I think you can build such an index using Realm but with a bit more application logic.
It sounds a bit like an Inverted Index problem as in this SO question.
For all the single and probably at least double-word combinations which map to your words, you want another table that links to all the matching words. You can create those with a fairly easy loop that creates them from the existing words.
eg: your OneLetter table would have an entry for a that used a one-to-many relationship to all the matching words in the main Words table.
That will delivery a very fast Ilist you can iterate of the matching words.
Then you can flip to your Contains approach at 3 letters or above.

The SQL provider for SQL Server can handle a contains operation with more items than is the limit for maximum number of SQL parameters in one query, which is 2100.
This sample is working:
var ids = new List<int>();
for (int i = 0; i < 10000; i++)
{
ids.Add(i);
}
var testQuery = dbContext.Entity.Where(x => ids.Contains(x.Id)).ToList();
It means, you could try to rewrite your method to use ids.Contains(x.Id).
An example how to do it is in this post.
UPDATE: sorry did not mention this is of Realm, but perhaps it is still worth trying.

C# Translating a few smaller arrays to one big

Title may not explain fully what I want to do, so I made an image.
You can see there are 4 1D arrays(red numbers, black colored numbers are indexes), each of this array goes from 0 to 63. I want to somehow translate them, that for example, index 16 will point to first index of second array.
What I was thinking of, is a function where I give List of arrays and index that I want to get as input, and its returns me the index of array and exact index in this array as output.
I would like to have some hints or suggestions on how to proceed here to achieve this functionality.

Ok, your image shows an interleaved data of 16 elements, so you want to have (showing an example of only two matrices because of space :D)
Global index
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
-----------------------------------------------------------------------------------------------------
Array0 indexes - Array1 indexes
0 1 2 3 4 5 6 7 8 9 A B C D E F - 0 1 2 3 4 5 6 7 8 9 A B C D E F
-------------------------------------------------------------------------------------------------
Global index
33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65
-----------------------------------------------------------------------------------------------------
Array0 indexes - Array1 indexes
10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F - 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F
To get it you can do something like this:
public class CompositeIndex
{
public int ArrayNumber { get; set; }
public int ElementNumber { get; set; }
}
public static CompositeIndex GetIndex(int GlobalIndex, int ArrayCount, int ElementsToInterleave)
{
CompositeIndex index = new CompositeIndex();
int fullArrays = GlobalIndex / ElementsToInterleave; //In your example: 16 / 16 = 1;
index.ArrayNumber = fullArrays % ArrayCount; //In your example: 1 mod 4 = 1;
index.ElementNumber = GlobalIndex - (fullArrays * ElementsToInterleave); //In your example: 16 - (1 * 16) = 0;
return index;
}
Then, if you have 4 matrices and want to get the "global" index 16 you do:
var index = GetIndex(16, 4, 16);
This function allows you to use an indeterminated number of arrays and interleaved elements.
BTW, another time ask better your question, a lot more people will help you if they don't have to solve a puzzles to understand what you want...

Spliting into lines then into variables

i have this text file below:
001 Bulbasaur 45 49 49 65 65 45 Grass Poison
002 Ivysaur 60 62 63 80 80 60 Grass Poison
003 Venusaur 80 82 83 100 100 80 Grass Poison
004 Charmander 39 52 43 60 50 65 Fire
005 Charmeleon 58 64 58 80 65 80 Fire
I have written this piece of code to split it into lines then into variables but it refuses to work, my apology's if i am asking this in the wrong place. (unity C# question).
var lines = textFile.text.Split("\n"[0]);
allMonsters = new Monsters[lines.Length];
List<string> lineSplit = new List<string>();
for (int i = 0; i < lines.Length; i++) {
Debug.Log(lines[i]);
lineSplit.Clear();
lineSplit = lines[i].Split(' ').ToList ();
int ID = int.Parse(lineSplit[0]);
string Name = lineSplit[1].ToString();
float HP = float.Parse(lineSplit[2]);
float ATK = float.Parse(lineSplit[3]);
float DEF = float.Parse(lineSplit[4]);
float SPATK = float.Parse(lineSplit[5]);
float SpDEF = float.Parse(lineSplit[6]);
float speed = float.Parse(lineSplit[7]);
FirstType Ft = (FirstType)System.Enum.Parse(typeof(FirstType),lineSplit[8]);
SecondType ST = (SecondType)System.Enum.Parse(typeof(SecondType),lineSplit[9]); }
The code works for the first line but then on the second run of this code i get null reference to an object error, please help me.
Note, variables are assigned to so they aren't overwritten after code.
EDIT: LineSplit variable is over 1200 elements long, so i do not think unity is clearing the array properly could this be the issue?

I'm not sure about the second iteration, but on the 4th iteration line 004 Charmander 39 52 43 60 50 65 Fire have only 9 parameters (by split of space) and you using lineSplit[9], so there you will get NullPointerException.

When working with text files, linefeed often comes along with carriage return.
Try
var lines = textFile.text.Split(new string[]{"\r\n"}, System.StringSplitOptions.None);
in the first line.

Firmata over nRF24

I'm having some technical problems... I'm trying to use Firmata for arduino but over nrf24, not over Serial interface. I have tested nRF24 communication and it's fine. I have also tested Firmata over Serial and it works.
Base device is simple "serial relay". When it has data available on Serial, read it and send it over nRF24 network. If there is data available from network, read it and send it through Serial.
Node device is a bit complex. It has custom Standard Firmata where I have just added write and read override.
Read override id handeled in loop method in this way:
while(Firmata.available())
Firmata.processInput();
// Handle network data and send it to Firmata process method
while(network.available()) {
RF24NetworkHeader header;
uint8_t data;
network.read(header, &data, sizeof(uint8_t));
Serial.print(data, DEC); Serial.print(" ");
Firmata.processInputOverride(data);
BlinkOnBoard(50);
}
currentMillis = millis();
Firmata processInputOverrride is little changed method of processInput where processInput reads data directly from FirmataSerial, and in this method we pass data down to method from network. This was tested and it should work fine.
Write method is overloaded in a different way. In Firmata.cpp I have added an method pointer that can be set to a custom method and used to send data using that custom method. I have then added custom method call after each of the FirmataSerial.write() call:
Firmata.h
...
size_t (*firmataSerialWriteOverride)(uint8_t);
...
void FirmataClass::printVersion(void) {
FirmataSerial.write(REPORT_VERSION);
FirmataSerial.write(FIRMATA_MAJOR_VERSION);
FirmataSerial.write(FIRMATA_MINOR_VERSION);
Firmata.firmataSerialWriteOverride(REPORT_VERSION);
Firmata.firmataSerialWriteOverride(FIRMATA_MAJOR_VERSION);
Firmata.firmataSerialWriteOverride(FIRMATA_MINOR_VERSION);
}
I have then set the overrided write method to a custom method that just writes byte to network instead of Serial.
size_t ssignal(uint8_t data) {
RF24NetworkHeader header(BaseDevice);
network.write(header, &data, sizeof(uint8_t));
}
void setup() {
...
Firmata.firmataSerialWriteOverride = ssignal;
...
}
Everything seems to be working fine, it's just that some data seems to be inverted or something. I'm using sharpduino (C#) to do some simple digital pin toggle. Here's how output looks like: (< came from BASE, > sent to BASE)
> 208 0
> 209 0
...
> 223 0
> 249
< 4 2 249
and here communication stops...
That last line came inverted. So i tough that I only need to invert received bytes. And it worked for that first command. But then something happens and communication stops again.
> 208 0
> 209 0
...
> 223 0
> 249 // Report firmware version request
< 249 2 4
> 240 121 247 // 240 is sysex begin and 247 is systex end
< 240 121
< 101 0 67 0 0 1 69 0 118
< 117 0 115 0
< 0 70 0 105 0 116 0 111 0 109
< 0 97 0
< 0 109
< 116 0 97 0 247
> 240 107 247
So what could be the problem here? It seems that communication with Firmata works but something isn't right...
-- EDIT --
I solved that issue. The problem was that I didn't see Serial.write() calls in sysex callback. Now that that is solved, I came up to another problem... All stages pass right (I guess) and then I dont get any response from Node when I request pin states
...
< f0 6a 7f 7f 7f ... 7f 0 1 2 3 4 5 6 7 8 9 a b c d e f f7 // analog mapping
> f0 6d 0 f7 // sysex request pin 0 state and value
> f0 6d 1 f7
> f0 6d 2 f7
...
> f0 6d 45 f7
// And I wait for response...
There is no response. Any ideas why would that happen? Node receive all messages correctly and code for handling pin states exist.

Porting libmfcc to C# using Bass Library

I am currently using The Bass Library for Audio Analysis which can calculate FFT and return it as an Array, libmfcc uses this Data to calculate the Value of the MFCC Coefficients which I need. (Info: MFCC is like a Audio Spectrum but it fits more the way how the Human Hearing and Frequency Scaling works)
The Bass Library returns Values from 0 to 1 as FFT Values.
Now I encountered several Problems and Questions:
Their FFT Example Data seems to have a different Format, Values are very high and the total of the 8192 FFT Values Sum to 10739.24 , how can that be?[/li]
In their example Application they call the Function like the following. Why they Use 128 as FFT Array Size if they just loaded 8192 Values?
Using their MFCC Class which I copied and edited a bit to match C# Syntax/Functions I get negative Values for some Coefficients, I dont think that should be the case.
Can anyone help me out why it is returning negative Values or what I did wrong ?
I made a simple example Ready to Try Program which does the described above and is useful for debugging.
Link: http://www.xup.in/dl,17603935/MFCC_Test.rar/
Output from my C# Application (Most likely not correct)
Coeff 16 = 0,017919318626506 Coeff 17 = -0,155580763009355 Coeff 18 =
-0,76072865841987 Coeff 19 = 0,108961510335727 Coeff 20 = 0,819025783804398 Coeff 21 = -0,660508603974514 Coeff 22 =
-0,951623924906163 Coeff 23 = 0,424922129906254 Coeff 24 = 0,0129727009313168 Coeff 25 = -0,388796833267654 Coeff 26 =
0,270839393161931 Coeff 27 = -0,138515788828431 Coeff 28 =
-0,454837674981149 Coeff 29 = -0,448629344922371 Coeff 30 = -0,11908663618393 Coeff 31 = 0,237500036702818 Coeff 32 = 0,114874386870208 Coeff 33 = -0,100822381384326 Coeff 34 =
0,144242143551012 Coeff 35 = 0,209338502838453 Coeff 36 =
0,247588420953066 Coeff 37 = -0,451654204112441 Coeff 38 =
0,0346927542067229 Coeff 39 = 0,180816031061584
Their example FFT Data (Different Format?)
14.524506
38.176063
10.673860
3.705076
2.102398
1.461585
1.145616
0.974108
0.878079
0.825304
0.798959
0.789067
0.789914
0.797102
0.808576
0.822048
0.836592
0.851101
0.864869
0.877625
0.888780
0.897852
0.905033
0.910054
0.912214
0.912414
0.909593
0.904497

I can answer the first part:
The sample code clearly states that the input data was computed using FFTW, which produces an unnormalized result. You need to divide by sqrt(n) to get the normalized values, which is what I suspect BASS returns.
Perhaps multiplying your inputs by sqrt(n) will give you better results.

The MFCC routine returns cepstral coefficients (DCT of the log of mel magnitudes), not mel magnitude values. Cepstral coefficients can be negative. I believe the value 128 in the example code is indeed a mistake by the author. In order to preserve the signal energy an FFT requires normalization at some point (either after FFT, iFFT or split between the two). In the example you're looking at the raw (unnormalized) magnitudes, which is why they are so large.