What would be the best approach to allow users to define a WHERE-like constraints on objects which are defined like this:
Collection<object[]> data
Collection<string> columnNames
where object[] is a single row.
I was thinking about dynamically creating a strong-typed wrapper and just using Dynamic LINQ but maybe there is a simpler solution?
DataSet's are not really an option since the collections are rather huge (40,000+ records) and I don't want to create DataTable and populate it every time I run a query.
What kind of queries do you need to run? If it's just equality, that's relatively easy:
public static IEnumerable<object[]> WhereEqual(
this IEnumerable<object[]> source,
Collection<string> columnNames,
string column,
object value)
{
int columnIndex = columnNames.IndexOf(column);
if (columnIndex == -1)
{
throw new ArgumentException();
}
return source.Where(row => Object.Equals(row[columnIndex], value);
}
If you need something more complicated, please give us an example of what you'd like to be able to write.
If I get your point : you'd like to support users writting the where clause externally - I mean users are real users and not developers so you seek solution for the uicontrol, code where condition bridge. I just though this because you mentioned dlinq.
So if I'm correct what you want to do is really :
give the user the ability to use column names
give the ability to describe a bool function (which will serve as where criteria)
compose the query dynamically and run
For this task let me propose : Rules from the System.Workflow.Activities.Rules namespace. For rules there're several designers available not to mention the ones shipped with Visual Studio (for the web that's another question, but there're several ones for that too).I'd start with Rules without workflow then examine examples from msdn. It's a very flexible and customizable engine.
One other thing: LINQ has connection to this problem as a function returning IQueryable can defer query execution, you can previously define a query and in another part of the code one can extend the returned queryable based on the user's condition (which then can be sticked with extension methods).
When just using object, LINQ isn't really going to help you very much... is it worth the pain? And Dynamic LINQ is certainly overkill. What is the expected way of using this? I can think of a few ways of adding basic Where operations.... but I'm not sure how helpful it would be.
How about embedding something like IronPython in your project? We use that to allow users to define their own expressions (filters and otherwise) inside a sandbox.
I'm thinking about something like this:
((col1 = "abc") or (col2 = "xyz")) and (col3 = "123")
Ultimately it would be nice to have support for LIKE operator with % wildcard.
Thank you all guys - I've finally found it. It's called NQuery and it's available from CodePlex. In its documentation there is even an example which contains a binding to my very structure - list of column names + list of object[]. Plus fully functional SQL query engine.
Just perfect.
Related
We are investigating using LinQ to query an internal dynamic collection created by Dapper. The question is:
How to execute dynamic LinQ against the collection using Scott Guthrie dynamic linq (or some other technology if possible)? (http://weblogs.asp.net/scottgu/dynamic-linq-part-1-using-the-linq-dynamic-query-library)
This is what we want to do (much simplified):
Use Dapper to return a dynamic collection (here called rows):
rows = conn.Query(“select ACCOUNT, UNIT, AMOUNT from myTable”);
If we use a “static” LinQ query there is no problem. So this works fine:
var result = rows.Where(w => w.AMOUNT > 0);
But we would like to write something similar to this using dynamic Linq:
var result = rows.Where("AMOUNT > 0");
But we can’t get this to work.
The error we get is:
No property or field ‘AMOUNT’ exists in type ‘Object’
(We have tried a lot of other syntax also – but cant get it to work)
Please note: We do not want to use dynamic SQL when Dapper requests data from the database (that is easy). We want to execute many small dynamic Linq statements on the collection that the Dapper query returns.
Can it be that ScottGu dynamic Linq only works with ‘LinQ to SQL’?
Is there some other alternative approach to achieve the same thing?
(Performance is a key issue)
/Erik
conn.Query("...")
returns an IEnumerable<dynamic>. However, "dynamic LINQ" pre-dates dynamic, and presumably nobody has updated it to work with dynamic; it could certainly be done, but it is work (and isn't trivial). Options:
use Query<T> for some T
make the required changes to "dynamic LINQ", and preferably make those changes available to the wider community
I suspect that conn.Query(“select ACCOUNT, UNIT, AMOUNT from myTable”); returns IEnumerable<object>. In order for DLinq to work, you need to have IEnumerable<TheActualType>.
You can try this:
conn.Query<dynamic>("yourQueryString")
.ToList()
//.ToAnonymousList()
.Where("AMOUNT > 0");
If that doesn't work, you could try and use ToAnonymousList, that tries to return the IList<TheActualType.
I am in the process of prototyping an implementation of a rules engine to help us with our ordering portals. For example giving discounts on items or requiring approval if certain items are ordered. We would also like to be able to add rules for dollar amounts, user hierarchy positions, and be apply it one client or more.
I feel that WWF is a good answer to this need.
All of that said however I am having a little difficulty figuring out how best to set up some of the more complex rules. I have a "condition" that I feel is best described in a LINQ query, like so:
var y = from ol in currentOrder.OrderLines where ol.ItemCode == "MYITEMCODE" select ol;
I am not against using a different framework for a rules engine or adding additional properties/methods to our objects (ex: OrderHasItem(ItemCode), etc) to make these rules more simplified but I would like to avoid having to do that. It feels self-defeating in that it forces us down the road of potentially requiring code changes for new rules.
Yes, you can use Linq queries with Workflow. In WF what you are referring to as a rule is an expression that is evaluated at runtime. Your query is selecting a subset of the orderliness based on a criteria.
For example, if I have a collection of names and I want to see only names that begin with 'R'. I could write the following code.
private static void ShowQueryWithCode(IEnumerable<string> names)
{
Console.WriteLine("LINQ Query in Code - show names that start with 'R'");
// Assuming there are no null entries in the names collection
var query = from name in names where name.StartsWith("R") select name;
// This is the same thing as
// var query = names.Where(name => name.StartsWith("R"));
foreach (var name in query)
{
Console.WriteLine(name);
}
Console.WriteLine();
}
To do the same with Workflow
Step 1: Create a Workflow with an In Argument of type IEnumerable
Here you can see that I've added the in argument
Step 2: Add a Variable for the query of type IEnumerable
Before you can add a variable you need to include an activity which has variables. In this workflow I've added a sequence.
Step 3: Assign the query the LINQ expression you want to use
You can use a method chain or query syntax.
Step 4: Iterate over the collection
In the completed workflow I used a ForEach activity to iterate the list of names and write them to the console.
This example uses C# in .NET 4.5 but the same technique can be used with Visual Basic.
You can download the sample code here
WF is a workflow engine. It's used to run factories and banks. The rule engine is just a small part of it. To make sense, any normal WF project requires a dedicated team of professionals to build it. Seems like an overkill for your specific purpose. You are very likely to bury yourself and your project in a typical fight between requirements and real skills of your team.
The use of any available .NET engine would be more justified in your situation. Keep in mind that building a custom rule engine is not an easy tasks, no matter how simple it seems from the beginning. Setting property values of a class (typically called a "fact" or "source" object) or executing actions (invoking class' methods) is what rules engines do best. And it seems that it's exactly what you need. Check out some available .NET engines. They are inexpensive, if not free, and reliable.
We are still using .Net Framework 2.0 / VS 2005 so i do not have LINQ. If i don't want to go with the poor man's LINQ solution, what are some other alternatives for being able to query in memory custom objects in a dictionary?
I'm not sure if one of your poor man's LINQ solution is LINQBridge but I used it for a few weeks and it seemed to be working okay before we actually switched to .NET 3.5 and the real deal.
Dictionary<T> would seem like a good choice, although you haven't provided much information about what you mean by "query." Are you just looking to retrieve data based on some key value? Get a count of total items? Do a sum or average based on some condition? You really need to give more information to get a better answer.
To elaborate on what Chesso said, you'll have to iterate the loop just like LINQ does...
for example:
static T FindFirst<T>(IEnumerable<T> col, Predicate<T> predicate)
{
foreach(T t in col)
{
if(predicate(t))
{
return t;
}
}
return default(T);
}
I was not aware of the Predicate delegate, that seems to be pretty much what i was looking for. As far as the context for which i'm querying:
Say i have a object X with properties A (name, guaranteed to be unique) and B (age)
1) I have a series of objects in a dictionary whose keys are say Property A of a given object, and of course the value is the object iself.
Now i want to retrieve all objects in this dictionary which meet a certain criteria of B, say age > 20.
I can add all the values of the dictionary into a list then call the .FindAll on it, passing in a delegate. I can create an anonymous delegate to do this, but say i will reuse this many times. How can i dynamically specify an age criteria for the delegate method? Would the only choice be to encapsulate the Predicate method in a class, then create a new instance of that class with my criteria as an instance variable?
For a utility I'm working on, the client would like to be able to generate graphic reports on the data that has been collected. I can already generate a couple canned graphs (using ZedGraph, which is a very nice library); however, the utility would be much more flexible if the graphs were more programmable or configurable by the end-user.
TLDR version
I want users to be able to use something like SQL to safely extract and select data from a List of objects that I provide and can describe. What free tools or libraries will help me accomplish this?
Full version
I've given thought to using IronPython, IronRuby, and LuaInterface, but frankly they're all a bit overpowered for what I want to do. My classes are fairly simple, along the lines of:
class Person:
string Name;
int HeightInCm;
DateTime BirthDate;
Weight[] WeighIns;
class Weight:
int WeightInKg;
DateTime Date;
Person Owner;
(exact classes have been changed to protect the innocent).
To come up with the data for the graph, the user will choose whether it's a bar graph, scatter plot, etc., and then to actually obtain the data, I would like to obtain some kind of List from the user simply entering something SQL-ish along the lines of
SELECT Name, AVG(WeighIns) FROM People
SELECT WeightInKg, Owner.HeightInCm FROM Weights
And as a bonus, it would be nice if you could actually do operations as well:
SELECT WeightInKg, (Date - Owner.BirthDate) AS Age FROM Weights
The DSL doesn't have to be compliant SQL in any way; it doesn't even have to resemble SQL, but I can't think of a more efficient descriptive language for the task.
I'm fine filling in blanks; I don't expect a library to do everything for me. What I would expect to exist (but haven't been able to find in any way, shape, or form) is something like Fluent NHibernate (which I am already using in the project) where I can declare a mapping, something like
var personRequest = Request<Person>();
personRequest.Item("Name", (p => p.Name));
personRequest.Item("HeightInCm", (p => p.HeightInCm));
personRequest.Item("HeightInInches", (p => p.HeightInCm * CM_TO_INCHES));
// ...
var weightRequest = Request<Weight>();
weightRequest.Item("Owner", (w => w.Owner), personRequest); // Indicate a chain to personRequest
// ...
var people = Table<Person>("People", GetPeopleFromDatabase());
var weights = Table<Weight>("Weights", GetWeightsFromDatabase());
// ...
TryRunQuery(userInputQuery);
LINQ is so close to what I want to do, but AFAIK there's no way to sandbox it. I don't want to expose any unnecessary functionality to the end user; meaning I don't want the user to be able to send in and process:
from p in people select (p => { System.IO.File.Delete("C:\\something\\important"); return p.Name })
So does anyone know of any free .NET libraries that allow something like what I've described above? Or is there some way to sandbox LINQ? cs-script is close too, but it doesn't seem to offer sandboxing yet either. I'd be hesitant to expose the NHibernate interface either, as the user should have a read-only view of the data at this point in the usage.
I'm using C# 3.5, and pure .NET solutions would be preferred.
The bottom line is that I'm really trying to avoid writing my own parser for a subset of SQL that would only apply to this single project.
There is a way to sandbox LINQ or even C#: A sandboxed appdomain. I would recommend you look into accepting and compiling LINQ in a locked-down domain.
Regarding NHibernate, perhaps you can pass the objects into the domain without exposing NHibernate at all (I don't know how NHibernate works). If this is not possible, perhaps the connection to the database used within the sandbox can be logged in as a user who is granted only SELECT permissions.
Maybe the expressions will come handy for You.
You could provide simple entry places for:
a) what to select - user is expected to enter an expression only _ probably member and arithmetic expressions - those are subclasses of the expression class
b) how to filter the things = again only expressions are expected
c) ordering
d) joining?
Expressions don't let You do File.Delete because You operate only on precise domain objects (which probably don't have this functionality). The only thing You have to check is whether the parameters of the said expressions are of Your domain types. and Return types of said expressions are of domain types (or generic types in case of IEnumerable<> or IQuerable<>
this might prove helpful
I.E. expressions don't let You write multi-line statements.
Then You build your method chain in code
and voila.
There comes the data
I ended up using a little bit of a different approach. Instead of letting users pick arbitrary fields and make arbitrary graphs, I'm still presenting canned graphs, but I'm using Flee to let the user filter out exactly what data is used in the source of the graph. This works out nicely, because I ended up making a set of mappings from variable names to "accessors", and then using those mappings to inject variables into the user-entered filters. It ended up something like:
List<Mapping<Person>> mappings;
// ...
mappings.Add(new Mapping("Weight", p => p.Weight, "The person's weight (in pounds)"));
// ...
foreach (var m in mappings)
{
context.Variables[m.Name] = m.Accessor(p);
}
// ...
And you can even give an expression context an "owner" (think Ruby's instance_eval, where the context is executed with score of the specified object as this); then the user can even enter a filter like Weight > InputNum("The minimum weight to see"), and then they will be prompted thusly when the filter is executed, because I've defined a method InputNum in the owning class.
I feel like it was a good balance between effort involved and end result. I would recommend Flee to anyone who has a need to parse simple statements, especially if you need to extend those statements with your own variables and functions as well.
Ok, understand that I come from Cold Fusion so I tend to think of things in a CF sort of way, and C# and CF are as different as can be in general approach.
So the problem is: I want to pull a "table" (thats how I think of it) of data from a SQL database via LINQ and then I want to do some computations on it in memory. This "table" contains 6 or 7 values of a couple different types.
Right now, my solution is that I do the LINQ query using a Generic List of a custom Type. So my example is the RelevanceTable. I pull some data out that I want to do some evaluation of the data, which first start with .Contains. It appears that .Contains wants to act on the whole list or nothing. So I can use it if I have List<string>, but if I have List<ReferenceTableEntry> where ReferenceTableEntry is my custom type, I would need to override the IEquatable and tell the compiler what exactly "Equals" means.
While this doesn't seem unreasonable, it does seem like a long way to go for a simple problem so I have this sneaking suspicion that my approach is flawed from the get go.
If I want to use LINQ and .Contains, is overriding the Interface the only way? It seems like if there way just a way to say which field to operate on. Is there another collection type besides LIST that maybe has this ability. I have started using List a lot for this and while I have looked and looked, a see some other but not necessarily superior approaches.
I'm not looking for some fine point of performance or compactness or readability, just wondering if I am using a Phillips head screwdriver in a Hex screw. If my approach is a "decent" one, but not the best of course I'd like to know a better, but just knowing that its in the ballpark would give me little "Yeah! I'm not stupid!" and I would finish at least what I am doing completely before switch to another method.
Hope I explained that well enough. Thanks for you help.
What exactly is it you want to do with the table? It isn't clear. However, the standard LINQ (-to-Objects) methods will be available on any typed collection (including List<T>), allowing any range of Where, First, Any, All, etc.
So: what is you are trying to do? If you had the table, what value(s) do you want?
As a guess (based on the Contains stuff) - do you just want:
bool x= table.Any(x=>x.Foo == foo); // or someObj.Foo
?
There are overloads for some of the methods in the List class that takes a delegate (optionally in the form of a lambda expression), that you can use to specify what field to look for.
For example, to look for the item where the Id property is 42:
ReferenceTableEntry found = theList.Find(r => r.Id == 42);
The found variable will have a reference to the first item that matches, or null if no item matched.
There are also some LINQ extensions that takes a delegate or an expression. This will do the same as the Find method:
ReferenceTableEntry found = theList.FirstOrDefault(r => r.Id == 42);
Ok, so if I'm reading this correctly you want to use the contains method. When using this with collections of objects (such as ReferenceTableEntry) you need to be careful because what you're saying is you're checking to see if the collection contains an object that IS the same as the object you're comparing against.
If you use the .Find() or .FindAll() method you can specify the criteria that you want to match on using an anonymous method.
So for example if you want to find all ReferenceTableEntry records in your list that have an Id greater than 1 you could do something like this
List<ReferenceTableEntry> listToSearch = //populate list here
var matches = listToSearch.FindAll(x => x.Id > 1);
matches will be a list of ReferenceTableEntry records that have an ID greater than 1.
having said all that, it's not completely clear that this is what you're trying to do.
Here is the LINQ query involved that creates the object I am talking about, and the problem line is:
.Where (searchWord => queryTerms.Contains(searchWord.Word))
List<queryTerm> queryTerms = MakeQueryTermList();
public static List<RelevanceTableEntry> CreateRelevanceTable(List<queryTerm> queryTerms)
{
SearchDataContext myContext = new SearchDataContext();
var productRelevance = (from pwords in myContext.SearchWordOccuranceProducts
where (myContext.SearchUniqueWords
.Where (searchWord => queryTerms.Contains(searchWord.Word))
.Select (searchWord => searchWord.Id)).Contains(pwords.WordId)
orderby pwords.WordId
select new {pwords.WordId, pwords.Weight, pwords.Position, pwords.ProductId});
}
This query returns a list of WordId's that match the submitted search string (when it was List and it was just the word, that works fine, because as an answerer mentioned before, they were the same type of objects). My custom type here is queryTerms, a List that contains WordId, ProductId, Position, and Weight. From there I go about calculating the relevance by doing various operations on the created object. Sum "Weight" by product, use position matches to bump up Weights, etc. My point for keeping this separate was that the rules for doing those operations will change, but the basic factors involved will not. I would have even rather it be MORE separate (I'm still learning, I don't want to get fancy) but the rules for local and interpreted LINQ queries seems to trip me up when I do.
Since CF has supported queries of queries forever, that's how I tend to lean. Pull the data you need from the db, then do your operations (which includes queries with Aggregate functions) on the in-memory table.
I hope that makes it more clear.