I have a table that, some of its columns are unknown at compile time. Such columns could either be of an integer value, or some Enum value. There is a table that holds all the names of such dynamic columns and also holds the column's type. This "metatable" has the following columns:
DynamicColumnId (Pk)
Name
TypeId (Integer / Enum, as Fk from a separate table)
Integer columns have the Name from this table, whereas Enum columns are Fk columns from a table that has that Name, with some modification (e.g. a "DynamicTable" prefix).
The only solution I could think of for this situation is using Reflection.Emit to dynamically create an Entity class and a corresponding Mapping class. Admittedly, I'm new to NHybernate / Fluent NHybernate and it seems like a relatively simple hierarchy between the tables, and so I wanted to verify my solution isn't as ugly as it initially appears...
I would also welcome solutions that completely disregard my table hierarchy, in order to effectively acheive the same results (that is, to enumerate the rows on the dynamic table, going over all the columns, with knowledge of whether they are Enums and, if they are, their possible values as well).
(Edit: Additional information re problem domain)
I initially included minimal details, as to avoid Too-Much-Info related confusion.
This description is much more complex, but it unravels the motives behind this design.
The application involved is designed to automate log/dump analysis. Analysis-scenarios are frequently provided by the log/dump experts and so, in order to streamline the typical process of requirements=>implementation=>verification cycle, such analysis-scenarios are implemented by the experts directly as an Iron Python code snippet, with some domain-specific constructs injected into the snippets' scope. Each snippet has a "context" for which it is relevant. An example of "context" could be "product," "version," etc... So, the snippet itself is only invoked in certain contexts - this helps simplifying the Python code by eliminating branching (you could view it as Aspect Oriented Programming, to some extent). A non-expert could use the application, with a given code-context database, to analyze a log/dump, after choosing values for the various contexts.
When an expert decides that a new context is required for cataloging a certain code snippet, he could add a context, indicating the possible values it could have. Once a new context is added to the database, a non-expert that runs an analysis will be given the option to choose a value for the newly-added context.
The "dynamic table" is the table that associates a code snippet with values of the various contexts (columns) that existed when the snippet was issued, plus default values for the columns that did not exist at that time.
I won't claim to fully understand your scenario, but it seems to me that you'd be better off using a key-value store such as Redis or a schema-less database like CouchDB instead of SQL. This doesn't seem to be a problem for a relational database, but if you really need to use a RDBMS I'd map NHibernate as closely as possible to the real schema (DynamicColumnId, Name, TypeId) then build whatever data structure you need on top of that.
Related
I have a problem where I have to get the column names and their values from all the Tables in my schema and show that the result in a grid.
I have used the direct approach for this but I have to implement the SqlSiphon structure. For this I have to make getters and setters of each of the column of each Table in the schema which is impossible.
What should I use to get the Column names and their values dynamically from the table.
SELECT * FROM INFORMATION_SCHEMA.COLUMNS
WHERE
TABLE_NAME = '" + #Tablename1 + "' AND TABLE_SCHEMA='dbo'"
What will be the best dynamic solution?
And what will be Best to use List , Dictionay or something like 2d Array which will give the column names as well as column values?
A few suggestions:
I'm not completely clear on what you're trying to achieve, but consider using an ORM (Linq2SQL, WEF, NHibernate)
In .NET, a suitable type to represent a database table would be a DataTable
Edit: After a few more re-reads I think I understand what you're asking - you already have a database/schema and you want to automatically create the entity classes needed by SqlSiphon. This is called "database-first" (as opposed to model-first). However, from a brief scan of the SqlSiphon documentation it appears it does not support database-first. Is this why you are trying to put the columns into a grid - to make it easier to manually create the entity classes for SqlSiphon?
Edit2: Note that trying to use an ORM on top of a database whose schema is frequently modified will be problematic. My next guess is that you're trying to figure out how to create an entity class in SqlSiphon which you can use to retrieve database schema information like table columns? I'm still struggling to understand what you're actually asking here - perhaps you can update your question?
Edit3: I think the answer to your question is take a different approach to your design - ORM's like SqlSiphon are not intended to be used to retrieve and modify the database schema itself.
Might be worth taking a step back an comparing against how other people solve similar problems.
Typically, each table on a database represents an entity, and you also have a class per entity, and you may use an ORM system to avoid duplication of work. So, in a typical system, you have a table for customers, and a table for invoices, and a table for invoice lines, etc. and then a class that represents a customer, a class for an invoice, a class for an invoice line, etc. As you later add functionality (and possible columns/properties) you change the classes, rather than just seeing what columns are on the database - you can of course decorate these with XML documentation and get Intelisense goodness.
There are many ORM systems out there, and each have their strengths and weaknesses, but I personally like LINQ to SQL for adding onto an existing data model.
This may sound like a pipe dream, I'm wondering if it's possible. I want to be able to take a C# dynamic object, called info, and persist it to a database (I'm currently on a SQL Server 2008 database).
The info object, being dynamic, could have any number of properties: Id, Title, Content, DateExpires, DateAdded, Dateupdated, TypeOf, etc...
Each instance of it could/will contain differing number of properties, depending on what the instance is used for: blog post, classified ad, event, etc... However, there would be a core set of properties every info object would share: Id, MemberId, TypeOf...
The idea is, to have a central table which stores all dynamic info objects, yet, allow me to query based on any property (which may not exist for some objects).
For example, blog posts. They'd have: Id, MemberId, DateAdded, Title, Content, TypeOf, etc... An event would have: Id, MemberId, Title, Content, TypeOf, DateOf, Recurrance, MinAge, MaxAge, etc...
I'd like to build queries based on any given info object property.
Why? Flexibility. If I can get this working, I can use the info object for future cases within my web app. If this is an extremely bad idea, please let me know (and why) please. Thanks!
This is possible and I've seen many systems built like this...however those systems are usually the hardest to maintain due to this "generic nature". There is nothing inherently wrong with this approach. It's just that it's much harder to pull it off and in most instances it ends up being a poorly implemented.
In recent years non-relational databases (like document databases that #Marc Gravell mentioned) have caught up and they are very good for some domains but you need to make sure it's the right fit for your project.
When you take the path of building this "generic database" you are sacrificing other well-known technologies that we take for granted. For example database optimization in relational databases is well-known and there are many tools that work well with them with little or no effort. If you go a different path all of a sudden the tools that you are used to might not work and will end up either building your own to make up for the stuff that does not work (or buying/choosing some esoteric tools.)
Depending on the size of your project it might be wise to build one or two of those systems that you think would be common and then try to see if they are as common as you think.
you could use a 'base' table for the common properties, and a property name-value table for the other properties. meaning:
Table Info
int Id (PK) (FK),
int MemberId,
Date DateAdded //etc...
Table Properties
int InfoId (PK),
varchar PropertyName (PK),
varchar PropertyValue,
string PropertyType //optionaly store information about the type of property
after querying, you can use reflection to translate the properties from (name,value) pairs into proper properties.
That said, I think that this is a very bad idea, for several reasons:
1. this creates further complexity on your CRUD logic
2. you don't have well-defined entities in your domain model, which I don't like
3. validation is that much more difficult- you have to manually verify that Post, for example, doesn't have a property called Recurrance field.
I would use this method only if you truly need this flexibility- for example: if a user can choose to save custom properties that you don't know in advance.
otherwise, if you know that your entities are limited to Post, Event, Employee etc.., I would just limit myself to that.
I think this is a question of the best technique or best way to skin a cat!
Imagine a menu with items (menu choices) on it. I have a table called MenuItem, which for example "Spaghetti Bolognese", it has lots of other information associated with it aside from just a better description and picture.
Eg.
Basic Information (Name, Description, Picture, etc)
Nutritional Information (approx 15 columns)
Allergy Information (approx 16 columns)
Dietary Information (another 7 columns) (religious etc)
As it is at the moment I have it all in the one table in SQL server, which is logical database design to me as it doesn't repeat, despite it making the field list for the table longer than I would like. I'd already been feeling a bit bad about just continually extending the database table. But now we also want to add 'Recipe' information, approx another 7 columns.
I'm using Entity Framework 4.latest, and feel there is probably functionality to help me split this off within the EDMX? (Is that what ComplexTypes are?) Or do I just need to do this in the ViewModel class I call?
I think what I'm after using in my code to segregate things better is something like
MenuItem.Recipe.Ingredients
MenuItem.Nutrition.Fat
etc
Complex types can help you but be aware that complex types cannot contain navigation properties, cannot be null and are always loaded with the entity. Other possibility is to use table splitting - this will allow you to map multiple one-to-one related entities to the same table. The main features of table splitting are:
Entities can share only primary key properties
There is one main entity and others are considered as relations (navigation properties)
Related entities must exists - they are not optional so when you insert new main entity you must insert these related entities as well even if they are empty
Related entities must be loaded with eager, lazy or explicit loading
Currently our new database design is changing rapidly and I don't always have time to keep up to date with the latest changes being made. Therefore I would like to create some basic integration tests that are basically sanity checks on my mappings against the database.
Here are a few of the things I'd like to accomplish in these tests:
Detect columns I have not defined in my mapping but exist in the database
Detect columns I have mapped but do NOT exist in the database
Detect columns that I have mapped where the data types between the database and my business objects no longer jive with each other
Detect column name changes between database and my mapping
I found the following article by Ayende but I just want to see what other people out there are doing to handle these sort of things. Basically I'm looking for simplified tests that cover a lot of my mappings but do not require me to write seperate queries for every business object in my mappings.
I'm happy with this test, that comes from the Ayende proposed one:
[Test]
public void PerformSanityCheck()
{
foreach (var s in NHHelper.Instance.GetConfig().ClassMappings)
{
Console.WriteLine(" *************** " + s.MappedClass.Name);
NHHelper.Instance.CurrentSession.CreateQuery(string.Format("from {0} e", s.MappedClass.Name))
.SetFirstResult(0).SetMaxResults(50).List();
}
}
I'm using plain old query since this version comes from a very old project and I'm to lazy to update with QueryOver or Linq2NH or something else...
It basically ping all mapped entities configured and grasp some data too in order to see that all is ok. It does not care if some field exists in the table but not on the mapping, that can generate problem in persistence if not nullable.
I'm aware that Fabio Maulo has something eventually more accurate.
As a personal consideration, if you are thinking on improvement, I would try to implement such a strategy: since mapping are browsable by API, look for any explicit / implicit table declaration in the map, and ping it with the database using the standard schema helperclasses you have inside NH ( they eventually uses the ADO.NET schema classes, but they insulate all the configuration stuff we already did in NH itself) By playng a little with naming strategy we can achieve a one by one table field check list. Another improvement can be done by, in case of unmatching field, looking for a candidate by applying Levensthein Distance to all the available names and choosing one if some threshold requisites are satisfied. This of course is useless in class first scenarios when the DB schema are generated by NH itself.
I use this one too:
Verifying NHibernate Entities Contain Only Virtual Members
I'm thinking of building a ecommerce application with an extensible data model using NHibernate and Fluent NHibernate. By having an extensible data model, I have the ability to define a Product entity, and allow a user in the application to extend it with new fields/properties with different data types including custom data types.
Example:
Product can have an addition fields like:
Size - int
Color - string
Price - decimal
Collection of ColoredImage - name, image (e.g. "Red", red.jpg (binary file))
An additional requirement is to be able to filter the products by these additional/extended fields. How should I implement this?
Thanks in advance.
I think this link describes kind of what you want...
http://ayende.com/Blog/archive/2009/04/11/nhibernate-mapping-ltdynamic-componentgt.aspx
More info on dynamic-component:
http://www.mattfreeman.co.uk/2009/01/nhibernate-mapping-with-dynamic-component/
http://bartreyserhove.blogspot.com/2008/02/dynamic-domain-mode-using-nhibernate.html
The idea behind dynamic-component is that you can build your data model by not having a one to one mapping of databse columns with properties. Instead you have only a dictionary property that can contain data from as many properties as you like. This way when you fetch the entity, the dictionary gets the data of all columns configured to belong in there. You can extend the database table's schema to include more columns and that will be reflected to the databse model if you update the mapping file accordingly (manually or though code at application start).
To be honest I do not know you can query such entity using the "attributes" property but if I had to guess I would do an IN statement to it.
One of the options is EAV model (Entity-Attribute-Value).
This model is good to apply if you have a single class in your domain, which table representation would result in a wide table (large number of columns, many null values)
It's originally designed for medical domain, where objects may have thousands of columns (sympthoms).
Basically you have
Entity (Id) (for example your Product table)
Attribute(Id, ColumnName)
Value(EntityId, AttributeId, value)
You can have some additional metadata tables.
Value should better be multiple tables, one for a type.
For example:
ShortStringValue(EntityId, AttributeId, Value nvarchar(50));
LongStringValue(EntityId, AttributeId, Value nvarchar(2048));
MemoValue(EntityId, AttributeId, Value nvarchar(max));
IntValue(EntityId, AttributeId, Value int);
or even a comple type:
ColorComponentsValue(EntityId, AttributeId, R int, G int, B int );
One of the things from my experience is that you should not have EAV for everything. Just have EAV for a single class, Product for example.
If you have to use extensibility for different base classes, let it be a separate set of EAV tables.
Onother thing is that you have to invent a smart materialization strategy for your objects.
Do not pivot these values to a wide row set, pivot just a small number of collumns for your query criteria needs, then return a narrow collection of Value rows for each of the selected objects. Otherwise pivoting would involve massive join.
There are some points to consider:
. Each value takes storage space for foreign keys
. For example row-level locking will behave different for such queries, which may result in performance degradation.
. May result in larger index sizes.
Actually in a shallow hellow world test my EAV solution outperformed it's static counterpart on a 20 column table in a query with 4 columns involved in criteria.
Possible option would be to store all extra fields in an XML structure and use XPath/XQuery to retrieve them from the database.
Each extensible entity in your application will have an XML field, like ExtendedData, which will contain all extra properties.
Another option is to use Non-relationnal Databases which are typically suited for this kind of things.
NOSQL databases(couchDB, mongoDB, cassandre...) let you define dynamically your propretyfields, you could add fields to your product class whenever you want.
I'm searching for similar thing and just found N2 CMS (http://n2cms.com) which implements domain extensibility in quite usable way. It also supports querying over extension fields which is important. The only downside I find out is that it's implemented using HQL so it would take some time to reimplement it to be able to query using QueryOver/Linq, but the main idea and mappings are there. Take a look on ContentItem, DetailCollection, ContentDetail classes, their mappings and QueryBuilder/DetailCriteria.