The product I'm working on will need to support different database types. At first, it needs to support SQL Server and Oracle, but in the future it may need to support IBM DB2 and Postgre SQL.
And the product will need to be working for different customers who might have slightly different schemas. For example a column name on one client with SQL Server might be _ID and on another client with Oracle it could be I_ID.
The general schema will be the same except the column names. They all could potentially be mapped to the same object. But, there may be some extra columns that are specific to each customer. These do not need to be mapped to an object though. They can be retrieved in a Master-Detail scenario using a simpler way.
I wanted use an ORM as we will need to support different types of database providers. But as far as I can understand, ORMs are not good with creating a mapping on runtime.
To support these requests (summary):
Column names may be different for each customer, but they are pretty much the same columns except names.
Database provider may be different for each customer.
There may be extra columns for each customer.
Edit: Program should be able to support a new database by changing the configuration during runtime.
What is a good way to create a data access for such specifications? Is there a way to do it with ORMs? Or do I need to write code specific to each database to support this scenario? Do I have any other option that would make it easier than using ADO.NET directly?
Edit: I think I wrote my question a bit too broad, and didn't explain it clearly, sorry about that. The problem is I won't be creating the databases. They will be created already, and the program should be able to work with a new database by configuring the program during runtime. I have no control over the databases.
The other thing is, of course it is possible to do it by creating SQL statements in the program, but that is really cumbersome. All these providers have slightly different rules and different SQL implementations, so it is a lot of work. I was wondering if I could use something like an ORM to make it easier for me.
Edit 2: I am totally aware that this is a stupid way to do things, and it shows bad design decisions. But I have spent so many hours trying to convince my company to not do it this way. They don't want to change their way of thinking because an intern tells them so. So any help would be appreciated.
Column names may be different for each customer, but they pretty much the same columns except names.
Because of this requirement alone you're going to have to build your SQL statement dynamically, on your own, but it's really pretty straight forward. I would recommend building a table like this:
CREATE TABLE DataTable (
ID INT PRIMARY KEY NOT NULL,
Name SYSNAME NOT NULL
)
to store all of the tables in the database. Then build one like this:
CREATE TABLE DataTableField (
ID INT PRIMARY KEY NOT NULL,
DataTableID INT NOT NULL,
Name SYSNAME NOT NULL
)
to store the base names for the fields. You'll just have to pick a schema and call it the baseline. That's what goes in those two tables. Then you have a table like this:
CREATE TABLE Customer (
ID INT PRIMARY KEY NOT NULL,
Name VARCHAR(256) NOT NULL
)
to store all the unique customers you have using the product, and then finally a table like this:
CREATE TABLE CustomerDataTableField (
ID INT PRIMARY KEY NOT NULL,
CustomerID INT NOT NULL,
DataTableFieldID INT NOT NULL,
Name SYSNAME,
IsCustom BIT
)
to store the different field names for each customer. We'll discuss the IsCustom in a minute.
Now you can leverage these tables to build your SQL statements dynamically. In C#, you might cache all this data up front when the application first loads and then use those data structures to build the SQL statements. But get started on that and if you have specific questions about that then create a new question, add the code you already have, and let us know where you're having trouble.
Database provider may be different for each customer.
Here you're going to need to use something like Dapper because it works with POCO classes (like what you'll be building) and it also simply extends the IDbConnection interface so it doesn't matter what concrete class you use (e.g. SqlConnection or OracleConnection), it works the same.
There may be extra columns for each customer.
This is actually quite straight forward. Leverage the IsCustom field in the CustomerDataTableField table to add those fields to your dynamically built SQL statements. That solves the database side. Now, to solve the class side, I'm going to recommend you leverage partial classes. So consider a class like this:
public partial class MyTable
{
public int ID { get; set; }
public string Field1 { get; set; }
}
and that represents the baseline schema. Now, everything maps into those fields except those marked IsCustom, so we need to do something about those. Well, let's build an extension to this class:
public partial class MyTable
{
public string Field2 { get; set; }
}
and so now when you build a new MyTable() it will always have these additional fields. But, you don't want that for every customer do you? Well, that's why we use partial classes, you define these partial classes in external assemblies that only get installed for the right customer. Now you have a bunch of small, customer specific extensions to the system, and they are easily developed, installed, and maintained.
Related
I am currently working towards implementing a charting library with a database that contains a large amount of data. For the table I am using, the raw data is spread out across 148 columns of data, with over 1000 rows. As I have only created models for tables that contain a few columns, I am unsure how to go about implementing a model for this particular table. My usual method of creating a model and using the Entity Framework to connect it to a database doesn't seem practical, as implementing 148 properties for each column does not seem like an efficient method.
My questions are:
What would be a good method to implement this table into an MVC project so that there are read actions that allow one to pull the data from the table?
How would one structure a model so that one could read 148 columns of data from it without having to declare 148 properties?
Is the Entity Framework an efficient way of achieving this goal?
Entity Framework Database First sounds like the perfect solution for your problem.
Data first models mean how they sound; the data exists before the code does. Entity Framework will create the models as partial classes for you based on the table you direct it to.
Additionally, exceptions won't be thrown if the table changes (as long as nothing is accessing a field that doesn't exist), which can be extremely beneficial in a lot of cases. Migrations are not necessary. Instead, all you have to do is right click on the generated model and click "Update Model from Database" and it works like magic. The whole process can be significantly faster than Code First.
Here is another tutorial to help you.
yes with Database First you can create the entites so fast, also remember that is a good practice return onlye the fiedls that you really need, so, your entity has 148 columns, but your app needs only 10 fields, so convert the original entity to a model or viewmodel and use it!
One excelent tool that cal help you is AutoMapper
Regards,
Wow, that's a lot of columns!
Given your circumstances a few thoughts come to mind:
1: If your problem is the leg work of creating that many properties you could look at Entity Framework Power Tools. EF Tools is able to reverse engineer a database and create the necessary models/entity relation mappings for you, saving you a lot of the grunt work.
To save you pulling all of that data out in one go you can then use projections like so:
var result = DbContext.ChartingData.Select(x => new PartialDto {
Property1 = x.Column1,
Property50 = x.Column50,
Property109 = x.Column109
});
A tool like AutoMapper will allow you to do this with ease via simply configurable mapping profiles:
var result = DbContext.ChartingData.Project().To<PartialDto>().ToList();
2: If you have concerns with the performance of manipulating such large entities through Entity Framework then you could also look at using something like Dapper (which will happily work alongside Entity Framework).
This would save you the hassle of modelling the entities for the larger tables but allow you to easily query/update specific columns:
public class ModelledDataColumns
{
public string Property1 { get; set; }
public string Property50 { get; set; }
public string Property109 { get; set; }
}
const string sqlCommand = "SELECT Property1, Property50, Property109 FROM YourTable WHERE Id = #Id";
IEnumerable<ModelledDataColumns> collection = connection.Query<ModelledDataColumns>(sqlCommand", new { Id = 5 }).ToList();
Ultimately if you're keen to go the Entity Framework route then as far as I'm aware there's no way to pull that data from the database without having to create all of the properties one way or another.
I have a DB that I need all data to be able to load into. Lets say Table name is Person, and person has a column SSID (Don't worry this isn't the real table).
I bulk copy the data into this table, and don't want any issues, so SSID is a nvarchar (8000). Then with EF I want to load this data into memory and validate that it's string lengths and such are correct, and fix them before pushing them to a final table (this part is not trivial, a transform is needed).
So my EF model:
public class Person
{
[MinLength(9)]
[MxnLength(9)]
public string SSID { get; set;}
//... more code
}
So since we're using code first, the MaxLength attribute will be used to determine the varchar length in the table, which we don't want to happen. I just want to be able to do Validator.Validate(person) and get each validation issue.
Whats the best way to go about this?
One thought is 2 models, one for the actual DB, and another for validation. Seems like data binding from one to another might be expensive/tedious.
Another thought is to re-implement MaxLength and other attributes we need to do the same thing, but EF probably wouldn't know what to do with it so it wouldn't attempt to make a schema change.
Another thought is maybe there is a setting in EF to ignore certain data annotations?
Thanks for any help.
I have a problem where I have to get the column names and their values from all the Tables in my schema and show that the result in a grid.
I have used the direct approach for this but I have to implement the SqlSiphon structure. For this I have to make getters and setters of each of the column of each Table in the schema which is impossible.
What should I use to get the Column names and their values dynamically from the table.
SELECT * FROM INFORMATION_SCHEMA.COLUMNS
WHERE
TABLE_NAME = '" + #Tablename1 + "' AND TABLE_SCHEMA='dbo'"
What will be the best dynamic solution?
And what will be Best to use List , Dictionay or something like 2d Array which will give the column names as well as column values?
A few suggestions:
I'm not completely clear on what you're trying to achieve, but consider using an ORM (Linq2SQL, WEF, NHibernate)
In .NET, a suitable type to represent a database table would be a DataTable
Edit: After a few more re-reads I think I understand what you're asking - you already have a database/schema and you want to automatically create the entity classes needed by SqlSiphon. This is called "database-first" (as opposed to model-first). However, from a brief scan of the SqlSiphon documentation it appears it does not support database-first. Is this why you are trying to put the columns into a grid - to make it easier to manually create the entity classes for SqlSiphon?
Edit2: Note that trying to use an ORM on top of a database whose schema is frequently modified will be problematic. My next guess is that you're trying to figure out how to create an entity class in SqlSiphon which you can use to retrieve database schema information like table columns? I'm still struggling to understand what you're actually asking here - perhaps you can update your question?
Edit3: I think the answer to your question is take a different approach to your design - ORM's like SqlSiphon are not intended to be used to retrieve and modify the database schema itself.
Might be worth taking a step back an comparing against how other people solve similar problems.
Typically, each table on a database represents an entity, and you also have a class per entity, and you may use an ORM system to avoid duplication of work. So, in a typical system, you have a table for customers, and a table for invoices, and a table for invoice lines, etc. and then a class that represents a customer, a class for an invoice, a class for an invoice line, etc. As you later add functionality (and possible columns/properties) you change the classes, rather than just seeing what columns are on the database - you can of course decorate these with XML documentation and get Intelisense goodness.
There are many ORM systems out there, and each have their strengths and weaknesses, but I personally like LINQ to SQL for adding onto an existing data model.
I have the following scenario: there are a database that generates a new logTable every year. It started on 2001 and now has 11 tables. They all have the same structure, thus the same fields, indexes,pk's, etc.
I have some classes called managers that - as the name says - manages every operation on this DB. For each different table i have a manager, except for this logTable which i have only one manager.
I've read a lot and tried different things like using ITable to get tables dynamically or an interface that all my tables implements. Unfortunately, i lose strong-typed properties and with that i can't do any searches or updates or anything, since i can't use logTable.Where(q=> q.ID == paramId).
Considering that those tables have the same structure, a query that searches logs from 2010 can be the exact one that searches logs from 2011 and on.
I'm only asking this because i wouldn't like to rewrite the same code for each table, since they are equal on it's structure.
EDIT
I'm using Linq to SQL as my ORM. And these tables uses all DB operations, not just select.
Consider putting all your logs in one table and using partitioning to maintain performance. If that is not feasible you could create a view that unions all the log tables together and use that when selecting log data. That way when you added a new log table you just update the view to include the new table.
EDIT Further to the most recent comment:
Sounds like you need a new DBA if he won't let you create new SPs. Yes I think could define an ILogTable interface and then make your log table classes implement it, but that would not allow you do GetTable<ILogTable>(). You would have to have some kind of DAL class with a method that created a union query, e.g.
public IEnumerable<ILogTable> GetLogs()
{
var Log2010 = from log in DBContext.2010Logs
select (ILogTable)log;
var Log2011 = from log in DBContext.2011Logs
select (ILogTable)log;
return Log2010.Concat(Log2011);
}
Above code is completely untested and may fail horribly ;-)
Edited to keep #AS-CII happy ;-)
You might want to look into the Codeplex Fluent Linq to SQL project. I've never used it, but I'm familiar with the ideas from using similar mapping techniques in EF4. YOu could create a single object and map it dynamically to different tables using syntax such as:
public class LogMapping : Mapping<Log> {
public LogMapping(int year) {
Named("Logs" + year);
//Column mappings...
}
}
As long as each of your queries return the same shape, you can use ExecuteQuery<Log>("Select cols From LogTable" + instance). Just be aware that ExecuteQuery is one case where LINQ to SQL allows for SQL Injection. I discuss how to parameterize ExecuteQuery at http://www.thinqlinq.com/Post.aspx/Title/Does-LINQ-to-SQL-eliminate-the-possibility-of-SQL-Injection.
Okay, so i've studied c# and asp.net long enough and would like to know how all these custom classes i created relate to the database. for example.
i have a class call Employee
public class Employee
{
public int ID { get; set; }
public string Name { get; set; }
public string EmailAddress { get; set; }
}
and i have a database with the following 4 fields:
ID
Name
EmailAddress
PhoneNumber
it seems like the custom class is my database. and in asp.net i can simple run the LINQ to SQL command on my database and get the whole schema of my class without typing out a custom class with getter and setter.
so let's just say that now i am running a query to retrieve a list of employees. I would like to know how does my application map to my Employee class to my database?
by itself, it doesn't. But add any ORM or similar, and you start to get closer. for example, LINQ-to-SQL (which I mention because it is easy to get working with Visual Studio), you typically get (given to you by the tooling) a custom "data context" class, which you use as:
using(var ctx = new MyDatabase()) {
foreach(var emp in ctx.Employees) {
....
}
}
This is generating TSQL and mapping the data to objects automatically. By default the tooling creates a separate Employee class, but you can tweak this via partial classes. This also supports inserts, data changes and deletion.
There are also tools that allow re-use of your existing domain objects; either approach can be successful - each has advantages and disadvantages.
If you only want to read data, then it is even easier; a micro-ORM such as dapper-dot-net allows you to use our type with TSQL that you write, with it handling the tedious materialisation code.
Your question is a little vague, imo. But what you are referring to is the Model of the MVC (Model-View-Controller) architecture.
What the Model , your Employee Class, manages data of the application. So it can not only get and set (save / update) your data, but it can also be used to notify of a data change. (Usually to the view).
You mentioned you where using SQL, so more then likely you could create and save an entire employee record by sending an Associative Array of the table data to save it to the database. Your setting for the Class would handle the unique SQL syntax to INSERT the data. In larger MVC Frameworks. The Model of your application inherits several other classes to handle the proper saving to different types of backends other than MS SQL.
Models will also, normally, have functions to handle finding records and updating records. This is normally by specify a search field, and it returning the record, of which would include the ID and you would normally base this back into a save / update function to make changes to record. You could also tie into this level of the Model to create revision of the data you are saving
So how the model directly correlates to your SQL structure is dependent on how you right it. Or which Framework you decide to use. I believe a common one for asp.net is the Microsoft's ASP.Net MVC
Your class cannot be directly mapped to the database without ORM tool, The ORM tool will read your configuration and will map your class to DB row as per your mappings automatically. That means you don't need to read the row and set the class fields explicitly but you have to provide mapping files and have to go through the ORM framework to load the entities, and the framework will take care of the rest
You can check nHibernate and here is getting started on nHibernate.