Develop Reference

Better way to work with multiple databases

So the problem is: I am working with a legacy project. Creating a web interface to databases. ASP.NET Core, CRUD, that usual stuff. BUT!
There are 3(three) databases. Let's say there are Companies and Employees tables in each one of them. And there are Companies in the first database which employees may be found in first, second and third databases.
Currently, I am writing custom queries to target multiple databases. Working directly with MySqlConnection classes and MySqlDataReader.
It gets the job done, but I am really missing Entity Framework's lazy loading
I know I can create multiple EF contexts, one for each database, but it's not gonna work because as I said - data may be spread out through all 3 DB.
Is there any better way to communicate with multiple databases?

You can create synonyms in one database for tables located in another database and try to use it in EF.

Multiple databases with slightly changing models. How do I allow `EF` to work with different database structures at run-time?

I am working with EF6, MSSQL, Oracle, .NET4.5 on a system that is used globally across company (many departments) to query different databases that belong to our department, that have mostly same EF model, some databases are Oracle and some are Microsoft SQL, some are development or uat, some are logs.
I am using different EF models for Oracle and for MSSQL databases.
One requirement is to switch between databases at run time, and this is easy,
public AggregatorEntities(string connectionString)
: base(connectionString)
{
}
however it does have side effects - many databases (dev, uat, dr, logs,...) are out of sync from what Live is (model is generated from Live), which results in errors when querying those databases.
Management knows about situation and they are ok for devs that work on some specific database to do changes to global querying system that would allow testers and uat to query the data. However they want changes they have to do to take minimum time to do this - as it is additional cost to each project that involves database changes. I will basically need to build a 'can handle all' resilient system, that when one changes database in EF will do something to accommodate to specific database.
There are different failure scenarios:
1. Name of column on table is the same but Type is different in entity
2. No column on table but there is one on entity in EF
3. Additional columns on table that are not on EF
4. Additional tables in database that are not in EF model
5. No table in database but there is entity in EF model.
I have done some thinking and this question is broad and might get closed for same reason. However I am not sure if it is worth splitting the question into each scenario, as it depends on the answer. The way I understand if single answer can answer all points then no need to split, however if each situation has different 'cure' then question should be split for that part only, but without answer no way to know.... (catch 22).
Only option I see ATM is to generate it's own model for each mirroring database, but then I end up with 50+ models.
How do I allow EF to work with different database structures at run-time?

This now officially cannot be done in a proper manner.
However end result of being able to switch between different databases with similar structures still can be achieved (for those without morals). Part with removing columns can used.
Solution is to have all inclusive EF model that is generated from database that has all the tables and all the columns (that are in any database think like logical OR of everything). Then model with all entities that have all properties from all db environments can be removed specific to environment that is queried at runtime in mechanism described here. This does not cover cases where type of column changes.
Hope this saves you some time as it took 2 weeks from mine...

Database schema for multiple companies

I am working on a inventory app using c# and entity framework code first approach.
One of the design requirements is that user should be able to create multiple companies and each company should have a full set of inventory master tables.
For example each company should have its own stock journal and list of items. There would also be a way to combine these companies in future to form like a 'group' company, essentially merging the data.
Using one of the file based RDBMS like sqlite, its very simple, I would just need to create a separate sqlite database for each company and then a master database to tie it all together. However how should I go about doing it in a single database file! not multiple file databases.
I do not want to have a 'company' column on every table!
The idea that I had given my limited knowledge of DB's is to separate using different schemas. One schema for each company with the same set of tables in each schema, with a separate schema holing the common tables and tables to tie up the other schemas together. Is that a good approach? Because I am having a hard time finding a way to 'dynamically' create schemas using ef and code first.
Edit #1
To get an idea of the number of companies, one enterprise has about 4-5 companies, and each financial year the old companies are closed off and a fresh set of companies created. It is essentially good to maintain data for multiple years in the same file but it is not required as long as I can provide a separate module to load data for several years, from several of the db files to facilitate year on year analysis.
As far as size of individual companies data, it can hit the GB mark per company.
Schema changes quite frequently at least on the table level as it will be completely customizable by the user.
I guess one aspect that drives my question is the implementation of this design. If it is a app with discrete desktop interface and implementation and I have my on RDBMS server like SQL Server the number of databases do not matter that much. However for a web-based UI hosted on third party and using their database server, the number of databases available will be limited. The only solution to that would be to use serverless database like SQLite.
But as far as general advice goes, SQLite is not advised for large enterprise class databases.

You've provided viable solutions, and even some design requirements, but it's difficult to advise "what's best" without knowing the base requirements like:
How many companies now - and can be reasonably expected in the future
How many tables per instance
How many records per 'large' table, per company
How likely are things to change frequently, dataschema-wise
With that in mind, off to some general opinion on your solutions. First off, considering the design requirements, it would make sense to consider using seperate databases per company. This would seperate your data and allow for example roles and security quite easily to be defined on a database level. Considering you explicitely mention you could "make it simple" using this approach, you could just create a database (file) per company. With your data access layer through Entity Framework you could also easily change connection strings between databases, and even merge data from A=>B using this. I see no particular reason, besides a possible risk in maintaining and updating different instances, why this shouldn't be a solution to consider.
On the other hand, using the one-big-database-for-all approach, isn't bad by definition either. The domain of maintenance becomes more compact and easily approachable. One way to seperate data is to use different database schemas, as you suggest yourself. However, database schemas are primarily intended to seperate the accessability on a role based level. For example, a backoffice employee e.g. user role should only communicate to the "financial" schema, whilst the dbo can talk to pretty much anything. You could extend this approach on a company base, seeing a company as a "user", but think of the amount of tables you would get if you have to create more and more companies. This would make your database huge. Therefor, in my opinion, not the best approach.
Finally, I'm intrigued by your statement "I do not want to have a 'company' column on every table". In my opinion, you should consider this as well. Having a discriminator property, like the companyId column on several tables are pretty easy to abstract using Entity Framework (or any ORM for that matter). This is what the concept of foreign keys is all about. Also, it would give you the advantage of indexing this column for performance. Your only consideration in this approach would be to make sure you provide this 'company discriminator' on all relevant tables.
The latter would be quite simple to enforce using EF Code First if you use a contract for each seperate data class to inherit from:
interface IMyTableName {
int companyId;
}
Just my quick thoughts, though.

I agree with Moriarty for the most part. Our company chose the one database per company approach, and we're paying for it every time we want to do a schema change. Since our deployments are automated, they should all be the same, but there are small differences each time. Moreover, these databases are really independent, so it's hard to keep our backups in sync as well.
It has been painful working with all these databases. The only plus side is that we can spread them out over multiple servers to increase performance. So I'm going to cast my vote for the one big database design.

c# design so that I can easily add/update objects which mapped to databases

I have a table in the db called task. This table has many other tables associate with it. For example, a task has an individual associate with it, and has many resources. Many of the columns in task has codes such as language id, group id and so on.
I want to implement a class called Task. This class will be able to insert a task and all the related values in different tables. I want it to be flexible and robust enough so that I can keep added different objects to insert/update the underlying db. What kind of design pattern is best for it? Can you give me an example? thanks!

You're asking about ORM (Object-Relational Mapping).
There's a whole list of possibilities at Wikipedia here.
Link to SQL, Entity Framework (as noted in comments above) and SubSonic are all popular choices. The first two come included with .NET and Visual Studio in more recent versions.
Come to think of it, even Typed DataSets with Table Adapters are not too far off from what you're looking for.

Entity Attribute Value (EAV) frameworks?

I'd seen Entity Attribute Value in lots of contexts before I actually learnt what its name was. Its that technique that often crops up when instead of storing data in database columns you 'flip it' and have a table with Entity, Attrbute, Value columns and each piece of data becomes a row in that table. Sometimes its also known as 'Open-Schema'.
Its good for some things, bad for other things. This wikipedia article has a good discussion of the theory behind it.
It seems like the sort of oft-used technique that should have Frameworks or Engines or NoSQL Databases or general software tools to build and support it.
So, do you know of any? I'm particularly interested in the Microsoft stack (.Net, SQL Server, etc), but also in other technology stacks.
For example, here's a project to build an ASP.NET EAV engine that is exactly what I'm looking for, but apparently never got started.

If you can live with the drawbacks of NoSQL databases, the best way to approach the EAV pattern is with a NoSQL alternative like CouchDB or MongoDB. These databases offer a "schemaless" design which allows each row to have its own schema. Doing an EAV with a traditional RDBMS is asking for trouble as the querying becomes very difficult and performance suffers the larger the dataset.
An alternative that I have used successfully in the past is to combine an RDBMS with a NOSQL variant (MySql and MongoDB). I use MySQL to store the EAV data (gaining the transactional integrity), and I use MongoDB as a reporting store to get around the querying issues with the EAV model.

You could store it in SQL-XML, donno of a lib tho, but you could do the de/serialization in .NET, then X-LINQ aginst it.
Performance will also be a tremendous issue obviously.

I'll get the ball rolling with one I found via this blog post:
An early Beta for a SQL Server EAV framework:
http://eav.codeplex.com/
"A sample EAV pattern for SQL Server with: Tables and indexes, Partial referential integrity, Partial data typing, Updatable views (like normal SQL table)"
Provides some SQL scripts to download, here.