Copy data from SQL Server to Mysql - c#

I have a MS SQL database, that i want to copy to a mysql database,
its a very large database, with a table having about 2 million rows,
Currenlty i am copying it using code in C#, that uses datareader to loop through all rows in SQL table and inserts in a mysql table, but this is taking very long time, is there any other alternative to copy database, ?
both databases mysql and ms sql have same tables structure.
Please advice,
Thanks

Depending on the SQL Server edition you have available, you can use Integration Services, that handles the reading from SQL Server, type conversions, and writing to the target MySQL. It's optimized to do it as quickly as possible.
And it's not too hard to learn to use it.
You simply have to create a package, with a source (SQL Server), a destination (MySQL) and the extra components that you can find useful (probably only Change Type).

You can use the
MySql migration Toolkit
or
MSSQL2MySQL
Haven't used either personally but they are a couple of the toolkits available.

Much like the other answer, there are a number of ways you could go about doing this. The first thing that popped into my head was using SSIS to migrate the data.
http://www.packtpub.com/article/mysql-data-transfer-using-sql-server-integration-services-ssis

You could also look into Talend Open Studio. I use TOS to migrate data from several databases, including MS SQL, to a MySQL database. Its easy to learn and easy to maintain.

I dunno if you are willing to pay for that(I think you should in such scenario) I would recommend using DBSync for MS SQL & MySQL.
I had the task to migrate a huge 25Gb database from SQL Server to MySQL and this little program (only 17Mb compared to 277Mb Talend Open Studio ) works fine.
It s deadly simple to use , simply follow the steps (5 steps) and you go your database migrated.
I have tried Talend Open Studio and the documentation is not good at all and there is a long learning curve there whereas with DBSync, if you ever installed a software on you PC then you can also do it very easily.
Also if you one of those who like testing before buying, you can try the free version which can at least migrate the 50 first rows of all tables in a database.
I would defiantly give is a go and if you are dealing with a huge amount of sensible data,$99 is a treat. if you find any simpler or faster way to do it please let us know cause I often migrate databases and I would appreciate if you d have any better.

Related

How to find any missing columns, constraints, indexes on a database as compared to another one

I have an c#.net windows based application that uses a database in Microsoft SQL Server 2008. During deployment for very first time to our client(s), we create a copy of our database and deploy it on client(s) remote server along with the UI application. The client database can be on version SQL Server 2005 and higher.
During times the UI application and associated database has gone lots of changes. Since this is a thick client application the client(s) database is not sync with our latest database and unfortunately no one ever made notes of all the changes done. So my challenges are as follows:
How to find any missing columns on database table in Client's Database as compared to my Database? if any?
How to find any missing Primary/Unique Constraints on database table in Client's Database as compared to my Database? if any?
How to find any missing Indexes on database table that exist in Client's Database as compared to my Database? if any?
Please keep in mind the client(s) database size may ranges from 10-100GB, so i cannot plan to just drop all client tables and recreate it.
You can use Data-tier applications. It's built-in feature of SQL Server, so you don't need to use any extra tools.
You can extract data-tier application from your database (in SSMS right-click -> Tasks -> Extract data-tier application) to a DACPAC file, copy the file to the client's server and use it to upgrade the DB there (or generate update script).
It also integrates nicely with SQL Server Data Tools.
For this task, you need a software that compare SQL database. Just like there is a lot of software to compare text, there is a lot to compare database.
Personally, I use AdoptSQLDiff, but there is a bunch. RedGate has developed one also and I know others exists. Just type SQL Database compare in google to find them. You probably can have the job done with the trial period.
These softwares show you which tables was added, deleted or changed. It does the same for views, indexes, triggers, Stored Procedures, User Defined Functions, Constraints. More importantly, those tools generate script to push modifications into the target database. Very handy, but have a look at the script generated, it sometime messes it up by deleting data, but it can be fixed very easily.
There is also the option to compare data in a specific table if you need to.
Here is a screen shot of the interface of another so you know what it's look like.
With SQLServer Management Studio, you can try selecting a database and then Task->Generate Script, selecting appropriate options.
Do the same thing for the 2 db you want to compare. You will get two text files you can compare with a text file software comparer.
Comparison will highlight difference in the db structure.
Not the best way to do it, of course. But it can be a start. If the two dbs are not too different, you should be able to handle the differences
Better option, use some db comparer software. They are meant to compare db structure, constraint indexes and so on. Never used any of them, so cannot give any advice on that
If it is one time thing use any diff tool for DB, VS2010+ has a build in one, allows you to get difference for schema and data in two different files.
If you want to solve problem of your development process, you have wide range of options to implement versioning for data base.
If you are using EF - use Migrations, can't beat that.
If you are only on SQL Server and never looking at other RDBMS, check DAC ( Data-Tier applications, mentioned by Jakub)
Otherwise take a look at more generic solutions, among them I would reccomend you to take a look at DB.UP and if python code is good for you , check Alembic, it allow you to write your migrations using really nice python API.
if nothing works for you, create snapshot of current db schema and start doing differential scripts that you can use with self written tool or DB.UP
I am not sure if this can help, but who knows.
So is there any way to restore the server database on your local environment? If the answer is yes, you can try to join system views for each database and compare them?
I propose something like this(was a quick solution, so please sorry for formatting and other common stuff).
USE [master]
GO
SELECT
LocalDataBaseTable.name AS TableName,
LocalDataBaseTableColumns.name AS [Column],
LocalDataBaseTypes.name AS DataType,
LocalDataBaseTableColumns.max_length,
LocalDataBaseTableColumns.[precision]
INTO #tmpLocalInfo
FROM LocalTable.sys.columns as LocalDataBaseTableColumns
INNER JOIN LocalTable.sys.tables AS LocalDataBaseTable
ON LocalDataBaseTableColumns.object_id = LocalDataBaseTable.object_id
INNER JOIN LocalTable.sys.types AS LocalDataBaseTypes
ON LocalDataBaseTypes.user_type_id = LocalDataBaseTableColumns.user_type_id
SELECT
ServerDataBaseTable.name AS TableName,
ServerDataBaseTableColumns.name AS [Column],
ServerDataBaseTypes.name AS DataType,
ServerDataBaseTableColumns.max_length,
ServerDataBaseTableColumns.[precision]
INTO #tmpServerInfo
FROM ServerTable.sys.columns as ServerDataBaseTableColumns
INNER JOIN ServerTable.sys.tables AS ServerDataBaseTable
ON ServerDataBaseTableColumns.object_id = ServerDataBaseTable.object_id
INNER JOIN ServerTable.sys.types AS ServerDataBaseTypes
ON ServerDataBaseTypes.user_type_id = ServerDataBaseTableColumns.user_type_id
SELECT
#tmpServerInfo.*
FROM #tmpLocalInfo
RIGHT OUTER JOIN #tmpServerInfo
ON #tmpLocalInfo.TableName = #tmpServerInfo.TableName COLLATE DATABASE_DEFAULT
AND #tmpLocalInfo.[Column] = #tmpServerInfo.[Column] COLLATE DATABASE_DEFAULT
WHERE #tmpLocalInfo.[Column] IS NULL
DROP TABLE #tmpLocalInfo
DROP TABLE #tmpServerInfo
This will return all information about missed columns in your local database. The idea is to investigate 'sys' views and to find out if there any suitable solution for you.
You can use this simple script, which show you differences between tables, views, indexes etc.
Compalex is a free lightweight script to compare two database schemas. It
supports MySQL, MS SQL Server and PostgreSQL.
or look at this question Compare two MySQL databases. This question about comparing two MySQL schemas, but some of listed tools supports MSSQL or have MSSQL version (for example http://www.liquibase.org/).
Another answer What is best tool to compare two SQL Server databases (schema and data)?

database in visual studio

I'm currently developing app that requires storage of lots of information from different controls. I've never done this before, and after reading couple things on this topic got a little confused. Do I need to download SQL server 2008 and work with it in order to store data from WinForm? If yes, then what is the service-side item for? Can I use for storage?I don't need to import data from database(at least not for now), I just need stuff to save somewhere, and I would like to know where exactly. Thank you!
You can choose to store your data in a serverless database (SQL CE, SQLite for example) if you don't need fancy database stuff like stored procedures, weird indexes etc. Both of the above mentioned technologies are compatible with Entity Framework, hence you can use code first approach (you can write the classes and the database would be generated on the fly).
here you can read more about it.
http://www.codeproject.com/Articles/318010/Entity-Framework-Code-First-Let-s-Try-It
And here's SQLite assembly for .Net
http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki
Good luck
You don't have to store anything in a database, you could for example serialize whatever you need to store into isolated storage
I have read one line from your question " SQL server 2008 and work with it in order to store data from WinForm"; Mean you want to save some data and that will be coming from data entry from winform.. RIght?
If that is the case you can adopt any of the RDBMS software that suites your finance + capacity needs and other perameters.
You can surely go for SQL server 2008 or Oracle or MySql or (List will go one).
What is the nature of your data?
Just because it's big does not necessarily mean a full-blown DBMS (or even an embedded database) is the best fit. Have you considered simply serializing the data to plain files?
But if your data is "relational" and you need to do the things such as querying, integrity enforcement, transactions, concurrent access by multiple clients (etc..), then you should definitely consider a database approach, including MS SQL Server, Oracle, IBM DB2, Sybase etc...
There are good open-source DBMSes as well: PostgreSQL, Firebird, MySQL...

Database Deployment Practices

I have deployed plenty of software to my clients. Mostly are Window Forms applications.
Here is my current practice.
Manually install SQLExpress and SQL Management Studio to each client PC.
Then use ClickOne to install the code from the server.
When there is a changes in code, I will use ClickOne to deploy -(NO PROBLEM with this step)
But when there is a change in a database column, what do I do?
I have even tried writing a database update script. Each time the program starts, it will read through the .sql update file and run them if the database exists. This solves the problem of updating the database columns, but it does not help in my DEBUGGING work when my customer complain there is a wrong data. At that point, I have to personally go to their site to check it out.
I find it difficult to have the database installed on the client PC as it make my debugging work very very difficult. I am thinking about moving my client database to a host on an Online server. But that then comes with these constraints:
What if the internet is down?
What if my customer has no internet?
Could you help to advise me? Is this a common problem faced by developer? What is the common practice out there? Does Window Azure or SQL CE help?
Depending on the data I would recommend using SQL CE.
If the data isn't too much, speed is not the primary goal (CE is slower than Express) and you don't need DB-Features not supported by CE (e.g. stored procedures) it is the better choice IMHO, because:
The client does not need to install a full SQL server (easier installation/deployment)
You do not have problems with multiple SQLExpress instances
Your SW doesn't need to worry if there even is a SQL instance
Less resources used on the client side
Additionally the clients could send you their SQL CE DB-File for inspection and you do not need to go to their site.
It is also relativly easy to implement an off site sync with SQL CE and MS Sync FW.
Installing one database per client PC can be tricky. I think you have a decent handle on how to deal with the issue currently. It seems like the real issue you are currently facing is debugging. To deal with this, there are a couple ways you could go:
Have the customer upload their copy of the database back to you. This would provide you with the data they have and you could use it with a debug copy of your code to identify the issues. The downside is that if the database is large it might be an issue transferring it.
Remote onto the customer's machine. Observe the system remotely using something like CoPilot. That way you could see what is happening in its natural environment.
There are probably other ways, but these are a couple of good ones. As for using an online database, this is an option but it brings its own set of issues with it. You mentioned a couple. As for Azure, that is cloud-based (online) so the same issues will apply. SQL CE won't help you any more than your current installation does.
Bottom line is that I would recommend you look into the ways to fix your one issue (as listed above) instead of creating a whole new set of issues by moving to an Internet-based solution. I would only recommend moving to the Internet if it was addressing a larger business need (for example, mobility). Doing the same thing you have been doing only online will probably just make life harder.
To recap the comments below since they are so pertinent to the issue, if you are choosing between file-based databases that don't need to be physically installed on the machine, your best choices are probably between SQLite and SQL CE. Microsoft supports SQL CE better but it is a larger package and has less features than the trim SQLite. Here is a good discussion on the differences:
https://stackoverflow.com/questions/2278104/sql-ce-sqlite-what-are-the-differences-between-them
However, the issue gets more complicated when you start looking at linq2sql since that is designed for SQL server. Microsoft does not support SQL CE with linq2sql out of the box, although there is a work-around that will get it to work:
http://pietschsoft.com/post/2009/01/Using-LINQ-to-SQL-with-SQL-Server-Compact-Edition.aspx
SQLite is not supported at all with linq2sql but there is a way to use linq to talk with SQLite:
LINQ with SQLite (linqtosql)
This library also supports other common databases including MySQL and Firebird.
You could use the SQLCMD utility to execute the change script, as mentioned in this related question

Easiest way to use SQL in a C# application?

I'm making an application in C# (with Express Edition) for which I would like to add some SQL functionality so that the user can query a database. I don't care how or where I store the database. I can store it in a DataTable or a bi-dimensional array or any kind of file. But I want the user to be able to SQL-query it. Apparently this should be quite simple since the .net seems to be full of database libraries and stuff. I was thinking about downloading MySQL and see if I can connect it to my application. I guess if I want to distribute my application then the user would need to download MySQL as well, which is not a big deal but would be great if I can avoid it. Anyway, for now I would like to start working on my program ASAP, so whatever is the easiest way to do what I want, even if it's not distributable, (but if it is then that's even better), will be good. Thanks in advance.
There are embeddable databases. SQL Server Compact Edition and SQLite are common ones. You can execute queries against these just as you can MySQL or SQL Server.
SQLite (.NET)
SQL Server Compact
You can use most popular databases with .NET. SQL Server, Oracle, MySQL, etc. But you're gonna need drivers of each. So, I'd suggest using SQL Server Express Edition to you to get started.
Then you can easily use SqlConnection and SqlCommand classes to connect and execute queries.
You could use a dbml file in your project and link it to your sql database and then run a sql statement using Linq2SQL documented here
I would look at using and embedded database that you can distribute with your application. SQLite is a very good option. You can then use a free ADO.Net library like System.Data.SQLite for data access. It also provides design time support for Visual Studio.
You can use LINQ to Objects or LINQ to Datasets to run LINQ queries with no database whatsoever. You can't use a bi-dimensional array, but you can use a List<> of objects with properties as a LINQ context.
From your question it sounds like your application, like most applications, may need to store the data for later use: that's where a database will come in handy. .NET Datasets have built in support for persistence to an XML file if your data storage requirements are simple enough to use that. .NET also supports persistence for objects, but you may find that using a database is the simplest solution, especially if you require multi-user access and editing.

Recommendations for single user access Database

I am writing a desktop utility application to manage a small set of data. This application will be used by a singular person so I'd like to keep the database as simple as possible. I am considering XML or SQL Server Compact 3.5 (SQL CE). I am leaning towards SQL CE because it will probably be easier/quicker to develop than XML. Are there any other worthwhile solutions worth considering? Is SQL CE the way to go?
Edit - Here are some more specifics on the data:
Maybe a half a dozen tables
No more than 5000 records
Mostly CRUD operations
Basic reporting/exporting to excel
SQLite would be my choice.
SQL Server Express
It depends on a number of parameters:
How much data will you store
Will you perform complex queries on the data
What kind of performance demands to you have
and more...
If you are going to store relatively small amounts of data, without complex relations, and without a great need to query the data in complex ways, XML might be enough. If you on the other hand expect a greater amount of data, need good query support and performance, SQL Server Express or some other lightweight database manager would be the way to go.
You can take a look at Firebird Embeeded.
link text
I've had good experiences with Sql CE, that seems like a very reasonable solution given the scenario you're describing. It has the added advantage of being much simpler to deploy than a separate solution as well, and I think that is a very big deal for a simple app like you describe.
I'm using it with Linq2Sql right now in my personal project, in fact. :)
SQL Server CE is lightweight and simple, and if you're using Visual Studio you already have it.
If this is for a single user and a limited set of data, I'd recommend looking into db4o. http://db4o.com
It's an object database that would allow you to store objects directly without having to translate them into tables.

Categories

Resources