I'm importing data from a remote MySQL server. I'm connecting to the MySQL database through a SSH connection and then pulling the data into MS SQL Server. There are a couple of type checks that need to be performed, especially the MySQL DateTime to the MS SQL DateTime. Initially I thought about using the MySqlDataReader to read the data into a List<T> to ensure correct types and then pushing the data into a DataSet and then into MS SQL Server.
Is this a good approach or should I be looking into doing this a different way? I can certainly do a bulk insert into SQL Server but then I'll have to deal with the data types later.
Thoughts?
I personally wouldn't use a dataset in the process, but moving it into a .NET type, then using a parameterized SQL statement will work just fine.
If you have a very large set, you might think of looking at a bulk insert, but that will depend on the size of the set.
Here's a Microsoft Guideline for going from MySQL to Sql Server 2000:
http://technet.microsoft.com/en-us/library/cc966396.aspx
SQL Server has a rich set of tools and utilities to ease the migration from MySQL. SQL Server 2000 Data Transformation Services (DTS) is a set of graphical tools and programmable objects for extraction, transformation, and consolidation of data from disparate sources into single or multiple destinations.
From reading this article you can import your MySQL without writing a line of C#
You know, if it's a process you need to perform but you aren't limited to writing your own code, you might want to look at Talend. It's an open source tool for ETL (essentially data transforms between data sources).
It's open source and has a nice GUI for designing the transform - where things come from and where they go to, plus what happens in the middle.
http://www.talend.com/index.php
Just a thought, but if you're just trying to reach a goal as opposed to write the tool, it may be quicker and more flexible in the long run for you.
The easiest way to do it to convert it to Timestamp.
Timestamp SetupStart_ts = rs.getTimestamp("SetupStart");
String SetupStart = SetupStart_ts.toString()
Push it to mssql server straightaway and it will save automatically in the datetime but varify it.Thank you.
Related
I'm currently developing app that requires storage of lots of information from different controls. I've never done this before, and after reading couple things on this topic got a little confused. Do I need to download SQL server 2008 and work with it in order to store data from WinForm? If yes, then what is the service-side item for? Can I use for storage?I don't need to import data from database(at least not for now), I just need stuff to save somewhere, and I would like to know where exactly. Thank you!
You can choose to store your data in a serverless database (SQL CE, SQLite for example) if you don't need fancy database stuff like stored procedures, weird indexes etc. Both of the above mentioned technologies are compatible with Entity Framework, hence you can use code first approach (you can write the classes and the database would be generated on the fly).
here you can read more about it.
http://www.codeproject.com/Articles/318010/Entity-Framework-Code-First-Let-s-Try-It
And here's SQLite assembly for .Net
http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki
Good luck
You don't have to store anything in a database, you could for example serialize whatever you need to store into isolated storage
I have read one line from your question " SQL server 2008 and work with it in order to store data from WinForm"; Mean you want to save some data and that will be coming from data entry from winform.. RIght?
If that is the case you can adopt any of the RDBMS software that suites your finance + capacity needs and other perameters.
You can surely go for SQL server 2008 or Oracle or MySql or (List will go one).
What is the nature of your data?
Just because it's big does not necessarily mean a full-blown DBMS (or even an embedded database) is the best fit. Have you considered simply serializing the data to plain files?
But if your data is "relational" and you need to do the things such as querying, integrity enforcement, transactions, concurrent access by multiple clients (etc..), then you should definitely consider a database approach, including MS SQL Server, Oracle, IBM DB2, Sybase etc...
There are good open-source DBMSes as well: PostgreSQL, Firebird, MySQL...
I realize you can use the upsize wizard in access to convert this normally but as this is a server side process where we are getting the mdb files from a third party on a daily basis, I have to be able to ingest these with a no touch architecture.
Currently, I'm about to set out to write it all by hand (ugh) where I read the access database through a datasource and punch it up into sql server through bulk inserts or entity framework. I really wish there were a better way to do this though. I'm willing to entertain lots of creative methods as there are a LOT of tables and a TON of data.
There are a number of methods that come to mind, which do all indeed involve custom programming, but should be relatively simple and straightforward to implement.
From another Access DB, open the source DB programmatically (i.e., with VBA). Create linked tables to SQL backend in source DB. Copy the data from the source DB to linked table (using insert dest select * from source).
Use OPENDATASET or OPENROWSOURCE with SQL Server to directly connect to the Access DB and copy the data. You can use again insert dest select * from source to copy the data, or select * into dest from source to create a new table from the source data. This involves tweaking some system settings on sql server since it's not enabled by default, but a few google searches should get you started.
From a .NET program, use SqlBulkCopy (which is the .NET class for automating bcp) to upload data from the Access database. Just work with the data directly with ADO.Net, as there's no reason to build an entire EF layer just for migrating data from one source to another.
I have used variations of all three methods above in various projects, but for moving a large number of tables, I have found option #2 to be relatively efficient. It will involve some dynamic SQL code if your table names are dynamic on a daily basis, but if they are static, you should only have to write the logic once and use a parameter for the filename to read from.
I have a MS SQL database, that i want to copy to a mysql database,
its a very large database, with a table having about 2 million rows,
Currenlty i am copying it using code in C#, that uses datareader to loop through all rows in SQL table and inserts in a mysql table, but this is taking very long time, is there any other alternative to copy database, ?
both databases mysql and ms sql have same tables structure.
Please advice,
Thanks
Depending on the SQL Server edition you have available, you can use Integration Services, that handles the reading from SQL Server, type conversions, and writing to the target MySQL. It's optimized to do it as quickly as possible.
And it's not too hard to learn to use it.
You simply have to create a package, with a source (SQL Server), a destination (MySQL) and the extra components that you can find useful (probably only Change Type).
You can use the
MySql migration Toolkit
or
MSSQL2MySQL
Haven't used either personally but they are a couple of the toolkits available.
Much like the other answer, there are a number of ways you could go about doing this. The first thing that popped into my head was using SSIS to migrate the data.
http://www.packtpub.com/article/mysql-data-transfer-using-sql-server-integration-services-ssis
You could also look into Talend Open Studio. I use TOS to migrate data from several databases, including MS SQL, to a MySQL database. Its easy to learn and easy to maintain.
I dunno if you are willing to pay for that(I think you should in such scenario) I would recommend using DBSync for MS SQL & MySQL.
I had the task to migrate a huge 25Gb database from SQL Server to MySQL and this little program (only 17Mb compared to 277Mb Talend Open Studio ) works fine.
It s deadly simple to use , simply follow the steps (5 steps) and you go your database migrated.
I have tried Talend Open Studio and the documentation is not good at all and there is a long learning curve there whereas with DBSync, if you ever installed a software on you PC then you can also do it very easily.
Also if you one of those who like testing before buying, you can try the free version which can at least migrate the 50 first rows of all tables in a database.
I would defiantly give is a go and if you are dealing with a huge amount of sensible data,$99 is a treat. if you find any simpler or faster way to do it please let us know cause I often migrate databases and I would appreciate if you d have any better.
I'm making an application in C# (with Express Edition) for which I would like to add some SQL functionality so that the user can query a database. I don't care how or where I store the database. I can store it in a DataTable or a bi-dimensional array or any kind of file. But I want the user to be able to SQL-query it. Apparently this should be quite simple since the .net seems to be full of database libraries and stuff. I was thinking about downloading MySQL and see if I can connect it to my application. I guess if I want to distribute my application then the user would need to download MySQL as well, which is not a big deal but would be great if I can avoid it. Anyway, for now I would like to start working on my program ASAP, so whatever is the easiest way to do what I want, even if it's not distributable, (but if it is then that's even better), will be good. Thanks in advance.
There are embeddable databases. SQL Server Compact Edition and SQLite are common ones. You can execute queries against these just as you can MySQL or SQL Server.
SQLite (.NET)
SQL Server Compact
You can use most popular databases with .NET. SQL Server, Oracle, MySQL, etc. But you're gonna need drivers of each. So, I'd suggest using SQL Server Express Edition to you to get started.
Then you can easily use SqlConnection and SqlCommand classes to connect and execute queries.
You could use a dbml file in your project and link it to your sql database and then run a sql statement using Linq2SQL documented here
I would look at using and embedded database that you can distribute with your application. SQLite is a very good option. You can then use a free ADO.Net library like System.Data.SQLite for data access. It also provides design time support for Visual Studio.
You can use LINQ to Objects or LINQ to Datasets to run LINQ queries with no database whatsoever. You can't use a bi-dimensional array, but you can use a List<> of objects with properties as a LINQ context.
From your question it sounds like your application, like most applications, may need to store the data for later use: that's where a database will come in handy. .NET Datasets have built in support for persistence to an XML file if your data storage requirements are simple enough to use that. .NET also supports persistence for objects, but you may find that using a database is the simplest solution, especially if you require multi-user access and editing.
I am about to start on a journey writing a windows forms application that will open a txt file that is pipe delimited and about 230 mb in size. This app will then insert this data into a sql server 2005 database (obviously this needs to happen swiftly). I am using c# 3.0 and .net 3.5 for this project.
I am not asking for the app, just some communal advise here and potential pitfalls advise. From the site I have gathered that SQL bulk copy is a prerequisite, is there anything I should think about (I think that just opening the txt file with a forms app will be a large endeavor; maybe break it into blob data?).
Thank you, and I will edit the question for clarity if anyone needs it.
Do you have to write a winforms app? It might be much easier and faster to use SSIS. There are some built-in tasks available especially Bulk Insert task.
Also, worth checking Flat File Bulk Import methods speed comparison in SQL Server 2005.
Update: If you are new to SSIS, check out some of these sites to get you on fast track. 1) SSIS Control Flow Basics 2) Getting Started with SQL Server Integration Services
This is another How to: on importing Excel file into SQL 2005.
This is going to be a streaming endeavor.
If you can, do not use transactions here. The transactional cost will simply be too great.
So what you're going to do is read the file a line at a time and insert it in a line at a time. You should dump failed inserts into another file that you can diagnose later and see where they failed.
At first I would go ahead and try a bulk insert of a couple of hundred rows just to see that the streaming is working properly and then you can open up all you want.
You could try using SqlBulkCopy. It lets you pull from "any data source".
Just as a side note, it's sometimes faster to drop the indices of your table and recreate them after the bulk insert operation.
You might consider switching from full recovery to bulk-logged. This will help to keep your backups a reasonable size.
I totally recommend SSIS, you can read in millions of records and clean them up along the way in relatively little time.
You will need to set aside some time to get to grips with SSIS, but it should pay off. There are a few other threads here on SO which will probably be useful:
What's the fastest way to bulk insert a lot of data in SQL Server (C# client)
What are the recommended learning material for SSIS?
You can also create a package from C#. I have a C# program which reads a 3GL "master file" from a legacy system (parses into an object model using an API I have for a related project), takes a package template and modifies it to generate a package for the ETL.
The size of data you're talking about actually isn't that gigantic. I don't know what your efficiency concerns are, but if you can wait a few hours for it to insert, you might be surprised at how easy this would be to accomplish with a really naive technique of just INSERTing each row one at a time. Batching together a thousand or so rows at a time and submitting them to SQL server may make it quite a bit faster as well.
Just a suggestion that could save you some serious programming time, if you don't need it to be as fast as conceivable. Depending on how often this import has to run, saving a few days of programming time could easily be worth it in exchange for waiting a few hours while it runs.
You could use SSIS for the read & insert, but call it as a package from your WinForms app. Then you could pass in things like source, destination, connection strings etc as parameter/configurations.
HowTo: http://msdn.microsoft.com/en-us/library/aa337077.aspx
You can set up transforms and error handling inside SSIS and even create logical branching based on input parameters.
If the column format of the file matches the target table where the data needs to end up, I prefer using the command line utility bcp to load the data file. It's blazingly fast and you can specify and error file for any "odd" records that fail to be inserted.
Your app could kick off the command if you need to store the command line parameters for it (server, database, username / password or trusted connection, table, error file etc.).
I like this method better than running a BULK INSERT SQL command because the data file isn't required to be on a system accessible by the database server. To use bulk insert you have to specify the path to the data file to load, so it must be a path visible and readable by the system user on the database server that is running the load. Too much hassle for me usually. :-)