I need to insert around 5 million rows of data from one table to another using C#. Please recommend a good way to do this. Thanks.
write a Stored Procedure or query and then run it from C# code.
the query will look like this -
INSERT INTO table2
SELECT * FROM table1;
Related
I want to perform bulk insert from CSV to MySQL database using C#, I'm using MySql.Data.MySqlClient for connection. CSV columns are refereed into multiple tables and they are dependent on primary key value, for example,
CSV(column & value): -
emp_name, address,country
-------------------------------
jhon,new york,usa
amanda,san diago,usa
Brad,london,uk
DB Schema(CountryTbl) & value
country_Id,Country_Name
1,usa
2,UK
3,Germany
DB Schema(EmployeeTbl)
Emp_Id(AutoIncrement),Emp_Name
DB Schema(AddressTbl)
Address_Id(AutoIncrement), Emp_Id,Address,countryid
Problem statement:
1> Read data from CSV to get the CountryId from "CountryTbl" for respective employee.
2> Insert data into EmployeeTbl and AddressTbl with CountryId
Approach 1
Go as per above problem statement steps, but that will be a performance hit (Row-by-Row read and insert)
Approach 2
Use "Bulk Insert" option "MySqlBulkLoader", but that needs csv files to read, and looks that this option is not going to work for me.
Approach 3
Use stored proc and use the procedure for upload. But I don't want to use stored proc.
Please suggest if there is any other option by which I can do bulk upload or suggest any other approach.
Unless you have hundreds of thousands of rows to upload, bulk loading (your approach 2) probably is not worth the extra programming and debugging time it will cost. That's my opinion, for what it's worth (2x what you paid for it :)
Approaches 1 and 3 are more or less the same. The difference lies in whether you issue the queries from c# or from your sp. You still have to work out the queries. So let's deal with 1.
The solutions to these sorts of problems depend on make and model of RDBMS. If you decide you want to migrate to SQL Server, you'll have to change this stuff.
Here's what you do. For each row of your employee csv ...
... Put a row into the employee tbl
INSERT INTO EmployeeTbl (Emp_Name) VALUES (#emp_name);
Notice this query uses the INSERT ... VALUES form of the insert query. When this query (or any insert query) runs, it drops the autoincremented Emp_Id value where a subsequent invocation of LAST_INSERT_ID() can get it.
... Put a row into the address table
INSERT INTO AddressTbl (Emp_Id,Address,countryid)
SELECT LAST_INSERT_ID() AS Emp_Id,
#address AS Address,
country_id AS countryid
FROM CountryTbl
WHERE Country_Name = #country;
Notice this second INSERT uses the INSERT ... SELECT form of the insert query. The SELECT part of all this generates one row of data with the column values to insert.
It uses LAST_INSERT_ID() to get Emp_Id,
it uses a constant provided by your C# program for the #address, and
it looks up the countryid value from your pre-existing CountryTbl.
Notice, of course, that you must use the C# Parameters.AddWithValue() method to set the values of the # parameters in these queries. Those values come from your CSV file.
Finally, wrap each thousand rows or so of your csv in a transaction, by preceding their INSERT statements with a START TRANSACTION; statement and ending them with a COMMIT; statement. That will get you a performance improvement, and if something goes wrong the entire transaction will get rolled back so you can start over.
I have some data that needs to be imported into SQL Server.
I have the following fields:
ID Param1 Param2
The way it needs to go into the table is not that straighforward.
It needs to go in as
ID Param1 5655 DateTime
ID Param2 5555 DateTime
as such, it needs to insert 2 records into the table for one row from the input file. Wondering what the best way to do this in SQL Server is in terms of importing the file. I can do a BULK INSERT but I the columns need to match exactly. In my case it does not
I am also using .NET C#. Wondering if importing file to datatable, etc. and then using foreach look to further manipulate it may be the best approach.
As the question was a little bit unclear for me but if I'm getting you well then there is many ways for doing it one simple way is using a temp table:
create a temp table:
CREATE TABLE #TBL (ID int, param1 datetime, param2 datetime);
bulk insert from file into temp table
BULK INSERT #TBL FROM 'D:\data.txt' WITH (FIELDTERMINATOR = ' ');
now you can insert into permanent table using a specific query on the temp table (assuming your table structure is: (ID,param) ):
INSERT INTO TABLE_NAME(id,PARAM)
SELECT DISTINCT T.ID,T.PARAM1
FROM #TBL
UNION
SELECT DISTINCT T.ID,T.PARAM2
FROM #TBL
Since you are using C#, you can make use of Table-Valued Parameters to stream in the data in any way you like. You can read a row from a file, split it apart, and pass in 2 rows instead of mapping columns 1 to 1. I detailed a similar approach in this answer:
How can I insert 10 million records in the shortest time possible?
The main difference here is that, in the while loop inside of the GetFileContents() method, you would need to call yield return twice, once for each piece.
I have two database for example(A and B). I have already made a crystal report which take values from database 'B'. But now I want to upload this report to the Database 'A' with an unique id. When the user will see the report the program will fetch the report from Database 'A'. And the the report will take the data from database 'B'. Need some suggestions.
We can copy all columns from one table to another, existing table:
INSERT INTO table2
SELECT * FROM table1;
Or we can copy only the columns we want to into another, existing table:
INSERT INTO table2
(column_name(s))
SELECT column_name(s)
FROM table1;
Assuming that your are calling a stored procedure to fetch data for your report.
In your SP after writing your select statement for your report, you can fire insert queries to databaseA from databaseB itself
If your databases are across different servers and both the servers can talk, you could do in with a SQL insert statement.
first run the following on your first server:
Execute sp_addlinkedserver SERVER_NAME1
Then just create the insert statement:
INSERT INTO [SERVER_NAME1].DATABASE_NAME.dbo.TABLE_NAME (Names_of_Columns_to_be_inserted)
SELECT Names_of_Columns_to_be_inserted
FROM TABLE_NAME
If both the databases are on the same server then you can exclude the [SERVER_NAME1] part from the above query.
If this doesnt solve your query then please elaborate your question.
Hope this helps
I'm currently calling a stored procedure from a .net application that inserts records into the database. However, I need to get a list of the records that I've just inserted successfully.
I know that I could return the inserted rows from the stored procedure directly, but I was hoping there was a way to do this programatically in C#.
Is it possible to implement the SQLDataReader class in order to achieve this functionality i.e. reading from the inserted/deleted tables? Or is there some other class that can accommodate this request?
The inserted/deleted tables are available in the OUTPUT clause of the INSERT statement.
You could use the OUTPUT clause in the INSERT in your stored procedure and use SQLDataReader to pick up the result.
How can I insert values into two tables at once?
if it not successful, both table should rollback.
I am using SQL server and the query passe throw C# code.
You could either run the two queries as one statement
insert into table1 (...) values (...); insert into table2 (...) values (...)
or write a trigger to do the second INSERT.
I would typically write a stored procedure to take in all of the values you want to write out, then call a series of INSERT INTO statements wrapped in a transaction.
If you provide more information, such as table structure and sample data, we can help you further.
get ans here of ur question
SQL Server: Is it possible to insert into two tables at the same time?
How can I INSERT data into two tables simultaneously in SQL Server?