I have a CSV file at the client side, and I want to develop a C# application to bulk insert the data into a table of a database to minimal log output. I am confused about if I use ADO.NET at the client side to call stored procedures in the database server. What kind of code needs to develop at the client side and what kind of code needs to be implemented at the server side in the form of stored procedures?
But I did not find any samples from Google. What are some ready to use samples? :-)
EDIT: Some more information:
I have a lot of data at the client side and I want to import to the database, but I do not want the overhead of all the many transaction logs. For security reasons, I want to develop a stored procedure at the server side and call from client side (ADO.NET). I want to know to achieve such goal. What kind of T-SQL needs to be developed in stored procedures at the server side and how to call/fill data efficiently at the client side?
If anything is still unclear, please feel free to let me know.
You can hook CsvReader to SqlBulkCopy, which does the job very nicely... something like (untested):
using (CsvReader reader = new CsvReader(path))
using (SqlBulkCopy bcp = new SqlBulkCopy(CONNECTION_STRING))
{
bcp.DestinationTableName = "SomeTable";
bcp.WriteToServer(reader);
}
edit you would typically do the bulk-insert into a staging table, and then use a regular stored procedure to move the data into the real table.
Are you using SQL Server 2008? And are you able to execute dynamic SQL (not that I'm advocating it)?
If so, you could construct an insert statement that makes use of "Row contructors". Essentially an insert statement will now accept an array of arguments for each row, like so:
INSERT INTO TableA (Col1, Col2)
VALUES ('A', 'B'), ('C', 'D')
There's more about it in the blog post "SQL Server 2008 – Insert Multiple Records Using One Insert Statement – Use of Row Constructor".
I hope this helps.
Related
I have thousands of records from a SQL Server 2014 stored procedure result set, and I insert them one by one into a SQLite DB table through C# code, which takes around 4-5 minutes. I need to reduce this time.
I am looking for something like:
insert into 'sqlite_table'
select *
from 'sql_server_table'
Any answer with C# code or anything direct from SQL Server script can be helpful
On your C# Code. Use SqlTransaction to fasten the insertion of records from one table to another.
I Have done same thing in Sql Server to MySql. For that first I stopped the AutoCommit (Set AutoCommit=False). After the query run the commit command. It gave me good performance. You can try the same if it works for you.
Because I'm using a shared Microsoft SQL Server instance, I have started to use MongoDB as it's faster to have around 10 inserts per second.
At the end of the day, I need to read from the MongoDB (MongoHQ) and insert all data into the shared SQL Server instance (hosting provider).
Currently I'm doing this flow:
Get 1000 rows from MongoDB
open a connection to SQL Server
For each row
insert the data into SQL Server using a stored procedure (as calculations need to be done)
update MongoDb row * (so we don't pick up the processed row later)
close the connection to SQL Server
GO TO 1
And with this, I'm "wasting" around 3 seconds per row...
Is there a way to speed things up?
Currently using C# to perform all the code.
You have SQLbulkcopy class to your rescue.
Read more details here.
http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlbulkcopy(v=vs.100).aspx
Thanks,
Naval
Please consider BULK INSERT functionality
http://msdn.microsoft.com/en-us/library/ms188365.aspx
I have a Windows Service application that receives a stream of data with the following format
IDX|20120512|075659|00000002|3|AALI |Astra Agro Lestari Tbk. |0|ORDI_PREOPEN|12 |00000001550.00|00000001291.67|00001574745000|00001574745000|00500|XDS1BXO1| |00001574745000|ݤ
IDX|20120512|075659|00000022|3|ALMI |Alumindo Light Metal Industry Tbk. |0|ORDI |33 |00000001300.00|00000001300.00|00000308000000|00000308000000|00500|--U3---2| |00000308000000|õÄ
This data comes in millions of rows and in sequence 00000002....00198562 and I have to parse and insert them according to the sequence into a database table.
My question is, what is the best way (the most effective) to insert these data into my database? I have tried to use a simple method as to open a SqlConnection object then generate a string of SQL insert script and then execute the script using SqlCommand object, however this method is taking too long.
I read that I can use Sql BULK INSERT but it has to read from a textfile, is it possible for this scenario to use BULK INSERT? (I have never used it before).
Thank you
update: I'm aware of SqlBulkCopy but it requires me to have DataTable first, is this good for performance? If possible I want to insert directly from my data source to SQL Server without having to use in memory DataTable.
If you are writing this in C# you might want to look at the SqlBulkCopy class.
Lets you efficiently bulk load a SQL Server table with data from another source.
First, download free LumenWorks.Framework.IO.Csv library.
Second, use the code like this
StreamReader sr = new TextReader(yourStream);
var sbc = new SqlBulkCopy(connectionString);
sbc.WriteToServer(new LumenWorks.Framework.IO.Csv.CsvReader(sr));
Yeah, it is really that easy.
You can use SSIS "Sql Server Integration Service" for converting data from source data flow to destination data flow.
The source can be a text file and destination can be a SQL Server table. Your conversion executes in bulk insert mode.
Here I am facing a problem that I want to pass a dataset to a SQL Server stored procedure and I don't have any idea about it and there is no alternate solution (I think so ) to do that, let me tell what I want ...
I have an Excel file to be read , I read it successfully and all data form this excel work book import to a dataset. Now this data needs to be inserted into two different tables and there is too many rows in Excel workbook so it is not good if I run it from code behind that's why I want to pass this dataset to stored procedure and than ........
please suggest me some solution .
Not knowing what database version you're working with, here are a few hints:
if you need to read the Excel file regularly, and split it up into two or more tables, maybe you need to use something like SQL Server Integration Services for this. With SSIS, you should be able to achieve this quite easily
you could load the Excel file into a temporary staging table, and then read the data from that staging table inside your stored procedure. This works, but it gets a bit messy when there's a chance that multiple concurrent calls need to be handled
if you're using SQL Server 2008 and up, you should look at table-valued parameters - you basically load the Excel file into a .NET DataSet and pass that to the stored proc as a special parameter. Works great, but wasn't available in SQL Server before the 2008 release
since you're using SQL Server 2005 and table-valued parameters aren't available, you might want to look at Erland Sommarskog's excellent article Arrays and Lists in SQL SErver 2005 - depending on how big your data set is, one of his approaches might work for you (e.g. passing as XML which you parse/shred inside the stored proc)
Using C# (vs2005) I need to copy a table from one database to another. Both database engines are SQL Server 2005. For the remote database, the source, I only have execute access to a stored procedure to get the data I need to bring locally.
The local database I have more control over as it's used by the [asp.net] application which needs a local copy of this remote table. We would like it local for easier lookup and joins with other tables, etc.
Could you please explain to me an efficient method of copying this data to our local database.
The local table can be created with the same schema as the remote one, if it makes things simpler. The remote table has 9 columns, none of which are identity columns. There are approximately 5400 rows in the remote table, and this number grows by about 200 a year. So not a quickly changing table.
Perhaps SqlBulkCopy; use SqlCommand.ExecuteReader to get the reader that you use in the call to SqlBulkCopy.WriteToServer. This is the same as bulk-insert, so very quick. It should look something like (untested);
using (SqlConnection connSource = new SqlConnection(csSource))
using (SqlCommand cmd = connSource.CreateCommand())
using (SqlBulkCopy bcp = new SqlBulkCopy(csDest))
{
bcp.DestinationTableName = "SomeTable";
cmd.CommandText = "myproc";
cmd.CommandType = CommandType.StoredProcedure;
connSource.Open();
using(SqlDataReader reader = cmd.ExecuteReader())
{
bcp.WriteToServer(reader);
}
}
Bulk Copy feature of ADO.NET might help you take a look at that :
MSDN - Multiple Bulk Copy Operations (ADO.NET)
An example article
I would first look at using SQL Server Intergration Services (SSIS, née Data Transfer Services (DTS)).
It is designed for moving/comparing/processing/transforming data between databases, and IIRC allows an arbitrary expression for the source. You would need it installed on your database (shouldn't be a problem, it is part of a default install).
Otherwise a code solution, given the data size (small), pull all the data from the remove system into an internal structure, and then look for rows which don't exist locally to insert.
You probably can't do this, but if you can't, DON'T do it with a program. If you have any way of talking to someone who controls the source server, see if they will set up some sort of export of the data. If the data is as small as you say, then xml or csv output would be 100x better than writing something in c# (or any language).
So let's assume they can't export, still, avoid writing a program. You say you have more control over the destination. Can you set up an SSIS package, or setup a linked server? If so, you'll have a much easier time migrating the data.
If you set up at bare minimum the source as a linked server you could write a small t-sql batch to
TRUNCATE DestTable
INSERT INTO DestTable
SELECT SourceTable.Star FROM [SourceServer].[Schema].[Table]
wouldn't be as nice as SSIS (you have more visual of what's happening, but the t-sql above is pretty clear).
Since I would not take the programming route, the best solution I could give you would be, if you absolutely had to:
Use SqlClient namespace.
So, create 2 SqlConnections, 2 SqlCommands, and get the instance of the 1 SqlReader.
Iterate through the source reader, and execute the destination SqlCommand insert for each iteration with the.
It'll be ugly, but it'll work.
Doesn’t seem to be huge quantity of data you have to synchronize. Under conditions you described (only SP to access the remote DB and no way to get anything else), you can go for Marc Gravell’s solution.
In the case the data can only grow and existing data can not be changed you can compare the record count on remote and internal DB in order to optimize operation; if no change in remote DB no need to copy.