I need to import around 100k records a few dozen times a day into an AWS hosted SQL Server Web 13.00.2164.0.v1
AWS doesn't support bulk insert, and SQL Server Web doesn't support SSIS.
I am reading records from a csv file with a C# console app, performing some data transformations and then inserting 1 record at a time with SqlCommand under a single transaction per 100k record file.
My current rate is around 25k records per 30 minutes, which seems ridiculously slow. I had originally developed this process using bulk inserts and could get 100k records inserted in about a minute. Is there anything I can do to speed this up?
The native way to do it is to use Table-Valued Parameters.
See also Table-Valued Parameters in .NET for explanations how to use them in your C# code.
So, create a stored procedure that accepts a table-valued parameter and pass all 100K rows in one call. You may try to experiment with the size of the batch and try smaller batches, but 100K is not too much.
In this stored procedure there will be a single INSERT statement that inserts rows from the parameter table into the permanent table.
It will be definitely faster than inserting one row at a time.
You have to convert your data in xml format and then transfer this xml data in to sql. Like bellow example
getting data from a dataset to xml format.
string objStr = ds.GetXml();//ds is a dataset object which contains table data
at database end:
CREATE PROCEDURE [dbo].[procedure name]
-- Add the parameters for the stored procedure here
#Val varchar(max) = null
AS
BEGIN
declare #xml xml
set #xml = convert(xml,#Val)
SELECT
T.Node.value('colname[1]', 'numeric(18, 2)') AS colname
FROM
#xmlValue.nodes('/NewDataSet/Table0') AS T(Node)
END
Related
I have thousands of records from a SQL Server 2014 stored procedure result set, and I insert them one by one into a SQLite DB table through C# code, which takes around 4-5 minutes. I need to reduce this time.
I am looking for something like:
insert into 'sqlite_table'
select *
from 'sql_server_table'
Any answer with C# code or anything direct from SQL Server script can be helpful
On your C# Code. Use SqlTransaction to fasten the insertion of records from one table to another.
I Have done same thing in Sql Server to MySql. For that first I stopped the AutoCommit (Set AutoCommit=False). After the query run the commit command. It gave me good performance. You can try the same if it works for you.
How can I convert rows from dataset/datatable of SQL to rowset of Cassandra using C#?
Using Cassandra prepared statement for bulk insert gives the ff. error:
Index was outside the bounds of the array.
Values parameter have 100+ values.
batch.Add(userTrackStmt.Bind(values));
I'd say, that tools that can be used for fast import of million records would depend of your SQL complexity.
If you have issues with bulk insert (you haven't provided examples of your code and table structures so I could not help you here).
You could also try:
1. COPY command.
2. Use Spark Streaming API using [Mobius] (https://github.com/Microsoft/Mobius). So technically, you would read the first table (or SQL result), and write data to the second table in a streaming way.
I am using EF6 and SQL Server 2012. I have a table which contains text files in a varchar(max) column. Average file size is a few MBs but I am looking for a solution that will work for larger files too. I add or update this column using string data type but I don't think this is efficient way to update a few thousand records where each record is 4 to 5 Mb. Is there any efficient way to insert or update large data types in EF (or even in ADO.net)?
For large files the field can be changed to varbinary(max) and the files can be converted into binary format and inserted into the database. For files that are greater than 1MB,Filestreams provide a better option. Try to visit Convert file to binary in C# for binary conversion.
I ended up creating a stored procedure which uses write function of varchar(max). I was able to make my code 10x faster by calling SP instead of updating EF object. This SP then executes query similar to
update MyTable set col1.Write(#data, #oldLength, #data_length)
I am using an OLEDB connection (Microsoft.ACE.OLEDB.12) to read records from a C# data table and put the records into an excel spreadsheet.
I am looping round each record in my data table and building up OLEDBCommand.CommandText using an "Insert into table name values (" etc. I am having to format the values appropriately so that string/char/integer/decimal are inserted correctly. This all works fine and my spreadsheet is created, however when there are loads of records to insert (e.g. 500,000 plus), then the performance is really slow and it takes for ever. Is there a quicker way of doing this, rather than reading one record from my C# datatable at a time, making sure that the SQL statement has the correct syntax for the datatype and inserting it one at a time?
Any help appreciated
Thanks
Make sure that you use prepared statements for inserts http://msdn.microsoft.com/en-us/library/system.data.oledb.oledbcommand.prepare.aspx
Here I am facing a problem that I want to pass a dataset to a SQL Server stored procedure and I don't have any idea about it and there is no alternate solution (I think so ) to do that, let me tell what I want ...
I have an Excel file to be read , I read it successfully and all data form this excel work book import to a dataset. Now this data needs to be inserted into two different tables and there is too many rows in Excel workbook so it is not good if I run it from code behind that's why I want to pass this dataset to stored procedure and than ........
please suggest me some solution .
Not knowing what database version you're working with, here are a few hints:
if you need to read the Excel file regularly, and split it up into two or more tables, maybe you need to use something like SQL Server Integration Services for this. With SSIS, you should be able to achieve this quite easily
you could load the Excel file into a temporary staging table, and then read the data from that staging table inside your stored procedure. This works, but it gets a bit messy when there's a chance that multiple concurrent calls need to be handled
if you're using SQL Server 2008 and up, you should look at table-valued parameters - you basically load the Excel file into a .NET DataSet and pass that to the stored proc as a special parameter. Works great, but wasn't available in SQL Server before the 2008 release
since you're using SQL Server 2005 and table-valued parameters aren't available, you might want to look at Erland Sommarskog's excellent article Arrays and Lists in SQL SErver 2005 - depending on how big your data set is, one of his approaches might work for you (e.g. passing as XML which you parse/shred inside the stored proc)