Develop Reference

Bulkcopy 100 million rows of data to Azure SQL Server using C#?

I have a local SQL Server 2017 database, and I need to copy two tables to an Azure SQL Server database; one table has over 100 million rows of data including a "geography" type column. How do I do that?
I am right now running a bulk copy:
using (SqlConnection streamsConnection = new SqlConnection(streamsConnectionString))
{
streamsConnection.Open();
using (SqlConnection cloudConnection = new SqlConnection(cloudConnectionString))
{
cloudConnection.Open();
using (SqlCommand cmd = streamsConnection.CreateCommand())
using (SqlBulkCopy bcp = new SqlBulkCopy(cloudConnection))
{
bcp.DestinationTableName = "GroundDataNodes";
bcp.BatchSize = 200000;
bcp.BulkCopyTimeout = 1200;
bcp.NotifyAfter = 100000;
bcp.SqlRowsCopied += new SqlRowsCopiedEventHandler(s_SqlRowsCopied);
cmd.CommandText = "SELECT [Id],[nodeid],[latlon],[type],[label],[code],[lat],[lon]FROM[dbo].[GroundDataNodes]";
using (SqlDataReader reader = cmd.ExecuteReader())
{
bcp.WriteToServer(reader);
}
Console.WriteLine("Finished!");
Console.ReadLine();
}
}
}
But I'm quite new to the bulk load side and am wondering how I can improve this so it doesn't take weeks to run...

Give Azure Database Migration Services a Try. I Migrated an on-premise SQL Server with 10 Million Rows in a Days Time. But of course it also largely depends on your bandwidth.
Also:
Use the multi CPU General Purpose Pricing Tier when you create your service instance to allow the service to take advantage of multiple vCPUs for parallelization and faster data transfer.
Temporarily scale up your Azure SQL Database target instance to the Premium tier SKU during the data migration operation to minimize Azure SQL Database throttling that may impact data transfer activities when using lower-level SKUs.

so I have tried all sorts of upload stuff like the bulk upload, migration, backup etc but all had the same problem and that was my upload speed isn't up to it. It would work but take days to run. So I decided to write a server side bit of code to just populate it from there directly onto the database therefore taking my upload speeds out of the equation. I imagine if I had better upload speeds the migration tool would have worked fine even with the Geographic field etc. not quick but it would work.

Unable to begin a distributed transaction using Entiy Framework

I'm having the following error executing this piece of code:
private bool _updateList(SysInfo _sysInfo, List<pList> _pList)
{
try
{
foreach (var p in _pList)
{
_context.spUpdatePListApprovalFlow(p.countryID, _sysInfo.User.JobRoleID, p.src, p.id, p.status, _sysInfo.User.Username);
}
return true;
}
catch (Exception ex) //debug only
{
throw; //throws error to the main try catch
}
}
ERROR
The operation could not be performed because OLE DB provider "MSDASQL"
for linked server "AS400_LINKEDSRV" was unable to begin a distributed
transaction.
However, everything works fine when I run the Stored Procedure in SQL Management Studio:
exec [dbo].[spUpdatePListApprovalFlow]
#CountryID = 123456,
#UserTypeID = 23456,
#Src = 1,
#Id = '123456789',
#Status = 30,
#Username = 'username'
I'm tired of digging for an answer nothing works... Few things I've tried:
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED
insert multiple transactions in the stored procedure
This sp has 2 sub stored procedures on it. One that writes into a table of the application's database, and another that updates a table in as400.

In EF6 stored procedures are called in an explicit transaction, by default. You can turn this off for a particular DbContext instance by changing its configuration after creating it, or for all instances of a DbContext type by changing it in the constructor. EG
using (var db = new Db())
{
db.Configuration.EnsureTransactionsForFunctionsAndCommands = false;
//. . .
}
See: https://msdn.microsoft.com/en-us/library/system.data.entity.infrastructure.dbcontextconfiguration.ensuretransactionsforfunctionsandcommands

Ok, after half a day trying to solve this I've finally solved it.
Resolution
Downgraded from Entity Framework 6 to Entity Framework 5 and the
distribuited transactions error has gone.
Just pay attention, if you're going to do this, you have to change some usings in your code. ( in auto generated code in the Data Model as well)
EF 6 uses
using System.Data.Entity.Core.Objects
EF 5 uses
using System.Data.Objects;

If you don't need distributed transactions you can try to disable them in the settings of the linked server:
EXEC master.dbo.sp_serveroption
#server=N'AS400_LINKEDSRV',
#optname=N'remote proc transaction promotion',
#optvalue=N'false'

Refer to this Microsoft page on Linked Servers.
Your System Administrator and/or DBA will probably need to make changes to address the missing linked server definition to your AS/400 server.
Another possible issue is that the AS/400 server (from IBM) lacks software support for the OLE DB data sources. This too would be something that the System Administration staff may need to address.

SQL Server CE records gone when app is forced close or restarted device while app is open

I have this odd problem with my records using SQL Server CE in my WPF application. So far I have been successful in storing records on my database. I have a set of settings that I access when opening the application. I'm using LINQPad in order to view my database records.
Here's how I save my database records.
using (var dbc = new MyDbContext())
{
var settings = new UserSetting();
settings.UserId = AppUser.User.Id;
settings.MinimizeKey = minimizeKey;
settings.StartStopKey = startStopKey;
settings.IsStartAtStartUp = isStartAtStartUp ?? false;
settings.IsStoreState = isStoreState ?? false;
dbc.UserSettings.Add(settings);
dbc.SaveChanges();
}
After saving, I view my settings table and I can see that the records are there. My problem is, when I restart my device while the application is currently open. The record seems to be gone. On the other hand, when I close my application before restarting, the records are still there. Obviously restarting the device while the app is still open is not a good practice, but this will happen once in a while and I want my records to still be there.
Any ideas? Thanks!

By default, sql ce has a flush interval of 10 seconds. You can set it to 1 second by adding:
;Flush interval=1
To your connection string
Or you can force immediate flush, as described here: http://erikej.blogspot.dk/2013/05/sql-server-compact-code-snippet-of-week_21.html

Entity Framework Bulk Load Too Slow Add Seed

protected override void Seed(Fitlife.Domain.Concrete.EFDBContext context)
{
List<List<string>> foodweights = GetLines(basePath + "FoodWeights.txt");
int counter = 0;
foodweights.ForEach(line =>
{
FoodWeights newVal = new FoodWeights()
{
FoodCode = int.Parse(line[0]),
PortionCode = int.Parse(line[1]),
PortionWeight = decimal.Parse(line[2])
};
context.FoodWeights.Add(newVal);
if (++counter == 1000)
{
counter = 0;
context.SaveChanges();
}
});
}
Above method is used to populate my database. But it takes 50 seconds for 1000 entries i have a file with 470k entries, how can i improve performance i am using entity framework and this method is called when i do
PM> update-database
with Package manager. i need similar functionality, i am very new to asp.net and entity framework any guidance will be appreciated thanks.
PS: Is it ok to take 50 seconds for 1000 entries or am i doing something wrong.

The Seed method runs every time the application starts, so the way you have coded it will attempt to add the FoodWeights over and over again. EF have provided the AddOrUpdate as a convenient method to prevent that but it is really not appropriate for bulk inserts.
You could use sql directly on the database - and if you are using sql server that sql could be 'BULK INSERT'.
I would put the sql in an Up migration because you probably only want to run the insert once from a known state, and it avoids having to worry about the efficiency of the context and tracking changes etc.
There is example code and more information here: how to seed data using sql files

SQL SMO To Execute Batch TSQL Script

I'm using SMO to execute a batch SQL script. In Management Studio, the script executes in about 2 seconds. With the following code, it takes about 15 seconds.
var connectionString = GetConnectionString();
// need to use master because the DB in the connection string no longer exists
// because we dropped it already
var builder = new SqlConnectionStringBuilder(connectionString)
{
InitialCatalog = "master"
};
using (var sqlConnection = new SqlConnection(builder.ToString()))
{
var serverConnection = new ServerConnection(sqlConnection);
var server = new Server(serverConnection);
// hangs here for about 12 -15 seconds
server.ConnectionContext.ExecuteNonQuery(sql);
}
The script creates a new database and inserts a few thousand rows across a few tables. The resulting DB size is about 5MB.
Anyone have any experience with this or have a suggestion on why this might be running so slowly with SMO?

SMO does lots of weird .. stuff in the background, which is a price you pay for ability to treat server/database objects in an object-oriented way.
Since you're not using the OO capabilites of SMO, why don't you just ignore SMO completely and simply run the script through normal ADO?

The best and fastest way to upload records into a database is through SqlBulkCopy.
Particularly when your scripts are ~1000 records plus - this will make a significant speed improvement.
You will need to do a little work to get your data into a DataSet, but this can easily be done using the DataSet xml functions.